Update vendoring to k8s 1.20/1.21

[ggaurav10]

Logs of MCM:

I0201 07:37:20.264399       1 machine_safety.go:69] reconcileClusterMachineSafetyOvershooting: Start
I0201 07:37:20.264533       1 machine_safety.go:374] checkAndFreezeORUnfreezeMachineSets: MS:"shoot--garden--az-us2-cpu-worker-z1-6979c847b6" LowerThreshold:4 FullyLabeledReplicas:4 HigherThreshold:5
I0201 07:37:20.565096       1 machine_safety.go:95] reconcileClusterMachineSafetyOvershooting: End, reSync-Period: 1m0s
E0201 07:37:24.347148       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-wbmqr" not found
E0201 07:37:24.349910       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-4cpkc" not found
E0201 07:37:24.349993       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-r6wnm" not found
E0201 07:37:24.350033       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-5n7d8" not found
E0201 07:37:34.589908       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-wbmqr" not found
E0201 07:37:34.592394       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-r6wnm" not found
E0201 07:37:34.592409       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-5n7d8" not found
E0201 07:37:34.592444       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-4cpkc" not found
I0201 07:37:35.077956       1 machine_safety.go:105] reconcileClusterMachineSafetyAPIServer: Start
I0201 07:37:35.084685       1 machine_safety.go:174] reconcileClusterMachineSafetyAPIServer: Stop
E0201 07:37:55.073089       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-wbmqr" not found
E0201 07:37:55.075920       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-5n7d8" not found
E0201 07:37:55.075931       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847b6-4cpkc" not found
E0201 07:37:55.077233       1 machine.go:158] Could not fetch machine object machines.machine.sapcloud.io "shoot--garden--az-us2-cpu-worker-z1-6979c847

MCM doesn't reconcile machines anymore. Restarting MCM results in all the machines getting recreated again, after which the issue is resolved.

[ggaurav10]

Here there should be a check like this so that a non-existent machine key is not requeued. Also, please note that at the second link, the Get is from informer cache while the called function reconcileClusterMachine Get's from apiserver. Not sure why the get call from cache is not failing with NotFound for such long time.

[prashanth26]

I think this is related to this issue - #394

[prashanth26]

The above fix partially closes this issue. However, the larger issue of cache inconsistency is yet to be fixed.

[prashanth26]

We have observed that the cache get's outdated for different machine CRD objects causing MCM to stop reconciling objects. We need to validate why this happens.

[prashanth26]

With help from @rfranzke, it has been identified that the issue is with the Kubernetes version - 1.14.8. Updating to k8s version 1.15 and above seems to fix it.

Refer to - kubernetes/client-go#755. For more details.

[amshuman-kr]

Ideally, the client-side should implement the timeout by itself without depending on the server side to close the channel on timeout. This has already been done recently in client-go but not released yet. We should adopt this change as soon as it is released to insulate ourselves from such issues in the future.

[prashanth26]

The issue has been fixed on the server side with the K8s version update to 1.15. However, for the client-side fix we could wait for 1.18 client-go.

@amshuman-kr - Let's keep this issue open to track the same. And once 1.18 is released we could adopt the changes.