Move kubectl wait to informers with a cache to avoid hanging due to objects disappearing from the cluster

[mpuckett159]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Copied from #108086 as this PR attempts to address some regressions that were caused by this PR and add tests to ensure the regression is not done inadvertently again.

This moves the kubectl wait set of functions to using informers with cache updates for waiting on resources to reach a specified state. It prevents wait from hanging due to resources disappearing and outputs a descriptive error message when a resource it is waiting on disappears.

Example output:

☸  context: minikube (namespace: default) in ~/tmp 
❯ devkc wait --for=condition=ready pod --all --timeout=15s
pod/nginx-deployment-9456bbbf9-f9k7k condition met
pod/nginx-deployment-9456bbbf9-k2fhr condition met
pod/nginx-deployment-9456bbbf9-l886q condition met
pod/nginx-deployment-9456bbbf9-qsgqh condition met
pod/nginx-deployment-9456bbbf9-sjfdp condition met
pod/nginx-deployment-9456bbbf9-w85zc condition met
Error from server (NotFound): pods "nginx-deployment-9456bbbf9-zs5vq" not found

☸  context: minikube (namespace: default) in ~/tmp 
❯ devkc wait --for=condition=ready pod --all --timeout=15s
pod/nginx-deployment-9456bbbf9-f9k7k condition met
pod/nginx-deployment-9456bbbf9-l886q condition met
pod/nginx-deployment-9456bbbf9-sjfdp condition met
Error from server (NotFound): pods "nginx-deployment-9456bbbf9-k2fhr" not found

☸  context: minikube (namespace: default) in ~/tmp 
❯ devkc wait --for=condition=ready pod --all --timeout=15s
pod/nginx-deployment-9456bbbf9-f9k7k condition met

Which issue(s) this PR fixes:

Fixes kubernetes/kubectl#1120

Special notes for your reviewer:

I had to bump all the timeouts in the tests up to at least 1 second due to how the informer and caching works. From what I can tell on my local testing it doesn't actually increase testing time that significantly (2 seconds for me) but just fyi.

Does this PR introduce a user-facing change?

NONE

cleared release note since this is reverted in #110922, original release note was:

kubectl wait no longer hangs on resources that have disappeared from the cluster

[mpuckett159]

/triage accepted
/ok-to-test

[soltysh]

This ensures the comment from #108086 (comment) is addressed and adds tests ensuring that we don't break it in the future.

/lgtm
/approve

[soltysh]

/priority backlog

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mpuckett159, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/kubectl/OWNERS [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

wait unit tests have been significantly flakier since this merged:

https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=kubectl%2Fpkg%2Fcmd%2Fwait

[aleksandra-malinowska]

This also correlates with the start of #111111 (test times out because kubectl delete never completes).

[mpuckett159]

This also correlates with the start of #111111 (test times out because kubectl delete never completes).

Note for myself for fixing, it looks like the delete code doesn't set the timeout value properly in the waitOptions, using 0 for "forever" when it should be using it for "check once and report immediately."

@aleksandra-malinowska this may point to an underlying issue with the testing, however. Could you point me to the specific test code so I can check to see what specifically is being run to cause these hangs? It sounds like the resource is not being deleted as one would expect, and in combination with the timeout issue is causing this hanging to occur. If the test just does kubectl delete then returns an error that is ignored then the wrong timeout value issue would definitely be the source of these failures.