#50102 Task 3: Until, backed by retry watcher

[tnozicka]

What this PR does / why we need it:
This is a split off from #50102 to go in smaller pieces.

Introduces Until based on RetryWatcher. It can survive closed watches if last read ResourceVersion is still present in etcd.

Fixes: #31345

Requires:

• #50102 Task 1: Move apimachinery/pkg/watch.Until into client-go/tools/watch.UntilWithoutRetry #66906
• #50102 Task 2: Add UntilWithSync #66971

/hold

Release note:

watch.Until now works for long durations.

/priority important-soon
/kind bug
(bug after the main PR which is this split from)

[spiffxp]

ping, curious if any progress is being planned for this release cycle

[spiffxp]

/skip
trying to clear the Submit Queue status

[tnozicka]

/milestone v1.14

[liggitt]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, tnozicka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/client/OWNERS [liggitt]
• staging/src/k8s.io/client-go/OWNERS [liggitt]
• test/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tnozicka]

@liggitt thank you for reviewing this in detail!

[tnozicka]

moving the ResultChan close has confused unit test, fixed:

diff --git a/staging/src/k8s.io/client-go/tools/watch/retrywatcher_test.go b/staging/src/k8s.io/client-go/tools/watch/retrywatcher_test.go
index 7e09fa953e..cd57e51c52 100644
--- a/staging/src/k8s.io/client-go/tools/watch/retrywatcher_test.go
+++ b/staging/src/k8s.io/client-go/tools/watch/retrywatcher_test.go
@@ -541,8 +541,10 @@ func TestRetryWatcher(t *testing.T) {
 			// but have to tolerate some delay. Given this is best effort detection we can use short duration.
 			// It also makes sure that for 0 events the watchFunc has time to be called.
 			select {
-			case event := <-watcher.ResultChan():
-				t.Error(spew.Errorf("Unexpected event received after reading all the expected ones: %#+v", event))
+			case event, ok := <-watcher.ResultChan():
+				if ok {
+					t.Error(spew.Errorf("Unexpected event received after reading all the expected ones: %#+v", event))
+				}
 			case <-time.After(10 * time.Millisecond):
 				break
 			}

[liggitt]

/lgtm

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[nikhita]

Is this a flake? #56876 (comment)

(can't reproduce locally)

Edit: just to be clear, this is not flaking on k/k's master... https://testgrid.k8s.io/sig-release-master-blocking#integration-master is green :)

[nikhita]

cc @sttts @dims
re above ^ question, because it is a publishing-bot failure

[liggitt]

yes, looks like there are some races:

go test ./vendor/k8s.io/client-go/tools/watch -run TestRetryWatcher -count 1000 -race -v -failfast

produced at least these failures:

--- FAIL: TestRetryWatcher/survives_4_closed_watches_and_reads_4_items_for_nonconsecutive,_spread_RVs_and_skips_those_with_lower_or_equal_RV (0.03s)
    retrywatcher_test.go:554: expected 5 watcher starts, but it has started 4 times

--- FAIL: TestRetryWatcher/survives_1_closed_watch_and_reads_1_item (0.02s)
    retrywatcher_test.go:554: expected 2 watcher starts, but it has started 1 times

--- FAIL: TestRetryWatcher/survives_2_closed_watches_and_reads_2_items (0.04s)
    retrywatcher_test.go:554: expected 3 watcher starts, but it has started 2 times

[tnozicka]

I'll fix send a fix shortly, it's a race in the unit tests when counting the watch calls (the last one is theoretically optional based on timing). Looks like I've used too small -count when testing it.

[tnozicka]

fix is in #74663