Avoid going back in time in Reflector relist (revived)

[jpbetz]

(This is a based on #67998. It applies the fix @liggitt suggested in #67998 (comment), also addresses component restart and test coverage.)

Fixes the relist part of #59848

Update reflectors to relist with resourceVersion="{newest resource version observed}", falling back to resourceVersion="" if an HTTP 410 (Gone) status code is returned by the relist request. This ensures the reflector never "goes back in time" while running.

The pager was also modified to only fallback to a full list if an "Expired" error is returned on page 2 or later. This is to avoid needless reattempts of "too old" requests. See comment in code explaining rationale.

/sig api-machinery
/kind bug
/priority important-soon
@liggitt @wojtek-t @smarterclayton

When the go-client reflector relists, the ResourceVersion list option is set to the reflector's latest synced resource version to ensure the reflector does not "go back in time" and reprocess events older than it has already processed. If the the server responds with an HTTP 410 (Gone) status code response, the relist falls back to using resourceVersion="".

[jpbetz]

/test pull-kubernetes-integration

[jennybuckley]

/cc @yliaog

[jpbetz]

This is causing real issues with reflector ListAndWatch in TestCRDDeletionCascading. Investigating.

[jpbetz]

Code updated to handle HTTP 410 (Gone) status code responses correctly and to start with an initial list using resourceVersion="". I need to update the tests still.

[jpbetz]

Test failures should all be fixed now. This is ready for review.

@smarterclayton, this should solve both of the problems that caused stale reads

[jpbetz]

Test was failing because it expected Gone error responses and we've modified the server to always return ResourceExpired. I've updated the test to expect ResourceExpired instead.

[liggitt]

Looks like the _output_tests folder accidentally got committed

edit: actually, looks like a bunch of random changes got added under vendor

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apimachinery/pkg/OWNERS [smarterclayton]
• staging/src/k8s.io/apiserver/pkg/storage/OWNERS [smarterclayton]
• staging/src/k8s.io/client-go/OWNERS [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jpbetz]

Looks like the _output_tests folder accidentally got committed

edit: actually, looks like a bunch of random changes got added under vendor

heh, git failed me. That was a from a git add of a single file. The fun part was after that:

$ git stash
[1]    133831 segmentation fault  git stash

had to back out the commit and nuke my .git to restore things.

[wojtek-t]

/lgtm

[liggitt]

/hold cancel