Reflector: always succeed when listing a collection that is too large.

[lavalamp]

Today, if a collection is too large, a LIST might not be able to finish in the 60s time out, and then it is impossible for a controller to start.

If the controller uses multiple paginated LIST requests, that is better, but it must be able to get through the entire collection before the next compaction event (every 2.5 minutes).

Fortunately this can be fixed client-side without any server changes.

1. List the first page. Immediately start a watch at the given RV. Enqueue the watch events for later processing.
2. List subsequent pages.
3. When you get out of the history window, restart your watch at the same lexical place by using the supplied continue token.
4. Track the new RV acquired with each history window reset (i.e. we must be able to determine lexically from an object's name/namespace which RV list it came from).
5. When you get all the way through the collection, apply the enqueued watch events to the locally stored objects. We can see if we should apply the watch event or throw it away by comparing it lexically with the list RV tracked as in step 4.

This is a version of #90339 that is less efficient but doesn't require any complicated server changes. You could also consider this issue to be the client-side version of #90179

[lavalamp]

/sig api-machinery

[lavalamp]

There are optimizations possible, e.g. step 5 can be done simultaneously. Also if we don't want the client to compare RV numbers, the client has to start a separate watch that corresponds with each new list start.

[fedebongio]

/triage accepted

[Jeffwan]

Today, if a collection is too large, a LIST might not be able to finish in the 60s time out,

Do you know what's the bottle neck for this slow query? Trying to see anything we can do to optimize the list query.

[lavalamp]

what's the bottle neck

Most obviously, can't transmit all the bytes from every object in the collection over the network within the 60s timeout. Many aspects of the problem could be improved, but fundamentally there's more bytes to transmit than is possible in a reasonable amount of time over a reasonable network connection.

[Jeffwan]

Anyone is working on this issue? If not, I want to give it a try. Looks like all changes can be done on the reflector side in client-go.

[lavalamp]

Feel free to give it a try--this issue is much described than implemented :)

[Jeffwan]

/assign @Jeffwan

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[lavalamp]

/reopen
/lifecycle frozen

[k8s-ci-robot]

@lavalamp: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[cici37]

/triage accepted

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[seans3]

/triage accepted