Known object count quota problems with uninitialized objects

[caesarxuchao]

Background: when is quota charged

The object count quota is charged when an object is created, no matter if it's initialized or uninitialized. This is to protect the system from having too many uninitialized objects.

Other quotas are charged when objects are initialized. They don't suffer from the problems.

Problems with replenishing the object count quota

• if the object is deleted uninitialized, the object count quota is not replenished immediately, because the quota replenishment controller doesn't observe uninitialized objects.
• the resource quota controller does a full resync every 5mins. It lists every objects in the informer and recalculate the quota usage. The informer doesn't include uninitialized objects, so the controller will wrongly replenish the object count quota for uninitialized objects.

Brain storming fixes

• Let the replenishment controller observe the uninitialized objects (e.g., Add options to sharedinformers to include uninitialized objects #51247).

• Or, for the replenishment of object counts, we build a special informer that only observes uninitialized objects. Assuming that the number of uninitialized objects is small, storing uninitialized objects outside of the shared informer causes small overhead.

  • We can add an fieldSelector or accept header to let apiserver only return uninitialized objects for list/watch.
  • Further, we can create a separate quota for uninitialized object counts.

Related issues

#51820 notes multiple problems with the quota system. Fixing them all-at-once requires a redesign, which could cost multiple quarters. I want to graduate initializers to beta independent of #51820, so I filed this issue to discuss more targeted solutions.

@kubernetes/sig-api-machinery-bugs @lavalamp @liggitt @derekwaynecarr

[caesarxuchao]

/sig api-machinery

[dixudx]

Or, for the replenishment of object counts, we build a special informer that only observes uninitialized objects. Assuming that the number of uninitialized objects is small, storing uninitialized objects outside of the shared informer causes small overhead.

• We can add an fieldSelector or accept header to let apiserver only return uninitialized objects for list/watch.
• Further, we can create a separate quota for uninitialized object counts.

+1. Agree. With fieldSelector, this will be quite concise. Currently, metadata.initializers is not supported yet. We may need to add it.

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Removed

@caesarxuchao @kubernetes/sig-api-machinery-bugs

Important: This issue was missing the status/approved-for-milestone label for more than 7 days.

Help

• Additional instructions
• Commands for setting labels

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[munnerz]

FYI, as of #54660 it is now possible to tweak the ListOptions used by a SharedInformerFactory in order to watch uninitialized objects. This would be a factory-wide tweak however, meaning all controllers in kube-controller-manager would need to handle uninitialized objects cleanly.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[nikhita]

/remove-lifecycle rotten
/lifecycle frozen

[liggitt]

the alpha initializers feature has been removed in 1.14

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

the alpha initializers feature has been removed in 1.14

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.