GC might delete objects during kube-apiserver startup

[tkashem]

What happened:

GC may delete objects during startup while the apiserver has not fully initialized yet.

One potential case where it can happen: CRDs are available through informers

• kube-apiserver is starting, its informers have not synced yet
• GC sends a query (GET a CR)
• since the informers have not synced yet, GC may get a 404
• This may potentially lead GC to delete objects

What you expected to happen:

kube-apiserver should respond with a Retry-After until it has fully initialized. GC will get a 429 with a Retry-After response header in this case.

How to reproduce it (as minimally and precisely as possible):

With an HA cluster it's hard to reproduce.

Anything else we need to know?:

Environment:

[tkashem]

/assign @tkashem

cc @sttts

[tkashem]

/sig api-machinery

[caesarxuchao]

/triage accepted

[caesarxuchao]

Hi @tkashem, I'm curious on the details of this bug.

GC ignores the CR if the CRD is not listed in the apiserver's API discovery doc (see this comment).
So is this bug happening because the CRD is listed in the API discovery doc, but the apiserver isn't ready to actually serve the CR? If that's the case, can we fix it in the apiserver by avoiding listing the CRD in the discovery doc until the server is ready to serve it?

[MikeSpreitzer]

@caesarxuchao : remember there are multiple processes that can restart here. I have not studied the GC code myself, but colleagues reported to me that they had done root cause analysis on some incorrect deletions of child objects. If I understood and recall correctly, they said to me that the failure scenario is a restart of a kube-apiserver --- long after the controller manager had started up GC --- and the parent object (as well as the falsely orphaned child) being served from an aggregated custom apiserver, and the garbage collector specifically querying for the parent in the startup time when the corresponding APIService had not yet been fully processed. This would be release 1.18 at the latest, possibly 1.16 or 1.17.

[tkashem]

So is this bug happening because the CRD is listed in the API discovery doc, but the apiserver isn't ready to actually serve the CR? If that's the case, can we fix it in the apiserver by avoiding listing the CRD in the discovery doc until the server is ready to serve it?

@caesarxuchao yes, i guess there is an inherent race between the CRD being available in discovery and serving the CR. I am not very familiar with the CRD logic, @p0lyn0mial is working on a PR #104748 to fix this race.

[MikeSpreitzer]

1. There may be some confusion here around the term "CRD". The opening comment of this issue uses that term where I think "CR" is meant. In the case my colleagues found, the problem concerned a custom resource served from an aggregated custom apiserver NOT an object defined by a CRD.
2. The scenario that troubled my colleagues involved a restart of a kube-apiserver between the time when the GC read discovery and when GC queried about a particular parent object.