Race between seeing a CRD added event and being able to select the kind

[JamesMcNee]

What happened?

We have observed a situation that looks to be a race condition whereby we have a watch on CustomResourceDefinitions and when we see an added event for a new one, we perform a list on the apiVersion + kind. When this runs on a pod however we get a 404 when performing the list operation unless we add an arbitrary delay.

The context behind this is that we use this as a check to make sure a kind exists before performing other actions, such as creating a watch. We do not expect any resources to be present immediately after adding a CRD.

What did you expect to happen?

We would expect that we would not see an added event for a custom resource before the API server is able to serve the resource.

How can we reproduce it (as minimally and precisely as possible)?

It's hard as it's a bit racey, this is only reproducible from a pod and even then, not always.

1. Create a watch for CustomResourceDefinitions
2. Add a handler so when an ADDED event is seen it will perform a get for the kind e.g. GET: /apis/GROUP/VERSION/KIND
3. Add a new custom resource

If the event is received fast enough, the handler will see a 404 for it's GET call.

Anything else we need to know?

We are running a single master, so there is only one etcd

Kubernetes version

$ kubectl version
Client Version: v1.28.8
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8-gke.1095000

Cloud provider

GKE: v1.28.8-gke.1095000

OS version

No response

Install tools

No response

Container runtime (CRI) and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[JamesMcNee]

/sig api-machinery

[karlkfi]

This seems working as intended. The CRD existing does not mean the resource is usable yet. You have to watch the CRD and wait for the status to say that it is established. This means that the CRD controller has seen the new CRD and created the necessary APIService for it. The APIService is then used by the apiserver to know where to route requests for that resource.

[JamesMcNee]

Hi @karlkfi, ah, thanks for clarifying, I was not aware of this! I will go ahead and close the issue.