Excessive API requests from operator with manage CRDs enabled causes high API request latency

[94DanielBrown]

Expected Behavior

CRDs are managed as expected by the operator now that they are no longer included in the yaml but we'd expect less requests to be made to the API server.

Current Behavior

We upgraded Calico to a supported version for 1.32 so we went from having the CRDs in the yaml file to the operator manage them which worked fine until we noticed API latency alerts going off in certain clusters.
We can see some API requests taking up to 5 seconds which isn't too bad but it seems to keep making them an awful lot as the total latency was very high causing our alerts to periodically trigger as a result.

Possible Solution

Could the frequency of requests be reduced for updating the CRDs so as not to cause so much overall latency to the API server, I didn't see any flags for changing this in the code currently and am happy to make a PR.
So a less agressive default and potential flag might be useful

Steps to Reproduce (for bugs)

If using EKS you can see in cloudwatch with

| sort @timestamp desc
| limit 10000
| filter objectRef.resource == "customresourcedefinitions"
| filter verb == "update"
| stats count(*) as c by responseObject.spec.group, user.username
| sort c desc

And see a high number for crd.projectcalico.org when --manage-crds=true is set

Context

We are setting manage-crds as false and having to do a more involved installation process to also get them updated in order to avoid the latency added.

Your Environment

• Calico version
  quay.io/tigera/operator:v1.38.0

• Calico dataplane (iptables, windows etc.)

• Orchestrator version (e.g. kubernetes, mesos, rkt):
  Using EKS
  Server Version: v1.32.3-eks-4096722

• Operating System and version:

• Link to your project (optional):

[caseydavenport]

Yeah we shouldn't be making that many requests. Will investigate!

[valereColleville]

Got the same issue here.
And i want to add, that generate an amount of Kubernetes Audit log where each customresourcedefinitions body retrieve is log in full, which is a huge quantity of logs as some customresourcedefinitions are not small, for 3 weeks our GB of audit logs goes from :
from 125.49 GB to 669.54 GB just with thoose call to the KubeApi server....

(EKS version 1.31 with audit log activated in AWS going to cloudwatch)

[caseydavenport]

I'll make sure we get some eyes on this. In the meantime, CRD management can be disabled in the operator - at least as a stopgap.

[sebhoss]

I think you cannot disable CRD management at the moment because of tigera/operator#3995

[valereColleville]

Hello @sebhoss ,

We used the workaround to disable our manage CRD, starting from version 0.30.0 and upgrade to 0.30.1 yesterday without issues. Tigera-operator is started and healthy with the manage-crd to false

spec:
      containers:
        - args:
            - '-manage-crds=false'
          command:
            - operator
          env:

We have just use the helm values to disabled it (let you my full config as it can be different, in my case we are using calico as a policy only level letting aws vpc-cni for the network provider):

tigera-operator:
  kubeletVolumePluginPath: None
  installation:
    variant: Calico
    cni:
      type: AmazonVPC
    kubernetesProvider: EKS
  apiServer:
    enabled: false
  manageCRDs: false

[caseydavenport]

I've got a tentative PR that aims to resolve this: tigera/operator#4000

It's been tricky for me to verify, since I have been unable to reproduce this behavior in my own environment. I don't see the tigera/operator generating repeated API calls for custom resource definitions at all.

This leads me to believe that it must be some subtle interaction with something else in your clusters that just isn't happening in mine, and thus makes it pretty hard for me to test myself!

@diranged you mentioned you were monitoring audit logs here, were you able to confirm that the only audit logs for these CRDs was coming from tigera/operator?

[diranged]

I've got a tentative PR that aims to resolve this: tigera/operator#4000

It's been tricky for me to verify, since I have been unable to reproduce this behavior in my own environment. I don't see the tigera/operator generating repeated API calls for custom resource definitions at all.

This leads me to believe that it must be some subtle interaction with something else in your clusters that just isn't happening in mine, and thus makes it pretty hard for me to test myself!

@diranged you mentioned you were monitoring audit logs here, were you able to confirm that the only audit logs for these CRDs was coming from tigera/operator?

First, we actually saw the same thing - our "testing" clusters did not exhibit this behavior, but all of our production clusters do. Interesting.

Second, yes, we were able to use the EKS audit logs to know for sure which service account was making the calls. After setting Values.manageCRDs: false in the helm chart, we saw the problem disappear:

[caseydavenport]

In case anyone does have a dev/demo cluster that is experiencing this an wants to help me confirm whether or not the fix works, I have published an image docker.io/caseydavenport/operator:10498 with the code from this pull request that I am hopeful will do the trick: tigera/operator#4000

Note that it is a master build, so very much not suitable for production. You'll want to use it with the master manifests here (but swapping the image): https://raw.githubusercontent.com/projectcalico/calico/refs/heads/master/manifests/tigera-operator.yaml

[diranged]

Hey... I'm sorry but it seems like ever since https://github.com/projectcalico/calico/releases/tag/v3.30.1 the problem went away for us..

I meant to get back in here and confirm the fix, but I lost track of it..

[caseydavenport]

Well I have no idea why that would be but glad it has gotten better!

I'm going to merge my other PR - I think it is a good thing regardless of this issue - and hopefully others can chime in to see if their experience matches.