API Server (and clients) becomes unresponsive with too many CRDs

[ulucinar]

What problem are you facing?

As part of the ongoing Terrajet-based providers effort, we have observed that registering 100s of CRDs has some performance impacts on the cluster. Observations from two sets of experiments are described here and here. As discussed in the results of the first experiment, Kubernetes scalability thresholds document currently does not consider #-of-CRDs (per cluster) as a dimension. However, sig-api-machinery folks suggest a maximum limit of 500 CRDs. And the reason for this suggested limit is not API call latency SLOs but rather, as we also identified in our experiments, due to the latency in the OpenAPI spec publishing. As the results of the second experiment demonstrate, the marginal cost of adding a new CRD increases as more and more CRDs exist in the cluster.

Although not considered as a scalability dimension officially yet, it looks like we need to be careful for the #-of-CRDs we install in a cluster. And with Terrajet-based providers we would like to be able to ship 100s of CRDs per provider package. Currently, for the initial releases of these providers, we are including a small subset (less than 10% of the total count) of all the managed resources we can generate. We would like to be able to ship all supported resources in a provider package.

How could Crossplane help solve your problem?

As part of the v1.Provider, v1.ProviderRevision, v1.Configuration and v1.ConfigurationRevision specs, we could have a new configuration field that tells the GVKs of package objects to be installed onto the cluster. To behave backwards-compatible, if this new API is not used, all objects defined in the package manifest get installed. If the new field has a non-zero value, then it's enabled and only selected objects get installed. In order to make the UX around installing packages using the new configuration field easier, we can also add new options for the install provider and install configuration commands of the Crossplane kubectl plugin.

As described in the experiment results, when a large #-of-CRDs are registered in a relatively short period of time, the background task that prepares the OpenAPI spec to be published from the /openapi/v2 endpoint may cause a spike in the API server's CPU utilization, and this may saturate the CPU resources allocated to the API server. Similar to what controller-runtime client does, we may also consider implementing a throttling mechanism in the package revision controller to prevent this. However, because of the reasons discussed above and in the experiment results (especially the latency introduced in OpenAPI spec publishing), it looks like we will need a complementary mechanism like the one suggested above in addition to a throttling implementation in the package revision reconciler.

[negz]

Breadcrumbs for this possibly related issue: kubernetes/kubernetes#101755

[negz]

A few questions:

1. Do we think this is the same issue as kube-apiserver memory consumption during CRD creation kubernetes/kubernetes#101755?
2. If not, have we raised an upstream issue tracking what we're seeing?
3. I've previously been told we'd address the issues we were seeing by rate limiting CRD installs - not filtering them. Why are we not taking that approach?
4. I've heard out of band that we're also seeing issues with client side throttling (probably in REST mapper discovery) when there are too many CRDs. Is this tracked anywhere? Could it be covered by this issue?

I don't believe disabling API groups will be a sustainable fix long term - we need to make sure the API server can handle hundreds or thousands of CRDs. It is important that we start a conversation around this upstream ASAP.

[muvaf]

Just to add more context from our earlier offline discussions with @ulucinar . I think there are two problems we're facing:

• Having more than 500 CRDs in a cluster causes problems during installation, even break some clusters like GKE for a while, as well as in runtime by making discovery client make thousands of calls.
  • One way or another, even without Terrajet providers, it's very easy to run into this by installing more than one Crossplane provider.
  • Even though it's not suggested to go over 500, I believe we should make as much effort as we can to be able to manage that many CRDs from our side, i.e. package manager creation pooling and follow up upstream to see whether there could be a way to improve.
• Users have to install more than 500 CRDs even if they need only, say 50.
  • We have many users using provider-aws in production which has under 100 CRDs. If they wanted to use Terrajet-based provider, they would have to install 700 more CRDs and bear the cost.
  • Even if we can install that many CRDs, it's not a good UX to force all of it to be installed because there are implications of that not only during installation time but also client behaviors, like discovery client making thousands of calls to find a kind unnecessarily.

In my opinion, we will need to take action for both problems but the selective API installation is the one that we can implement with low cost, it can address both problems for the most users and we will need it even if we do the pooling. Because the first problem can always be encountered, even with all the fixes we can think of in place, and users need the freedom of choice so that they can decide how much of that problem they would like to bear depending on their need.

Then I believe we can work on creation pooling, something like list all CRDs and make sure not more than 10 are in "establishing" state, if less create the next CRD. In the long run, apiserver changes regarding OpenAPI spec publishing and possibly discovery client improvements will make things better, nevertheless, I believe we'll still need both of these mechanisms in Crossplane because in Crossplane ecosystem, having 1000s of CRDs will not be very uncommon anymore so all of these solutions will help to some extent.

---

EDIT: @negz I just saw your new comment after submitting this.

[negz]

@muvaf I'm not convinced the second part of what you've mentioned - users have to install more CRDs than they need - is a real problem. If there were no performance downsides would it have any other meaningful impacts?

Presumably the main downside would be 'clutter' in OpenAPI discovery and kubectl get crds, but that seems like a very minor concern. Especially since if we do start having folks filter the set of installed CRDs those folks are presumably going to have to remember to go update their provider as they begin to need API groups that they didn't think they would before.

If we are indeed prioritising this mostly because it's a quick win for performance while we look into other options, could we do it without making a change to the Provider API? e.g. Add the flag to provider binaries and advise folks to use a ControllerConfig to set that flag if they're hitting performance issues, until we get long term fixes in place?

[ulucinar]

Hi @negz,
Regarding the points 1 and 2 above, I believe it makes sense to bring the high CPU utilization issue into the attention of the community in addition to the memory issue. I have found this discussion, which looks related to the case that concerns us. We can also get a better understanding if sig-api-machinery and/or sig-scalability folks have any planned limits on the number of CRDs in a cluster, whether they are actually planning to define this as a scalability threshold. And if so, what order the prospective limit will be in, e.g., will it really be something close to 500 as mentioned here. It may very well turn out to be a much higher limit, as my understanding is that the real concern they have, when mentioning max 500 CRDs limit as a scalability target for GA, is the cost of the OpenAPI spec publishing. If they are planning to define any SLI, for example, for the /openapi/v2 endpoint*, it may also impose a limit on the number of CRDs in a cluster. So my question is, why do they mention a max. limit of 500 CRDs per cluster as a scalability target. I think we need some clarification around this.

I also think that at some point in time, total number of CRDs in a cluster might become an "official" scalability dimension, i.e., be explicitly mentioned in the scalability thresholds document as a dimension. But we may choose not to act proactively about this, especially because we don't know what such a limit would be and we have no observations of a negative impact on API call latencies (apart from the saturated cases).

We don't expect the high number of CRDs in a cluster to affect API server call latency SLIs, as long as we do not saturate the CPU (as it was the case in @muvaf's and @turkenh's initial experiences when they tried to install 700+ CRDs from provider-tf-aws) or the memory. But for the saturation issue, we have an alternative solution as mentioned above, i.e., implementing throttling on our side.

One issue though with implementing a throttling mechanism on the Crossplane package revision controller is that it will be available only for our reconciler and any other Kubernetes clients registering CRDs on top of the 100s of them we have registered with throttling in-place, will still be susceptible to the discussed issues.

Regarding the controller-runtime client throttling, it looks like what we have so far observed is related to API discovery. My understanding is that kubectl (v1.21.x) configures a hardcoded cache TTL of 10m here for its file system cache. And for certain cases, the discovery client cache is also explicitly invalidated, like the api-resources or the api-versions commands.

*: Here, I'm talking about an SLI not on the response times of the /openapi/v2 endpoint. But something like an SLI that captures the interval from when the API server accepts a new CRD with an OpenAPI schema, to the time the aggregated OpenAPI spec published from that endpoint contains the new CRD's schema.

[ulucinar]

If we are indeed prioritising this mostly because it's a quick win for performance while we look into other options, could we do it without making a change to the Provider API? e.g. Add the flag to provider binaries and advise folks to use a ControllerConfig to set that flag if they're hitting performance issues, until we get long term fixes in place?

We have discussed implementing a filtering mechanism not via the Provider.pkg API but maybe via the ControllerConfig API with @muvaf and @hasheddan, but my understanding is that we do not want to treat ControllerConfig as a catch-all API. That has been the reasoning behind extending the Provider.pkg API.

[negz]

I believe it makes sense to bring the high CPU utilization issue into the attention of the community in addition to the memory issue.

@ulucinar I agree - can you please raise that issue and reference it here?

We can also get a better understanding if sig-api-machinery and/or sig-scalability folks have any planned limits on the number of CRDs in a cluster, whether they are actually planning to define this as a scalability threshold.

Definitely. We should be providing input to this process. We should be advocating for the API server to be able to handle our use cases, rather than waiting for the API machinery folks to tell us what the limit will be. 😄 If it can't natively, we're going to need to fork it and that would not be a great outcome.

One issue though with implementing a throttling mechanism on the Crossplane package revision controller is that it will be available only for our reconciler and any other Kubernetes clients registering CRDs on top of the 100s of them we have registered with throttling in-place, will still be susceptible to the discussed issues.

That's a fair point - though I imagine we're a bit of an edge case in both the frequency and number of CRDs that we install. i.e. I imagine most other projects would very infrequently be adding <10 CRDs and thus hopefully not be as impacted as e.g. a Crossplane provider.

This makes me wonder how impactful the OpenAPI issue will really be based on when folks typically install CRDs. If folks are typically installing CRDs at cluster bootstrapping time this may be an inconvenience but otherwise a non-issue. If folks are installing CRDs on existing clusters that are doing real work it seems like more of a potentially user-impacting issue.

Regarding the controller-runtime client throttling, it looks like what we have so far observed is related to API discovery. My understanding is that kubectl (v1.21.x) configures a hardcoded cache TTL of 10m here for its file system cache. And for certain cases, the discovery client cache is also explicitly invalidated, like the api-resources or the api-versions commands.

https://github.com/kubernetes/kubernetes/blob/0fb71846/staging/src/k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go#L111

Ah - I was wondering where this configuration lived. Nice find. I suspect tuning the above option will have a big impact. I see that there's a WithDiscoveryBurst option, but I'm unsure whether that's exposed anywhere (e.g. as a flag). Perhaps we can make a case for simply bumping the default burst up from 100 upstream? Of course this would only affect kubectl, but perhaps if we could have kubectl and controller-runtime increase their default discovery client rate limits we'd be fixing the problem for 80% of clients?

We have discussed implementing a filtering mechanism not via the Provider.pkg API but maybe via the ControllerConfig API with @muvaf and @hasheddan, but my understanding is that we do not want to treat ControllerConfig as a catch-all API. That has been the reasoning behind extending the Provider.pkg API.

Ah, sorry I missed that discussion. It's true that we don't want ControllerConfig to become a catch-all, but that doesn't rule it out for this feature in my mind.

I would feel comfortable making this feature a fully fledged part of the v1 Provider API if we were confident it was a long term thing that many folks would be choosing to do, but I'm not convinced that's the case. It seems to me that filtering the set of installed API groups is the quickest fix we have available to a problem we're seeing, but perhaps not the best fix long term. Put otherwise, would we still make this change if we could fix the API server performance issues and client rate limiting upstream? If the answer to that question is 'no' or 'probably not', then I would feel more comfortable with this not being a feature we had to commit to at v1. Using ControllerConfig (at least for now) is a good compromise in my mind; we can unblock ourselves while only committing to this as a v1alpha1 feature.

Let me know if it would be helpful to setup a call to discuss some of this synchronously.

[muvaf]

We don't expect the high number of CRDs in a cluster to affect API server call latency SLIs, as long as we do not saturate the CPU (as it was the case in @muvaf's and @turkenh's initial experiences when they tried to install 700+ CRDs from provider-tf-aws) or the memory. But for the saturation issue, we have an alternative solution as mentioned above, i.e., implementing throttling on our side.

TBH, I had this assumption that even if we do not use all those CRDs, having them installed costs us some apiserver and also controller CPU and/or memory consumption. I tested kubectl behavior and it didn't seem like a big deal and if @ulucinar 's experiments show that this isn't the case even at scale and the only problem thousands of CRDs cause are discovery client throttling and install time saturation, I can walk back from having it to be a first-class configuration on Provider v1 API today.

I feel like we will need it for various reasons at some point, because having say 1500 CRDs while you use 100 of them will cause some problems in runtime down the road but it doesn't seem like a problem we have today. So, I'm open to implementing the pooling/throttling in CRD installation of PackageRevision controller today instead of this feature OR merge this feature as alpha in some way (discussed below) and then once we have the pooling and we don't see any problem in runtime, remove it. Leaning towards the second option.

Using ControllerConfig (at least for now) is a good compromise in my mind; we can unblock ourselves while only committing to this as a v1alpha1 feature.

@negz ControllerConfig is tailored around affecting only controller but this list affects the CRDs applied, too, so I'm not sure if it will fit there, really. It seems like the only concern about where this will live at this point is about the api versioning. I wonder what upstreams folks do when they add a new field to Pod for example. I think I've seen usage of a special annotation first, like alpha.pkg.crossplane.io/api-groups: "*.ec2.tf.aws.crossplane.io,<some regex>,<another regex>", which is something we used to have in provider-aws for endpoint configuration and I quite liked it. In some cases, feature flags but I'm not sure if that's applied to schema changes.

@hasheddan since you're more involved with upstream folks, do you know how they manage the process of adding new alpha fields on stable APIs?

[muvaf]

FWIW, we're planning to roll with a set of default CRDs to be installed in Terrajet providers so that you get to install the provider with CRDs (like ~100 to ~200 CRDs) even if you don't provide this config and there is no throttling in package manager.

[negz]

@ulucinar, @hasheddan and I met this morning to discuss this synchronously.

We're agreed that the primary motivation behind adding this feature now is to alleviate performance issues we've observed when there are hundreds of CRDs installed. In summary, and in order of importance, those issues are:

1. Processing the OpenAPI spec of many new CRDs at once can starve the API server, making it unresponsive.
2. The more CRDs exist, the longer it takes a new CRD to become 'established' (usable) due to OpenAPI processing.
3. When many CRDs exist API server clients can take longer to perform discovery.

In addition to these issues, we think it may be useful to be able to selectively install only certain APIs, but we don't yet have any concrete use cases or user feedback to support our intuition.

The worst symptom we've observed is that installing hundreds of CRDs can lock up a kind cluster. This happens due to CPU starvation - recomputing the OpenAPI spec prevents the API server handling requests. So far we've only seen this happen on kind on Intel Macs - notably M1 Macs manage to process the CRDs without noticeable performance degradation. @ulucinar is currently testing whether 'production' API servers (e.g. GKE) are affected.

The second symptom we've observed is that the time for a new CRD to become usable increases as more CRDs exist in an API server. We believe this is because the API server recomputes its entire OpenAPI spec each time a new CRD is installed, and recomputing the spec requires processing all CRDs. We've observed that it can take 20 minutes to install a large provider, then ~70 minutes to install a second large provider (where 'large' means having hundreds of CRDs).

The final symptom we've observed is that 'discovery' takes a long time when there are many CRDs installed. API server clients use a process known as discovery to determine what APIs the API server supports. This process involves 'walking the tree' of API server endpoints, which can involve hundreds of requests when hundreds of CRDs are installed. This can trigger client-side rate limiting; REST clients are often capped at (for example) 20rps with 30rps burst. Discovery typically happens at client startup, so in practice this symptom primarily means slower client startup and potentially noisy logs (clients often log when they limit themselves).

We believe there are several possible remediations:

1. What this issue proposes. Make it possible to filter the number of CRDs installed to those that are needed.
2. Limit the rate at which CRDs are installed - this will only alleviate the saturation issue.
3. Work with upstream Kubernetes to improve the performance (or at least fairness) of the API server.
4. Work with upstream Kubernetes to increase discovery rate limits (and increase them in our clients).

I'd like us to hold off on moving forward with the remediation this issue proposes until we've raised an issue to determine how receptive Kubernetes folks are to this being fixed upstream. I'm also happy for us to move forward with rate limiting CRD installs, since that remediation has less of a direct UX implication (it doesn't require users to configure anything to avoid performance issues).

I'm okay moving forward with the remediation this issue proposes if we do find that it's our only option; i.e. that upstream will not fix this issue at all or in a timely fashion and we find that we cannot tolerate the symptoms we're seeing in the mean time.