Cache *Unstructured objects

[negz]

What problem are you facing?

crossplane/cmd/crossplane/core/core.go

Line 183 in e4a4180

Unstructured: false, // this is the default to not cache unstructured objects

The Kubernetes API client we use doesn't cache *unstructured.Unstructured objects unless you ask it to. We currently don't.

A significant amount of what core Crossplane does is reconcile claims, XRs, and composed resources. All of these resources are of arbitrary types, so they're all backed by *unstructured.Unstructured.

Right now I don't think any of the composition controllers are benefiting from caching.

How could Crossplane help solve your problem?

We should investigate whether caching *Unstructured resources is useful.

There is some danger in caching *Unstructured resources. I believe it's off by default because it would be easy to accidentally start caching resources you probably don't want to. For example if Crossplane somehow took a watch on all Pods in a large cluster it might run out of memory to cache them.

I imagine the benefit would primarily be reduced load on the API server. Composition controllers might be a little more performance too, at the expense of higher memory consumption.

[negz]

I could imagine putting this behind a flag (a feature flag?) to let folks experiment.

[jsanda]

I have a custom controller written in Go that is trying manage an XRD that I have written. I created Go structs for the XRD for use in my controller. I configure things like this:

func (r *MyReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&v1alpha1.MyCustomResource{}).
		Owns(&v1alpha1.MyXRD{}).
		Complete(r)
}

If I change the XR or claim generate by my controller, it isn't receiving an event to trigger a reconciliation. Could the reason for this be due to the fact that Crossplane doesn't cache unstructured resources?

[negz]

@jsanda Is your controller running as a separate process/pod? If so, Crossplane's caching behavior won't have an effect on it.

[jsanda]

Yes, it is running its own pod. I guess the behavior I am seeing is due to something else. Thanks!

[negz]

An older, related issue: #2645

[negz]

@enesonus I remembered today that the XR controllers aren't using the "main" controller-runtime cache that I configured to cache *Unstructured resources in #5339. Each XR controller creates its own cache:

https://github.com/crossplane/crossplane-runtime/blob/v1.15.1/pkg/controller/engine.go#L211

We do this because when we wrote the code it wasn't possible to stop an informer (i.e. a cache of a specific kind of resource), so instead we just stopped the entire cache when we shut down the controller.

Though given we're not caching *Unstructured in these XR caches, I wonder whether they're doing basically nothing. 🤔

[turkenh]

How does this relate to "real-time compositions" feature?

We already initialize caches per GVK, probably only to get notified when something changes. Could it be possible to leverage those caches while getting/reading composed resources? 🤔

[negz]

@turkenh Yeah, good question. I actually opened #5651 today to try using a single cache when realtime compositions are enabled. It needs more work though. It doesn't pass E2E tests yet.

[negz]

Could it be possible to leverage those caches while getting/reading composed resources? 🤔

I think we do when realtime compositions are enabled. There's code in runtime that wraps a manager's client with one backed by the GVK routed cache.