Test CPU and memory usage of Crossplane and providers

[muvaf]

What problem are you facing?

If I want to deploy Crossplane with some resource limits, I don't have much information other than guessing what it needs.

How could Crossplane help solve your problem?

We can test Crossplane & providers for resource usage and write down what affects the usage (how much a new XRD increases CPU usage?) so that users can reliably deploy it.

[hasheddan]

xref: #314

[jbw976]

I think it's a really good idea to understand our resource usage needs, especially with memory consumption. We've seen some provider pods (e.g. provider-aws) get OOM killed recently, so I'm wondering if memory usage has increased in recent releases.

We should at least do a memory profile to understand what our memory usage/needs are, and to see if there is any low hanging fruit of obvious fixes that we should put in as part of our v1.0.0 hardening efforts to be less resource intensive.

/cc @prasek @lukeweber

[khos2ow]

@jbw976 In #1940 we're enabling metrics by tapping into metrics server controller-runtime provides. There's also a way to extend that and add customized metrics (which I don't remember off the top of my head), but at the bare minimum some good handful of metrics are available (can be found here). That being said even proceeding with #1940 doesn't enable metrics gathering from providers, but it's gonna be very simple to add that (essentially a port definition and couple of annotation is needed).

On the other hand, neither of these will cover memory profiling, which I've done previously for some other projects and I can either squeeze it in the same PR or in a new one. Something like exposing additional /debug/pprof/ endpoints.

[muvaf]

#2983 will tackle the provider part of this by providing a tool to run standardized tests against common metrics. At the same time, you can test for CPU & memory usage just like any other Kubernetes pod.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.