Determine reasonable defaults for Spin Operator pod cpu/mem resources

[calebschoepp]

Decide on default/baseline cpu/mem configuration for the Spin Operator.

Main operator container: https://github.com/fermyon/spin-operator/blob/main/config/manager/manager.yaml#L92-L100
and kube-rbac-proxy: https://github.com/fermyon/spin-operator/blob/main/config/default/manager_auth_proxy_patch.yaml#L28-L34

I'm assuming we'll want to ensure the defaults are friendly to resource-constrained or small-footprint clusters (minikube, kind, k3d).

[calebschoepp]

CC @vdice

[bacongobbler]

Is this something critical for the release? Seems like it skirts the line between "nice to have" (we document this and let users figure it out initially) vs. "must have" (it's critical for the spin-operator to have these limits in place).

What's your take on this work item's priority?

[macolso]

From a UX perspective, resource limits and requests are table stakes in the k8s world. My vote would be to place this in the "must have."

QQ: do operators work like pods, in that if we don't define resource requests/limits they get assigned a lower QoS class? Or is that just a pod construct?

[jpflueger]

One thing to keep in mind here is that when setting resource requests/limits using the containerd shim, it drastically increases the time until the startup/readiness probes pass. So if we're setting the requests/limits I would suggest also setting defaults for probes.

[bacongobbler]

This work was completed a while ago and is already part of the chart:

spin-operator/config/chart/values.yaml

Lines 31 to 39 in f8b9ae7

	## resources represent default cpu/mem limits for the operator container.
	resources:
	# TODO: update these per https://github.com/spinkube/spin-operator/issues/21
	limits:
	cpu: 500m
	memory: 128Mi
	requests:
	cpu: 10m
	memory: 64Mi

spin-operator/charts/spin-operator/templates/deployment.yaml

Lines 52 to 53 in f8b9ae7

	resources: {{- toYaml .Values.controllerManager.manager.resources | nindent 10
	}}

[calebschoepp]

@bacongobbler I think that part of the work here was running some load tests to see if those numbers are reasonable e.g. put 500 SpinApps on a cluster and see if the controller can keep up given it's resource limits/requests.