pod: set both the resource CPU request and limit

[jlebon]

Right now, we're not setting any CPU limit in the pods we schedule. This
means our workloads runs without constraints and can hog the hosts
they're running on if there are no namespace-wide default resource
limits.

Unlike with memory requests/limits, limits are enforced at the CPU
scheduling level, so IIUC it's not possible to be evicted due to
excessive CPU usage.

So let's start defaulting to a reasonable not too large value and ensure
we set both the CPU requests and limits. This also allows cgroups-aware
applications to derive the correct number of "equivalent CPUs" to use
for multi-threaded/multi-processing steps.

[ravanelli]

lgtm

[dustymabe]

hmm. I kind of liked the fact that if there were spare resources available we could "go faster". I guess there's no difference between "use resources if they are available" and "prevent other people from using resources we are currently using".

i.e. it would be nice if CPUs are idling for us to use them, but if other pods got scheduled on the node to be able to yield that extra CPU back to them.

[bgilbert]

LGTM

[jlebon]

hmm. I kind of liked the fact that if there were spare resources available we could "go faster". I guess there's no difference between "use resources if they are available" and "prevent other people from using resources we are currently using".

i.e. it would be nice if CPUs are idling for us to use them, but if other pods got scheduled on the node to be able to yield that extra CPU back to them.

Yeah, this is an intricate topic, and I'm definitely not an SME. But the base problem as I understand it is that there's no reliable way to know how much CPU we should use when there is only a soft limit (e.g. Kubernetes' request, which gets translated into cpu.shares). It feels wrong to have e.g. a compressor think it can use 64 threads. And in theory being able to use idle CPU sounds nice, but in practice this makes us noisy neighbours which can affect performance (esp. latency) in colocated workloads.

Being able to declare upfront how much CPU we need also means we'll be nicer to others and ourselves (since we schedule multiple pods across multiple namespaces and jobs) and could actually improve CI and pipeline reliability. It might lead to realizing we're not reserving enough CPU, in which case we should feel free in e.g. the pipeline to request more as necessary.

[dustymabe]

LGTM