Set memory and cpu requests and limit values for all containers

[paulcwarren]

Relint is currently working on a scaling and Quality of Service (QoS) set of stories.

We are targeting 1.0 to be configured out-of-the-box as a "developer" edition aimed at those users who want to kick the tyres. As part of this, we would like to set limits on mem/cpu.

Since a "developer" edition may not be preferred by everyone, we want each component to be configurable to scale both horizontally (replicas) and vertically (mem/cpu). This will also allow users to deliver a Guaranteed QoS when required (although we are recommending that all of our pods and containers use the Burstable QoS) As part of this we would like to ask you to do several things:

1. consider which of your pods/containers you would like to expose for scaling properties for.
2. expose said configuration properties.
3. sets mem and cpu values for all containers in order to provide as much meta-data to k8s as possible so that its scheduler can do as good a job as possible. This PR is an initial attempt at setting these values, although we know you are much more likely to have insight into your components mem/cpu requirements than our guess.

If you have any questions or concerns, please let us know! Thanks!

#174462927

Co-Authored-By: Angela Chin achin@pivotal.io

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174559282

The labels on this github issue will be updated when the story is started.

[tcdowney]

I know you didn't change this, but for @cloudfoundry/cf-capi how did we choose this memory limit?

On BOSH the limit ~4GB:
https://github.com/cloudfoundry/capi-release/blob/6f8899976561e64eab8c9804b1ce772083bbf68c/jobs/cloud_controller_ng/spec#L878-L886

In production envs I've seen a well-used CF API settle in at around 2.5-3GB (occasionally bursting higher for certain requests), so I think we should make it match the capi-release limits.

[cwlbraa]

It was probably a raw multiple of the initially-deployed memory usage. We set these to keep the API from getting OOM killed during test, not based on any experiment to produce memory bloat or "realistic" workloads.

[tcdowney]

That's good context, thanks @cwlbraa.

I wonder if we should just make the very forgiving for now (either meet or exceed the BOSH defaults). The main use for them is to just be a good neighbor and avoid starving other Pods of resources, but I feel like if we're targeting a "developer edition" 1.0 they might be more frustrating than helpful if they're too strict.

[cwlbraa]

thanks for this! the main change I'd make to what's here is:

1. worker, deployment_updater, and clock should all be allowed 1000m CPU, and request 300m. They're GIL'd and sometimes do compute-intensive tasks, but are mostly idle.

are y'all interested in adding the configurability to this PR? maybe to build some empathy for how cf-for-k8s component teams add optional config parameters to cf-for-k8s?

Things that should be configurable:

1. cf-api-server instance count
2. worker instance count
3. clock memory limit (and request?)
4. deployment updater memory limit (and request?)
5. cf-api-controllers memory limit (and request?) AND CPU limit/request

Do we expect to have 4 scaling parameters exposed for each multithreaded system-singleton, vertical-scale only pod? It seems a bit much.

[njbennett]

I'm not sure if this is in-scope for this PR but if we're talking about automatic horizontal scaling...

One of the more dramatic failure modes I've seen in CF-for-VMs is the behavior of the platform when the shared internal database runs out of configured connections. If UAA can't get connections then the whole control plane becomes inaccessible. Does kubernetes have any means of taking this shared resource into account in its scaling decisions?

If not, enabling horizontal autoscaling seems like it creates an availability threat to the platform, even as it resolves another one.

[jamespollard8]

are y'all interested in adding the configurability to this PR? maybe to build some empathy for how cf-for-k8s component teams add optional config parameters to cf-for-k8s?

We've worked on a proposal for the Scaling Interface and it's under review now. So for now, we'd like to place configurability out of scope.

[jamespollard8]

worker, deployment_updater, and clock should all be allowed 1000m CPU, and request 300m. They're GIL'd and sometimes do compute-intensive tasks, but are mostly idle.

Done ✅

[jamespollard8]

I'm not sure if this is in-scope for this PR but if we're talking about automatic horizontal scaling...

One of the more dramatic failure modes I've seen in CF-for-VMs is the behavior of the platform when the shared internal database runs out of configured connections. If UAA can't get connections then the whole control plane becomes inaccessible. Does kubernetes have any means of taking this shared resource into account in its scaling decisions?

If not, enabling horizontal autoscaling seems like it creates an availability threat to the platform, even as it resolves another one.

Great thoughts, Nat. They're out of scope for this minimal PR but I'd recommend dropping these thoughts in #cf-for-k8s slack channel to get a conversation going.

FWIW, our initial take is that this is something Kubernetes should help handle. We'd hope the Horizontal Pod Autoscaler would help here