Epic: Enable overcommit

[vadim2404]

Motivation

Look at the image:

Resource	Requested	Used
CPU	121	6
RAM	457	53

Let's have at least 2x overcommit (because of the maximum number of pods that can be up and running)

DoD

Implementation ideas

TODO

Tasks

Tasks

Beta Give feedback

1. ...
2. List tasks as they're created for this Epic

Other related tasks, Epics, and links

[sharnoff]

Here's some design ideas I've considered. I believe the 3rd solution is probably the best one.

1. Implement exclusively within k8s scheduler plugin

Idea: Change the scheduler plugin so that it actually allows resource usage from

Good parts:

• Only requires changes in one component; other components don't need to know about it

Bad parts:

• If usage from regular pods fills up a node, the base scheduler + cluster-autoscaler will still interpret that correctly.
• May have strange consequences with cluster-autoscaler being overly willing to scale up, because its internal scheduler simulation is less permissive than our plugin
• Setting would be global, increasing risk of volatility during rollout or revert

2. Implement within neonvm-controller/neonvm-runner

Idea: When a VM is set to a certain amount of memory and/or CPU, we actually set QEMU to use some fixed multiple of that

Good parts:

• Has no effect on scheduler or cluster-autoscaler

Bad parts:

• Regular pods unaffected
• Counterintuitive (e.g. "why does setting it to 3 CPUs actually give it 6?")
• Will have knock-on effects on autoscaler-agent's scaling; it must be made aware of this global overcommit factor (even though it's not possible to atomically deploy changes to multiple components)
• Requires restart or migration of all VMs in order to get updated neonvm-runner pods
• Setting would be global, increasing risk of volatility during rollout or revert

3. Implement via "overcommit" factor per VM

Idea: Add a new VM setting determining an "overcommit" factor that both the scheduler and cluster-autoscaler respect. Kind of a combination of ideas 1 and 2, but per-VM rather than global.

Good parts:

• Rollout is gradual, does not require restarting any VMs

Bad parts:

• Blocked on Cleanup: Scheduler should refer to memory in bytes, not memory slots #590
• Regular pods unaffected
• Requires changes in NeonVM CRD, scheduler plugin, cluster-autoscaler, and control plane
  • The changes to the CRD are particularly high risk; this is a good motivating example for Internal feature: Backwards compatibility testing for API changes #580

[stradig]

We discussed this today and decided to start increasing the overcommit factor gradually over the course of the next few weeks by .1 increments. We will observe if there are any negative effects and probably ramp up to 1.5, only. For more we will prioritize neondatabase/cloud#14114 as a prerequisite to allow for faster reaction in case of node failures.

[sharnoff]

Made an initial implementation in #905, still need to test it and self-review. Opened a handful of other PRs while I was poking around in the area:

• agent,api: Remove custom Format()s for complex types #902
• plugin: Remove unused constants #903
• plugin: Replace readClusterState with existing watch events #904

[sharnoff]

Status: #936 is waiting on my response to review, then will need to create new PR rebasing #905 onto #936.