Pod priority enforcement without killing lower priority pods

[robertlagrant]

Is your feature request related to a problem? Please describe.
I have a system which has resource-hungry processes that must have priority when they are running, and need to run straight away when launched, and are sensitive to being swapped to disk, but don't need to always need to run.

I also have some resource-hungry lower-priority processes that need to complete eventually, but aren't so time-sensitive and are not sensitive to being swapped to disk.

Describe the solution you'd like
I'd love a way to prioritise running pods without them being killed, but rather just niced down the priority tree if a higher-priority pod is launched.

Describe alternatives you've considered
It's possible to get reasonably close to this using pod priority settings, but not close enough. It works if a high-priority pod is already running, and I want to launch a low-priority pod (which will then just wait for resource availability), but having the lower-priority one run first and then launching a higher priority pod will kill the lower-priority pod.

This seems particularly relevant to k3s, where we probably have relatively fixed compute that cannot scale out, and we are trying to allocate resources on a single compute as well as possible.