Join GitHub today
load based scheduler #294
This is interesting, but the tasks running may need that cpu at any time, may work if cpu-priority of tasks deployed like that were set to the lowest possible so they're always preempted when another task that has reserved the resources needs them.
Basically, if resources are reserved for one task and we oversubscribe them and launch a new task, which one is going to die when the resources run out?
Memory is an entirely different beast then cpu, what if a task is running and has reserved 80% of ram but is only using 5% at the moment, then another task is scheduled since the actual resource-usage is low for the moment and suddenly the first task needs the other 75% it reserved in the beginning?
one way to do it would by using a priority value, if the box run out of resources, start killing / moving low priority jobs to another box until there are free resources
it could also be as simple as a 5min avg load and based on that + threshold start killing processes
Describing the resource requirements of a job can be difficult. But from the scheduler perspective this is a required input to do things like constraint matching and bin packing. There are heuristics to try and intelligently model the resources usages of tasks but for the time being jobs will have to supply their constraints.
However, there are things that will eventually mitigate this. One is that Nomad will be able to pre-empt lower priority jobs when placing high priority jobs and that we can do resource over-subscription. I am going to close this bug as it is not something we plan on supporting in the near future. Hopefully the things we have planned down the line will help mitigate the pains of manual resource estimation!