New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load based scheduler #294

Closed
jippi opened this Issue Oct 16, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@jippi
Contributor

jippi commented Oct 16, 2015

instead of allocating resources based on cpu, memory etc - it could be beneficial to schedule jobs based on server load - since some jobs can fluctuate quite a lot between idle or super busy, making a set resource allocation of cpu hard to use on its own

@cetex

This comment has been minimized.

Show comment
Hide comment
@cetex

cetex Oct 18, 2015

This is interesting, but the tasks running may need that cpu at any time, may work if cpu-priority of tasks deployed like that were set to the lowest possible so they're always preempted when another task that has reserved the resources needs them.

Basically, if resources are reserved for one task and we oversubscribe them and launch a new task, which one is going to die when the resources run out?

Memory is an entirely different beast then cpu, what if a task is running and has reserved 80% of ram but is only using 5% at the moment, then another task is scheduled since the actual resource-usage is low for the moment and suddenly the first task needs the other 75% it reserved in the beginning?

cetex commented Oct 18, 2015

This is interesting, but the tasks running may need that cpu at any time, may work if cpu-priority of tasks deployed like that were set to the lowest possible so they're always preempted when another task that has reserved the resources needs them.

Basically, if resources are reserved for one task and we oversubscribe them and launch a new task, which one is going to die when the resources run out?

Memory is an entirely different beast then cpu, what if a task is running and has reserved 80% of ram but is only using 5% at the moment, then another task is scheduled since the actual resource-usage is low for the moment and suddenly the first task needs the other 75% it reserved in the beginning?

@jippi

This comment has been minimized.

Show comment
Hide comment
@jippi

jippi Oct 19, 2015

Contributor

one way to do it would by using a priority value, if the box run out of resources, start killing / moving low priority jobs to another box until there are free resources

it could also be as simple as a 5min avg load and based on that + threshold start killing processes

Contributor

jippi commented Oct 19, 2015

one way to do it would by using a priority value, if the box run out of resources, start killing / moving low priority jobs to another box until there are free resources

it could also be as simple as a 5min avg load and based on that + threshold start killing processes

@cetex

This comment has been minimized.

Show comment
Hide comment
@cetex

cetex Oct 19, 2015

yeah. i kinda agree. cpu is easy, just don't give it cpu-time. memory is trickier, only way out there is to kill it.

This should be done before OOM has a change to trigger as well since by then any task can be killed.

cetex commented Oct 19, 2015

yeah. i kinda agree. cpu is easy, just don't give it cpu-time. memory is trickier, only way out there is to kill it.

This should be done before OOM has a change to trigger as well since by then any task can be killed.

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Oct 19, 2015

Contributor

Hey guys,

Describing the resource requirements of a job can be difficult. But from the scheduler perspective this is a required input to do things like constraint matching and bin packing. There are heuristics to try and intelligently model the resources usages of tasks but for the time being jobs will have to supply their constraints.

However, there are things that will eventually mitigate this. One is that Nomad will be able to pre-empt lower priority jobs when placing high priority jobs and that we can do resource over-subscription. I am going to close this bug as it is not something we plan on supporting in the near future. Hopefully the things we have planned down the line will help mitigate the pains of manual resource estimation!

Thanks,
Alex

Contributor

dadgar commented Oct 19, 2015

Hey guys,

Describing the resource requirements of a job can be difficult. But from the scheduler perspective this is a required input to do things like constraint matching and bin packing. There are heuristics to try and intelligently model the resources usages of tasks but for the time being jobs will have to supply their constraints.

However, there are things that will eventually mitigate this. One is that Nomad will be able to pre-empt lower priority jobs when placing high priority jobs and that we can do resource over-subscription. I am going to close this bug as it is not something we plan on supporting in the near future. Hopefully the things we have planned down the line will help mitigate the pains of manual resource estimation!

Thanks,
Alex

@dadgar dadgar closed this Oct 19, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment