tracking issue: standby/preemptible jobs

[grondo]

From @ryanday36's list in #5165:

Preemptible jobs:
AKA 'standby' qos / queue. Allow users to submit jobs that can be killed automatically by the system instance if another job needs the resources.

In some offline discussion, it was proposed that we could add a preemptible (or similar) job submission flag for this purpose. Drawbacks to this approach:

• flags are currently not shared with the scheduler
• flags are not displayed in flux jobs output
• flags cannot be updated from the command line

Most of those can be easily overcome if a submission flag is the correct approach.

Alternate solutions include:

• a jobspec attribute. In this case an initial solution could be accomplished in the scheduler alone. If a job has the preemptible attribute and the scheduler needs its resources for a higher priority job, then the scheduler could raise a job exception.
• As noted above, "standby" is often implemented as a queue. To do this in Flux might require some work on the queues interface, since this implies overlapping queues. The benefit of using a queue is that queue limits (and user/bank access) can be applied.

[ryanday36]

I think that a submission flag would work as long as the drawbacks that you noted could be overcome. Generally we allow 'standby' jobs to be exempt from other queue limits and allow all users to access them. So, we would also want the preemptible flag could also be seen by the priority plugin so that it can not count those jobs against queue limits. I think that would provide the same benefits as the queue implementation, at least for how we use standby / preemption.

That said, there are a number of use cases that can be solved by overlapping queues (exempt / expedite, whole cluster DATs), so that could be considered a benefit of that approach. Exempt / expedite could probably all be done through accounting / the priority plugin. We should probably talk more about DATs where we want to be able to let a user run on all nodes on a cluster that we've split into multiple queues.