QOS limits and flex

QOS limits and flex

With [limits] mode set (and enforce), sjp helps jobs pending only on QOSMaxCpuPerUserLimit or MaxCpuPerAccount when:

• the job fits one node's free CPUs and memory now, and
• at least min_idle_share (25%) of placeable capacity is idle; with count_pending, after counting the CPUs of jobs waiting for room (Resources, Priority) as taken.

mode = "global"

Raises both caps of the job's QOS to ceiling (2×), for everyone, for pulse_seconds (60 s).

• starts after hysteresis (5) consecutive checks, ~75 s;
• ends early below lower_below (10%) idle; then cooldown_seconds (5 min) at base;
• base caps are read from the QOS before each pulse; manual changes during a pulse are kept;
• base caps are saved in state_file during a pulse and restored after a crash or stop.

mode = "flex"

Moves held jobs to the QOS flex_qos_name (flex), highest priority first, only as many as fit now (unstarted moves count).

• not started within flex_revert_after (60 s) → moved back; after a restart, all pending flex jobs move back;
• a job is moved at most once per cooldown_seconds;
• the move is recorded in AdminComment (;qos=normal>flex);
• each array task on its own; not GPU, pinned or multi-node jobs.

Setup:

sacctmgr add qos flex
sacctmgr modify qos flex set MaxTRESPU=cpu=864 MaxTRESPA=cpu=1760   # the extra allowed
sacctmgr -i modify account root set qos+=flex                       # inherited by all

Users without access to flex are skipped and named in the log; sjp never changes associations. Give flex the same priority and usage factor as your normal QOS if fair-share should not change.

Over-cap jobs keep running

In both modes, jobs started over the cap run to completion. To make the extra use temporary, make flex preemptible:

# slurm.conf
PreemptType=preempt/qos
PreemptMode=REQUEUE

sacctmgr modify qos normal set preempt=flex