Do you need to run a huge number of batch jobs on a single machine? Are you frustrated with manually launching processes in a screen(1)? Are cluster computing software such as SLURM and QSUB overkill for you?
If yes, then Conjob may be what you are looking for:
- Run batch jobs in the background on a single machine.
- Capture and log output.
- Limit number of running processes by machine load and available memory.
- Submit jobs in batches.
- Job priorities, process control, suspension, etc.
conjob [options] COMMAND [...]
Process job controller, letting a limited number of jobs run at the same
time.
Commands:
list List scheduled PIDs
schedule [COUNT] Perfom scheduling, starting/stopping processes
setprio JIDS PRIO Set a priority for a PID
run CMD Run a command and put the process in the queue
bg CMD Run a command in the background and insert to queue
Redirect the output to a file in ~/.conjob.log/
bg_many < CMDS Queue many commands at once; one per line
kill JIDS Kill jobs
tail JIDS Tail -f logfiles
less JIDS Page through logfiles
cleanup JIDS Remove completed jobs from the job list
requeue JIDS Kill and re-queue given jobs
dump_jobs [FILE] Dump a job listing to FILE or stdout
load_jobs [FILE] Load a job listing from FILE or stdin
'JIDS' can be a comma separated list of JIDs, or a shell glob pattern.
The syntax '{MIN-MAX}' matches all integers from MIN to MAX.
Prepending '/' to the pattern makes it match also completed jobs,
'@' the command lines, '%' the PIDs.
Configuration goes into ~/.conjob
:
# Max number of simultaneous processes
nprocs = 8
# Max load average
loadavg = 8.0
# Nice of processes
nice = 20
# Maximum amount of memory to hog
max_mem_percentage = 75
# Minimum free virtual memory (avoid swapping)
min_vm_free = 3e9
# Pager
pager = less -S +k