Repository navigation
Releases: compgenlab/batchq
Releases · compgenlab/batchq
Release list
v0.4.1
v0.4.0
Reliability and UX release focused on HPC deployments backed by networked filesystems (NFS/Lustre), plus a complete cleanup/archive story and SLURM-aware queue views.
Cleanup & archiving
batchq cleanup --delete | --archive [name]— permanently remove, or move completed jobs into a new read-only archive SQLite DB (per run), then remove them from the live DB.--older-thanage filter; automatic--vacuumafterward (--no-vacuumto skip);--vacuumalone just compacts.- Server-side bulk cleanup/archive that scales to a large backlog (hours → minutes on a slow FS), with live streamed progress and a concurrency guard against overlapping runs.
status/details/search --archivesfall back to the archive DBs for jobs no longer in the live DB.
Reliability on NFS/Lustre
- Idempotent
submit(client-assigned job id) — a filesystem latency spike that times a request out is now retried transparently instead of failing the job. - Resilient SLURM handoff —
MarkJobProxied/EndProxiedJobare idempotent and retried through transients, so a stalled DB write after an (irreversible)sbatchno longer loses the SLURM id and orphans the job. - Every generated sbatch script stamps
#SBATCH --comment="{\"batchq_id\":\"…\"}"onto the SLURM job record, so a rescue can map a SLURM job back to batchq. - Fixed an HTTP
WriteTimeoutthat cut off long streaming responses mid-flight.
SLURM scheduling
- Generic per-job scheduler passthrough (
--custom key=value, e.g. account/partition) andresource.*→--gres/--constraint/--nodelistsbatch directives.
Queue UX
batchq queue --state/-s RUNNING,QUEUED— filter to specific states (case-insensitive, comma-separated).- Job arrays collapse into a single summary row by default (
--expand-arrays, or--array-idto drill in), with anarray <done>/<total> · <status counts>breakdown. - The queue shows the effective SLURM status (PENDING/RUNNING/…) for proxied jobs and array tasks — not just
PROXYQUEUED— plus SLURM array ids.
Operational notes
- On a networked-filesystem deployment, keep
[server] read_pool_size = 1(the single-writer-connection invariant that prevents self-SQLITE_BUSYholds only at 1). - First server start after upgrading builds a new
(status, end_time)index on an existing large DB — a one-time cost.
Full changelog: v0.3.8...v0.4.0
v0.3.8
v0.3.7
- feat(priority): apply top/nice across a whole array (#49)
v0.3.6
Features
- web: combined UI + REST API gateway (
batchq web --api) — the browser UI at/(HTTP Basic,admin+[web] password) and the REST API reverse-proxied under/api/(bearer token) on one listener; the proxy forwards to the same backend the web process is a client of, preserving the single-DB-owner invariant. Missing gateway credentials are generated (crypto/rand) and persisted to the config file on TCP listeners. (#47) - slurm: store the submitted sbatch script on array tasks, so
batchq details <array-task-id>shows the exact--array=<indices>%Nscript the task was submitted with — matching single-job behavior — instead of the raw job body. Correct across drip-fed batches. (#48)
Fixes
- list: defer hydration on detail-filtered listings. (#46)
v0.3.5
v0.3.4
v0.3.4: add --since/--before submit-time filters to queue
v0.3.3
v0.3.3: normalize job names so SLURM submission can't break
v0.3.2
v0.3.2 — add `batchq backup` command (VACUUM INTO snapshot)
v0.3.1
- SLURM proxy runner array-batch gates:
- --slurm-min-array N / [slurm_runner] min_array: defer array batches
smaller than N (avoid a flurry of tiny sbatch --array submissions);
the array's final remainder is always submitted so the tail is never
stranded. Clamped to the effective job cap.
- --slurm-full-array / [slurm_runner] full_array: all-or-nothing — only
submit an array that fits one pass entirely; overrides min_array.
- Docs: "Running batchq" section in CLAUDE.md (server, clients, runners)
and the new [slurm_runner] knobs in docs/configuration.md.