Repository navigation
Releases: vibyverse/vibe-scheduler
Release list
vibe-queue v1.0.2 - source release
Public source release of vibe-queue v1.0.2.
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers. Follow the included installation instructions. Managed Git-based installers require a clone checked out at this tag.
Desktop availability is listed separately at https://vibe-qc.com/download/.
vibe-queue v1.0.1 - 2026-10-28 - Meyer's Contract
Added
-
Preserve contained native library file aliases during qualified slot seeding, rejecting escaping, cyclic, dangling and directory aliases before copying.
-
A broker job can carry its calibrated runtime estimate, and the completion-time policy plans with it.
vq submit BROKER --estimate CALIBRATEDsends the estimate's runtime interval (the receipt's lower bound, the calibrated upper bound) asvq broker accept --runtime-seconds LOW HIGH, recorded on the intent asruntime_seconds. With[broker] policy = "completion-time", a host where the predicted execution could overrun the job's time limit or the host's walltime limit is then refused with that reason. A raw receipt or an uncalibrated runtime is refused, and--mem-from-estimate/--time-from-estimatestay refused for broker jobs. The field is written only when set, so other broker records and intent digests are unchanged; a vq without it reports a record that carries it as corrupt, so use the option only when every host reading the broker store knows it. (#159) -
vq broker shownames the figures a completion-time blocker rests on. The broker job view (vq broker show,vq broker list --jsonand the broker part ofvq status BROKER b-...) now includeswall_time_secondsandruntime_seconds(nullwhen unset), andvq broker showprintswall_time_seconds:andruntime estimate:lines when they are set. (#159) -
A gateway submission can carry its calibrated runtime estimate.
POST /v1/jobsaccepts an optionalruntime_seconds: [low, high], checked likevq broker accept --runtime-secondsand recorded on the broker intent for the completion-time policy to rank by.vq remote submit --estimate CALIBRATEDsends it from a calibrated estimate and refuses a raw receipt or an uncalibrated runtime before uploading. Relay connectors receive the field only when it is set. (#159) -
The broker can place PBS and Slurm jobs through a driver daemon on another host. A broker scheduler lane's
scheduler_drivermay now name a daemon host reached over SSH. The broker and the driver each digest the lane's definition (lane_sha256: scheduler, dialect,submit_extra, scratch roots, job/CPU/walltime caps and runtime deployments) from their own config, leaving out per-machine routing such as SSH aliases and the driver's name. A lane whose definition differs, a driver that does not drive the lane itself, or a driver whose vq reports no lane digest gives no evidence. The payload is staged in the driver's home for the keyed submit, removed once the driver answers and kept while its answer is unknown. A driver on the broker machine is unchanged. (#160) -
A runtime bundle build can run on a remote build host.
vq runtime-bundle build --host NAME -d SRC ...checks the paths, builder and description locally, stages a tar of the source tree and the description in that host's home, and runs the host's ownvq runtime-bundle build -a TAR .... The staging is removed afterwards, or kept with a clear error when SSH cannot tell whether the build was accepted; a rerun is then refused by the build record already there.vq runtime-bundle build-finish --host NAME RECORDpacks on that host and relays its answer and exit code.build -a/--archivetakes a source archive in place of-d. (#161) -
A benchmark plan variant can take its identity from a portable runtime bundle.
vq benchmark variant BUNDLE --name NAMEchecks a packed or published #161 bundle's archive against its manifest and prints a planvariantsentry. Itsruntime_digestis the artifact ID, whichvq benchmark measure --runtime-rootreports for a deployment of it.build_flags, thetoolchainanddependencieslibraries andstorage_bytescome from the bundle;--libraryand the build and deployment costs are added as given. Plan and result schemas are unchanged. (#162) -
New guides: installing vq on a workstation, a server or a PBS/Slurm cluster without root access, what changes for a job on each kind of host, registering programs (binaries, Python environments and cluster-only programs with their scheduler hooks), running vibe-qc calculations through the queue, and using vq from vibe-sketch and vibe-view. Four tutorials walk from a first job to a pinned vibe-qc calculation and a headless vibe-view figure. The starter configuration
config/public-example.tomlnow shows a daemon host, PBS and Slurm login nodes and a registered vibe-qc program. Two more pages cover running ORCA and CRYSTAL with the shippedcontrib/wrappers, on a daemon host and on a cluster, and troubleshooting by symptom: unreachable hosts, refused submissions, pending gates, terminal states and fetch failures. A further page works through arrays, dependencies, chains and reruns with a parameter sweep and its analysis, including what each job can see of another's files. Another page covers asking for CPUs, memory and time: what a job sees, how the queue orders and admits it, and which limits stop a job on Linux and on macOS, where only the time limit is enforced. A page on managing jobs covers names and tags, pausing, killing, resubmitting, archiving and deleting finished jobs, and usage totals. -
Consume product-owned analytical memory allocations with domains and scopes in estimate receipts; incomplete route models remain unknown instead of falling back to a heuristic scalar.
Preserve resolved zero occupied/virtual dimensions in receipts and calibration ranges. -
Estimate calibrations now record, per group, the observed range of each numeric feature (Nbf, Naux, occupied/virtual, roots, k-points, ranks, threads).
vq estimate --calibrationandvq estimate-evaluate --calibrationleave an estimate outside those ranges, or missing such a feature, uncalibrated and say why, instead of extrapolating the multiplier. -
Preserve explicit runtime-bound empirical memory-baseline provenance separately from analytical allocations, and reject inconsistent identities, domains or residual evidence.
-
vq estimate-evaluatereports gainroute_groups, keyed by method, algorithm and hardware class. The new--thresholdsoption reads avq.estimate-thresholds/1document and gives each route group a pass, fail, undecided or insufficient verdict from its underprediction bounds. -
Consume product-observed input digests and resolved resource features, refusing mismatched input bytes and conflicting operator overrides while preserving legacy receipt serialization.
-
vq submit HOST -- CMDwith no-dor-cnow runs CMD in an empty workspace, so a command that needs no files, such as the-- hostnamecanary, no longer needs a directory. Without--the positionals still name one input file, and-- input.pykeeps the single-file form. The refusals for a command given without--say how to write it. -
Add opt-in managed local Docker execution with immutable image/container bindings, authenticated worker accounting, durable cancellation and restart recovery. Unknown outcomes retain capacity and refuse resubmission or cleanup. Run shell payloads inside this managed route; arbitrary scripts that launch daemon-owned containers remain unsupported.
Changed
-
Reconcile the roadmap with the published 1.0.0 release and accepted article demonstrations, retaining outstanding qualification limits and identifying 1.0.x maintenance as the next target.
-
vq submit --helpnow describes how--mem-mbis enforced: by cgroups or the watchdog on Linux, by the scheduler on PBS and Slurm, and not at all on macOS.
Fixed
-
A broker log or result stream through a local scheduler driver no longer reads the caller's stdin. The driver helper that
broker_target.stream()starts for a driver on the broker machine now gets/dev/nullas stdin, like every other vq subprocess (#118), so avq logs/vq fetchor gateway result collection through such a lane cannot consume input meant for the caller. (#195) -
Portable release-check launchers. Release checks use the executing Python for the real driver helper and explicitly restore catchable signals before testing scheduler finalization, including when CI ignores SIGHUP. The 1.0 codename is registered as "Meyer's Contract" for its stable interface and compatibility boundary. (#199)
-
vq estimatenow printsJob memory: unknowninstead ofunknown byteswhen the job's memory figure is unknown. -
Accept matching method labels with display capitalization in product dry-run evidence while still rejecting conflicting methods.
-
Add explicit campaign recovery through authenticated retained managed runtimes,
including packed archive and launcher identities, without changing campaign
bytes or submission keys after an active-runtime rollover. -
vq status,vq logs,vq outputandvq eventsnow refuse a malformed job id with a one-line usage error instead of a Python traceback. -
vq statuslabels a job's parent as(resubmitted from)instead of(auto-resumed from), which was wrong forvq resubmitandvq kill --resubmit. -
Submit examples in the README and guides that ran a command without a workspace (
vq submit HOST -- CMD), which vq refuses, now give one with-d, and the refusal says to. Thevq submit --helpwatchdog note no longer says macOS falls back to a process-group walk: there the watchdog reads no process usage. -
Scheduler polls recognize standalone OpenSSH address/UNKNOWN connection-close diagnostics and use the existing bounded refusal cooldown. Authentication and host-key errors remain visible even when a jump host appends connection-close text. Scheduler submission and staging replay behavior is unchanged.
-
vq submit HOST --program NAME input.pyfor avenvprogram now runsinput.pywith that program's interpreter, on the daemon host and on a remote host, including one withremote_pythonset; it used vq's own Python orremote_pythonwhile recording the program and its revision on the job. A...
vibe-queue v1.0.0 - source release
Public source release of vibe-queue v1.0.0.
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers. Follow the included installation instructions. Managed Git-based installers require a clone checked out at this tag.
Desktop availability is listed separately at https://vibe-qc.com/download/.
vibe-queue v0.26.48 - 2026-10-22 - Raymond's Bazaar
This patch extends placement planning: fleet files built from live daemon capacity, Slurm and Torque/PBS lane shapes and recorded queue waits, GPU-aware explanations, estimate-completed requests, deployed portable runtimes as placement runtimes, an opt-in completion-time broker ordering with a read-only comparison, and candidate hosts in the console. Multi-user daemons can opt in to GPU device isolation. Daemon GPU jobs now report live and final GPU use, including across daemon restarts, the GPU sampler no longer outlives its job, remote registry migration exchanges bounded stdin and output, and the gateway again shows and cancels jobs a broker pass left held. The default capacity placement policy is unchanged; completion-time ordering is opt-in and site acceptance remains limited to its recorded scope.
Added
-
The broker can order its placements by predicted completion time (opt-in). With
[broker] policy = "completion-time"and an absoluteplacement_fleet(avq.placement-fleet/1record of the daemon targets), each placement still admits only the hosts the existing capacity gates admit: open admission, a ready queue, and room now after outstanding charges. It then orders those hosts with the placement planner, using this pass's evidence laid over the record, so a faster target is preferred.- Never extra hosts. The policy never adds a host that the default policy would refuse.
- Blockers. A host missing from the record, or rejected by the planner, becomes a
completion-time:blocker. - Unreadable record. An unreadable record holds jobs rather than falling back.
- Default. The default
policy = "capacity"leaves broker placement unchanged. (#160)
-
vq broker capacityreports GPUs. Its answer gainsgpusandfree_gpus, taken from the daemon's own admission snapshot: the devices the daemon may hand to--gpusjobs (0when it lists none) and how many are free now. A daemon older than GPU admission givesnullfor both, meaning unknown, not none. The text form showsgpus=FREE (of TOTAL). The figures are evidence for placement plans; the broker's own placement is unchanged. (#160) -
The fleet console shows a broker job's candidate hosts. Under "Broker jobs" on
/fleet/jobs, a job with a placement attempt now lists the hosts its decision ranked, best first, with the target it went to in bold.vq broker showlists them too, ascandidates. A refusal by the completion-time broker policy now shows as its owncompletion-timereason code, instead of the genericplacement-blocked. Its detail stays invq broker show. (#160) -
vq placement fleet --scheduler-shapealso reads Torque/PBS lanes. On ascheduler = "pbs"host, it reads the lane's queue withqstat -Qf, which supplies its walltime and node-count maxima. Anenabledorstartedthat is false marks the lane under maintenance. The node shape comes frompbsnodes -a, using the smallest usable node the queue routes to: by the properties itsresources_default.neednodesnames on Torque, or by each node'squeueon PBS Pro. A queue that routes to no node keeps the base node record rather than guessing. (#160) -
vq placement compareshows what the completion-time broker policy would change. Given a fleet file, it places each of this host's recent finished local jobs twice: by the broker's default capacity ordering, and by[broker] policy = "completion-time"applied to that decision. Both run over capacity snapshots built from the fleet's daemon targets. Per job it shows both choices and their blockers, plus totals of same, different, held and skipped. Jobs without a declared--mem-mbare skipped. It is read-only: it places nothing and contacts no host. (#160) -
vq placement explain --estimatecompletes a request from a calibrated estimate. Given avq estimate --calibration ... --jsondocument, it fills two request fields that are left unset:runtime_seconds, from the receipt's lower bound and the calibrated upper bound;mem_mb_per_node, from the calibrated peak in MiB, when that figure is per node: node scope, or job scope on a single-node request.
Explicit request values win. The explanation gains an
estimatesection that says per field whether it was filled, kept or not filled, and why. A raw receipt is refused. Without the option, explanations are unchanged. (#160) -
vq placement fleet --collect NAMEasks daemons for their capacity directly. Each named daemon target, which must be a configured host, is asked for itsvq broker capacity --jsonover SSH, the same way the broker asks. Only the programs in its base record are checked. Its figures are dated by this host's clock less the daemon's snapshot age. A daemon that gives no answer is recorded as unreachable now, with nothing free, and the build continues. Collecting a scheduler lane, an unknown or unconfigured target, or a target that also has--capacityis refused before any host is contacted. (#160) -
vq placement fleetadds live daemon capacity to a fleet file. Runvq placement fleet BASE --capacity NAME=ANSWER.json ... [-o OUT], giving it your basevq.placement-fleet/1record and each daemon'svq broker capacity --jsonanswer. For each answered daemon it sets:- when the daemon was observed and whether it is reachable;
- its free CPUs, memory and GPUs, only when its admission is open and its queue ready;
- its GPUs per node.
It also removes programs the daemon reports unavailable. Scheduler lanes and daemons without an answer keep their base record. Malformed or misdirected answers refuse the build, and the base file is never overwritten. It contacts no host. (#160)
-
Placement explanations can require GPUs. A
vq placement explainrequest may declaregpus_per_node, whole GPUs per node, which for a single-node vq job is its--gpus. Targets recordnode.gpus, where0means none and a missing value is unknown, and a daemon may recordfree.gpus, its devices not assigned now. A target with too few GPUs, or an unknown count, is rejected with the reason. A daemon counts as fitting its free capacity now only when enough GPUs are free. Requests without GPUs are explained exactly as before. (#160) -
Scheduler lanes' queue waits can come from recorded history. The driver now records when the scheduler accepted each job (
scheduler_submitted_at) and when its poll first saw the job running (scheduler_started_at).vq placement fleet --queue-history NAME[=TARGET]sets a scheduler lane'squeue_wait_secondsfrom the waits that this host's recent jobs on that target recorded. It uses the nearest-rank 10th to 90th percentile of up to--history-jobsjobs from the last--history-days. With fewer than--history-minwaits, the lane keeps its base record. (#160) -
vq placement fleet --scheduler-shape NAME[=HOST]reads a Slurm lane's shape. It asks the lane's Slurm host, read-only, forsinfoon the lane's partition, itsqueue. From the answer it records the smallest node group's CPUs, memory and GPUs, the tightest walltime limit (or unlimited), and a finite job-size limit asmax_nodes. A host that gives no usable answer leaves the lane unreachable. A lane without a queue, a host that is not a configured Slurm host, or a non-scheduler lane is refused before any host is contacted. (#160) -
A running daemon GPU job shows its GPUs now. Each daemon tick reads the end of a running GPU job's local
nvidia-smisamples, only when the file has changed and never through a link. It keeps the newest complete sample on the job, as the scheduler poll does for a cluster GPU job.vq statusthen shows itsgpu now:line, and the console shows its live GPU figures. Reading is telemetry only: a missing or invalid file never affects the job. (#168) -
A finished daemon GPU job reports what its devices did. A job that a daemon gave GPU devices to (
--gpu-devices) is now sampled by the samenvidia-smisampler as a scheduler GPU job, limited to its own devices withnvidia-smi -i. The sampler runs outside the resource collector, so the job's resource receipt still measures only its command. When the job finishes,vq statusshows itsgpu use:line andvq status --jsonitsgpu_usage. In single-user mode the daemon also writes the summary that listings and the fleet table read; a multi-user daemon leaves the user's workspace alone, andvq statussummarises the samples when read. (#168) -
A daemon's deployed portable runtimes can feed its placement record.
vq runtime-bundle generations --root ROOT [--json]lists a deployment root's generations, read-only, with each one's program and whether it is verified.vq placement fleet --runtime-root NAME=ROOTreads that listing from daemon target NAME over SSH. It records each program's verified generations as the target's runtimes, current one first, so a request pinned to a--runtime-artifactis planned only where that generation is deployed. A target that gives no listing keeps its base record. (#160, #161) -
A multi-user daemon can isolate GPUs between jobs (opt-in). With
[daemon] gpu_isolation = true, every job's system scope getsDevicePolicy=closed. A GPU job may then open only its own/dev/nvidiaNand the shared NVIDIA control nodes, and a job without GPUs may open none. Each job may also open the standard pseudo-devices and whatever[daemon] gpu_isolation_allowlists:/dev/...nodes orchar-/block-classes, for examplechar-infiniband_verbsfor MPI. systemd can only allow, not deny, so other devices a job needs must be listed. The daemon refuses the setting when it is single-user, because a user manager cannot enforce device policy, or whengpu_devicesholds non-numeric IDs. (#168)
Fixed
- A daemon GPU job keeps its GPU use across a daemon restart. A job that a restarted daemon re-attaches now gets its live
gpu now:sample. When it finishes,...
vibe-queue v0.26.47 - 2026-10-21 - Raymond's Bazaar
This patch adds calibrated memory and runtime estimates with explicit opt-in submission limits, queue-constrained placement, signed portable runtime bundles and protected generation pruning. It improves direct-host migration and retained scheduler-runtime admission, and adds GPU admission and usage reporting. Site-specific functionality remains limited to its recorded acceptance; candidate CI does not provision new GPU capacity or official platform channels.
Added
-
A calibration can be applied to the estimate a job is submitted with.
vq estimate --calibration CALIBRATION --hardware-class LABEL --jsonprints a newvq.calibrated-estimate/1. It wraps the unchangedvq.resource-estimate/1receipt with its digest, the calibration's id, method and training set, the matching group's multiplier (and whether it is a lower bound), and the calibrated peak (raw peak times multiplier, rounded up). When nothing applies it isuncalibratedwith the reason, and the text output adds aCalibrated memory:line.vq submit --estimateaccepts either document. A calibrated one is checked, its receipt is checked against the payload as before and recorded asresource_estimate, and the calibration is recorded in a new JobSpec field,resource_estimate_calibration.vq statusshows the calibrated figure, and resubmission, reboot recovery and the idempotency intent keep it. Raw receipts are unchanged and nothing sets resources or placement from either figure. (#159) -
Peak-memory calibration, fitted on a training set and judged on held-out cases.
vq estimate-calibrate TRAINING_SET -o CALIBRATION [--min-cases N]writes avq.estimate-calibration/1document, with one multiplier per estimator, program version, method, algorithm, memory scope and hardware class. Each multiplier is the worst observed/predicted peak ratio among decided training cases, never below 1. Those are exact ratios where the measurement compares (a rank-scope estimate or a single process) and lower bounds otherwise, including OOM kills at a comparable limit. A lower-bound worst case is flagged, and a group with fewer than--min-cases(default 5) decided cases stays uncalibrated.vq estimate-evaluate HELD_OUT --calibration CALIBRATIONadds, beside the raw verdicts, each case's calibrated verdict and the calibrated underprediction bounds. It refuses the training set and any of its cases. Without--calibrationthe report is unchanged, and no receipt, admission or placement uses a calibration. (#159) -
vq submit --mem-from-estimatetakes a job's memory limit from its calibrated estimate. With a calibrated, job-scopevq.calibrated-estimate/1as--estimate, the job'smem_mbbecomes the calibrated peak rounded up to MiB, exactly as if given with--mem-mb. It is derived before placement, sovq submit autofits by it. It is refused with an explicit--mem-mb(which stays authoritative), a raw or uncalibrated estimate, or another memory scope. The job recordsmem_mb_source: "calibrated-estimate", checked again where it is created and shown byvq statusas amem_mb:line. Resubmission keeps it unless the limit is overridden, and reboot recovery keeps it. Without the flag no estimate changes a limit. (#159) -
Estimate calibrations also fit runtime upper bounds.
vq estimate-calibratewritesruntime_groupsbeside the peak-memory groups, keyed the same way but without the memory scope. Each group's multiplier is the worst observed wall time over the estimate's runtime upper bound, never below 1. A completed run counts exactly; a run stopped at its time limit, or one that passed the bound before it was stopped, counts as a lower bound.vq estimate-evaluate --calibrationadds acalibrated_runtimeverdict to each held-out case, calibrated coverage bounds to every summary, and aCALIBRATED COVEREDcolumn. Only the upper bound is scaled, and applying a calibration to a submitted estimate still uses only its memory groups. Calibrations fitted earlier load with no runtime groups. (#159) -
A job's wall-time limit can come from a calibrated estimate.
vq estimate --calibrationnow also applies the matching runtime group. Itsvq.calibrated-estimate/1records the calibrated runtime upper bound (calibrated_runtime_upper_seconds), or, when the runtime cannot be calibrated, why not.vq submit --estimate calibrated.json --time-from-estimatesets the job's wall-time limit to that bound, rounded up to whole minutes, exactly as if it had been given with--time-limit. It is refused with--time-limitor--wall-time-seconds, and refused for a raw estimate or an uncalibrated runtime. The job recordswall_time_source: "calibrated-estimate", which is checked where the job is created, shown byvq status, and kept on resubmission (unless the limit is overridden) and reboot recovery. Without the flag, no estimate changes a limit. (#159) -
Placement explanations honor an explicit scheduler queue or partition.
Requests may declarequeue, which must match a scheduler lane's recorded
PBS queue or Slurm partition exactly before completion-time ranking. Unknown
or different queues and daemon targets are rejected with reasons; no matching
eligible lane produces a hold. Text and JSON preserve the constraint, and
requests that omit it keep their existing behavior. (#160) -
Old portable runtime generations can be pruned.
vq runtime-bundle prune --root ROOT [--keep ID ...] [--apply]lists, and with--applyremoves, the deployed generations nothing keeps. A generation is kept if it is the current or previous selection, if a queued or running job in this host's queue (multi-user included) is bound to it, or if it is named with--keep. The default is a read-only plan with sizes.--applytakes the deployment lock, rereads the bindings, and renames each generation out of place before removing it without following links, so a racing job fails closed. It also finishes an interrupted prune and leaves non-generation entries alone. The artifact store keeps the bundle, sodeployrestores a pruned generation. Generations that scheduler jobs bind from another driver's queue are not visible to a prune on the cluster; protect them with--keep. (#161) -
Portable runtime bundles can be trusted by their publisher's OpenSSH signature. A
vq.runtime-trust/2policy can list publisher public keys undersigners, as an alternative to pinning every artifact digest.- Signing.
vq runtime-bundle sign BUNDLE --key KEYchecks the archive, then signs the canonical manifest withssh-keygen -Y sign, writingmanifest.json.sig. The key may be a private key file or a key held in ssh-agent. - Trust. Publish, deploy and HTTPS/SSH fetch trust an unpinned manifest only when
ssh-keygen -Y verify(OpenSSH 8.1+) confirms its signature with a key listed for its publisher. A deploy trusted that way records the key's fingerprint assigner. - Fetch. An unpinned fetch verifies the signature before it requests the archive. A pinned fetch requests no signature.
- Compatibility. Schema 1 policies are unchanged, and earlier versions refuse schema 2 policies. (#161)
- Signing.
-
Daemon hosts can admit GPU jobs. Start a daemon with
vq daemon run --gpu-devices 0,1(or[daemon] gpu_devices = ["0", "1"]) to list the GPUs it may hand out.- Admission. A local
vq submit --gpus Njob waits until N of those devices are free, then gets the lowest free ones. It sees them inCUDA_VISIBLE_DEVICESandVQ_GPU_DEVICES. - Records. The devices are recorded on the job as
gpu_devices, so a restarted daemon keeps them busy. Killed jobs whose processes survive hold theirs too. - Submit.
--gpusto a daemon host is accepted only when that host's running daemon lists enough devices. An older or stopped daemon refuses it, rather than oversubscribe. Brokered and auto-placed submits still refuse it. - Limits. Devices are assigned, not isolated, and a daemon job's GPU use is not sampled. Scheduler targets are unchanged. (#168)
- Admission. A local
-
A running scheduler GPU job shows its GPUs now. The driver's scheduler poll reads the end of each running GPU job's
nvidia-smisamples, with one boundedtailper 32 jobs. It keeps the newest complete sample on the job asscheduler_gpu_live.vq statusshows agpu now:line and the console's job detail page a gpu now section, both with the sample's node-clock time. They follow the same coverage rule as the finished summary and give device-level utilisation and device memory, plus the memory of the job's own processes. Only running jobs that requested GPUs are read, so a scheduler target without one makes no extra remote call, and a failed read never fails the poll.vq status --jsongains the nullablescheduler_gpu_livekey. (#168) -
GPU processes of a Slurm job's
srunsteps count as the job's. Ansrunstep's tasks are slurmstepd's children, not the job script's, so the GPU sampler did not attribute their memory to the job. Now a GPU process whose/proc/<pid>/cgroupnames this job,job_<SLURM_JOB_ID>matched exactly (cgroup v1 or v2), is attributed as well, and counted once. Without Slurm's cgroup containment, or on Torque/PBS, only descendants of the job script are attributed, as before. (#168) -
The fleet jobs table shows a GPU job's measured GPU use. Below its
+N gpuallocation, a running GPU job shows the mean utilisation over its listed devices and their memory in use out of their total. These come from its newestnvidia-smisample, with the sample's age, and are markedstalepast 10 minutes. A finished job shows its mean utilisation and the largest device's peak memory, markedpeak. Both are device-level, follow the detail page's coverage rule and are never added into the allocation tile. The owning host'svq queue --jsonrows carry a finished GPU job'sgpu_usage, and scheduler projections keep the driver's reading. ...
vibe-queue v0.26.46 - 2026-10-20 - Raymond's Bazaar
This patch adds optional per-target scheduler job caps and submission pacing, binds scheduler jobs and campaigns to verified runtime generations, and retains submission-time estimates with stable directory-input fingerprints. Site-specific limits remain operator configuration. Candidate CI qualifies the shipped source; configured remote-site acceptance and official platform delivery remain separate.
Added
-
A job can keep the estimate it was submitted with, and finished jobs become an evaluation set.
vq submit --estimate RECEIPTattaches avq estimate --jsonreceipt to the job asresource_estimate. The submission is refused unless every input file the receipt fingerprinted is in the payload by content: the input file, a regular file under--dir, or a--compressedarchive member. The estimate changes no resource, placement or admission.vq statusshows anestimate:line, and resubmission and reboot recovery keep the receipt. A remote host checks it again, and a broker refuses the option.vq estimate-collect OUT_DIR --hardware-class LABELwrites avq.estimate-evaluation-set/1from the local queue's finished jobs that carry an estimate, pairing each receipt with the job'sresource-usage.json, outcome and deciding limit, forvq estimate-evaluate. (#159) -
vq estimate --inputaccepts a directory. A directory stands for every regular file under it, in relative-path order, each opened without following a link. Links and special files are left out and counted in a warning, because avq submit -dpayload carries a link as a link. The existing bounds apply after expansion (64 files of 64 MiB each), a walk visits at most 10,000 entries, and an empty or partly unreadable directory is an error. An estimate of a directory is therefore accepted byvq submit --estimate RECEIPT -d DIRfor that directory, and refused once one of its files changes. Receipts for file inputs are unchanged. (#159) -
A scheduler job can be bound to a portable runtime generation, verified on its node.
vq submit TARGET --runtime-root ROOT --runtime-artifact IDrecords the named generation for a scheduler target, where ROOT is thevq runtime-bundle deploytarget on the cluster; the identity is required because the driver does not read the cluster's selection. Before the command runs, the job script checks the generation on the compute node. Nothing may be a link, the SHA256 of the canonicalmanifest.jsonmust be the identity, the deployment receipt must record it verified, and the payload must not be group or world writable. The node needssha256sumorshasum. If the check fails, the command never starts and the job fails with exit code 126, giving the reason instderr.log; it never runs another generation. The job getsVQ_RUNTIME_PAYLOADandVQ_RUNTIME_ARTIFACTas on daemon hosts. Unbound jobs' scripts are unchanged. (#161) -
A campaign node can bind its declared runtime generation on a scheduler target.
vq campaign runno longer refuses a node that declaresruntime_digestandruntime_rooton a configured scheduler target. It submits with--runtime-root/--runtime-artifactas on a daemon host, and the job script verifies the generation on the compute node before the command (#161). Preflight checks the root by the rulevq submitapplies to a cluster path, a plain absolute path, before anything is submitted. A scheduler job's start is recorded before its node checks the generation, so the controller takes the generation as run only once the job completed with exit code 0. While the job runs, the attempt records no generation, and the missing record is not taken as a mismatch. A job whose node refused the generation exits 126 without starting its command. It becomes a failed attempt with no generation recorded and nothing published, and the controller logs the node's reason. A failed scheduler job records no generation even if its command ran. (#165) -
Scheduler targets can set
scheduler_max_jobsand
scheduler_submit_interval_secondsto bound this driver's outstanding
allocations and pace submission attempts. Canonical targets and their aliases
share these limits without holding unrelated targets or local work. Site-wide
account quotas remain the scheduler's authority.
Changed
- Track the bounded v0.27 feature release with a GitLab milestone and acceptance issue, require weekly roadmap reconciliation, and distinguish v0.26.45 delivery from newer main changes.
Fixed
-
Document HTTPS gateway and outbound relay access alongside SSH in introductory and integration guides. Reconcile idempotency and backpressure wording with the separate gateway contract. Reconcile roadmap release status with v0.26.45.
-
Reconcile the roadmap with current development. Distinguish v0.26.44 from main-only increments, replace obsolete closed-issue gates with the eight active feature boundaries, and frame v0.27 as a bounded feature release. Correct related gateway, campaign and historical planning references without claiming pending live qualification has passed.
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.
Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/
vibe-queue v0.26.45 - 2026-10-19 - Raymond's Bazaar
This patch adds outbound relay cancellation, logs, verified result transfer and backpressure; binds daemon-host jobs and campaigns to portable runtime generations; improves scheduler GPU requests and allocation visibility; and includes an Apple Silicon app builder. Candidate CI qualifies the shipped source. Real site gateway and packaged-app acceptance remain separate from the source tests.
Added
-
Add an Apple Silicon macOS app, DMG and ZIP builder containing Python, the
Queue CLI and web console dependencies. Installed commands work without an
external Python environment and keep configuration and jobs outside the app. -
A daemon-host job can be bound to a deployed portable runtime generation. Use
vq submit HOST --runtime-root /abs/runtime [--runtime-artifact SHA256]. At submit, the execution host resolves the deployment's current selection, or the named generation, and accepts it only asvq runtime-bundle deployleft it. That means a matching manifest identity, a verification receipt that says it verified, and an operator-owned, non-writable payload. The job records it asruntime_bundle. Before dispatch the daemon checks that same generation again and runs the job withVQ_RUNTIME_PAYLOADandVQ_RUNTIME_ARTIFACT. A later deploy or rollback changes only later submissions, and a generation that no longer verifies fails the job rather than letting it run on another.vq statusshows aruntime:line, the console's software column names the generation,vq resubmitkeeps it, and it is part of an--idempotency-keysubmission's identity. Scheduler targets refuse the option for now. Without it, submissions, their hashes and the remote wire are unchanged. (#161) -
The relay connector (#164, slice 5) reports its free spool (
[connector] spool_mb, default 4096). Work that would not fit stays at the gateway, and clients seeconnector_backpressure. The new connection-loss acceptance harness covers an outbound-only coordinator and interruptions at every stage with restarts. -
Cancellation through a relay gateway (#164, slice 2). A job no connector has claimed is cancelled at the gateway and never delivered. For an admitted job the cancel is a durable request: the connector kills it on the broker host and acknowledges
stoppedornot-stopped, and clients seerequesteduntil then. -
Relay mode for the submission gateway (#164, slice 1). With
[gateway] mode = "relay"the reachable gateway holds admitted jobs (IDsg-…) and needs no broker, SSH keys or fleet inventory.vq connector runon the broker host makes only outbound HTTPS requests: it claims the project's jobs with fencing epochs, admits each to its broker under the same scoped idempotency key and reports their state. Connectors enroll with a one-use token (vq gateway connector enroll,vq connector enroll), rotate online and are revoked by the operator, one per project. -
Logs through a relay gateway (#164, slice 3). A client's logs request asks the connector for that job's output for 10 minutes. The connector reads it on the broker host and pushes new bytes by offset into a bounded per-job buffer (16 MiB per stream). Clients page through what has arrived, with its time.
-
Results through a relay gateway (#164, slice 4). A client's results request asks the connector to collect the finished job on the broker host. The connector pushes each file by offset within the remaining storage quota, and the gateway publishes them only after every file matches the manifest's size and SHA-256. Downloads then work as in local mode.
-
Campaign attempts record the runtime their job actually ran with. Until a declared
runtime_digestcan be bound to a submission, the controller archives what each job's queue resolved at dispatch: the node's program and version, the git SHA authenticated immediately before dispatch, the immutable artifact identity (such as a scheduler deployment's per-SHA launcher) and the runtime slot. It copies these from the job'svq status --jsoninto the attempt's journal record asruntime, and every published output's manifest carries them asexecuted_runtime.vq campaign statuslists them under Ran with and in its JSON, and/fleet/campaignsshows them beside each attempt. This is a record, not a binding: a declaredruntime_digestis still refused, and a job without a registered program, or not yet dispatched, records nothing. Journals and manifests without a recorded runtime are written and hashed exactly as before. Ones that record a runtime are refused by older controllers. (#165) -
A campaign node's
runtime_digestnow binds its job to that portable runtime generation. The node namesruntime_root, thevq runtime-bundle deploytarget on its daemon host (a root needs a digest), and the controller submits with--runtime-rootand--runtime-artifact. The queue then binds exactly that generation, whatever the deployment currently selects, or refuses the job. Once the job has started, the controller compares the generation in the job's own record with the declaration. A job that ran anything else, or that records none, is reported, and nothing it produced is published. The attempt's executed runtime and each output's manifest carry the generation, and the manifest'sruntime_digestis now the enforced one. A digest without a root, or one on a scheduler target, is still refused before anything is submitted. Campaign files that declare no root keep their digest, so existing journals stay valid. (#165) -
The fleet console's refresh cost is measured and documented.
scripts/measure_fleet_console.pyruns the console's real sweep and jobs table against a synthetic fleet. Only the SSH transport is replaced: it answers with the JSON the real commands print, after a simulated round trip. The script reports remote calls per sweep, sweep durations, table render times and sizes, the machine and its load.docs/fleet_console.mdgains a "What a refresh costs" section with three named workloads. They show one queue listing and one overview per host per refresh, plus onevq topfor a host running a direct job, independent of the number of jobs, and no remote call while pages render. The figures are labelled as those workloads on that machine. (#168) -
The fleet console shows which nodes a distributed scheduler job holds. The execution host string the scheduler reports is now read node by node. That covers PBS/Torque
exec_host(core ranges such asnode07/0-7+node08/0-7, Torque's one-entry-per-core form, and PBS Pro'snode07/0*8) and Slurm node lists (node[01-04,07],gpu[1-2]-ib). The jobs table's nodes column readsN nodesfor a multi-node job, with each node, and for PBS/Torque its allocated cores, on hover. The job drill-down gains a "nodes" section listing them. Slurm's node list names nodes only, so their cores are shown as not reported rather than split evenly. Expansion is bounded, and a string that does not parse is shown as reported, as before. This is the scheduler's allocation, not a per-node measurement. (#168) -
Scheduler jobs can request whole GPUs with
vq submit --gpus N. The count is a job total, recorded on the job asgpus. Torque renders it as#PBS -l nodes=1:ppn=CPUS:gpus=N(vq's Torque jobs are single-node) and Slurm as#SBATCH --gpus=N, so the scheduler allocates and enforces it.vq statusshows agpus:line, queue tables add aGPUScolumn when a listed job has one, the job seesVQ_GPUS, andvq resubmitkeeps it. In the fleet jobs table the cpus column shows+N gpu, and agpus allocatedtile sums what running jobs requested. A daemon host has no GPU admission or device assignment, so--gpusis refused there rather than ignored. PBS Pro'sngpusand GPU types stay with the host'ssubmit_extra, and GPU use is not measured yet. Without--gpus, submissions, their idempotency hashes and the remote wire are unchanged. (#168)
Fixed
-
Detect lost benchmark measurements before resuming. The runner now
requires a results row for each run the journal marks recorded. A missing
results file or missing row stops with an actionable error, preserving saved
files and job identities instead of reporting success with lost samples. (#162) -
Reject benchmark results from another plan before resuming. The runner
checks the saved results' plan digest before contacting the queue or writing
a checkpoint, preventing further jobs from being submitted under mismatched
results. The error asks for the campaign's matching results. (#162) -
Use the verified runtime launcher for QVF scheduler submissions. The managed
Python launcher is already verified in the target runtime pin, without requiring
an extra site wrapper hook that
could conflict with the existing interpreter mapping. Missing or wrong-target
launcher evidence and conflicting legacy hooks are refused before staging;
runtime SHA checks remain in force. (#188)
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.
Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/
vibe-queue v0.26.44 - 2026-10-18 - Raymond's Bazaar
Validation scope
This patch updates QVF protocol checks to toolkit 0.5.0, adds gateway client and placement constraints, and strengthens scheduler start fencing, benchmark isolation and fleet filters. The exact candidate runs the full Queue suite, lint and documentation gates. Broader placement, dashboard and benchmark qualification and a real TLS gateway calculation remain pending; historical scheduler attempts without start tokens stay protected.
Added
-
Require data access before ranking placement targets. Read-only placement
plans can declare required access labels and match them against each target's
recorded capabilities. Unknown or missing access rejects that target, with all
reasons retained in text and JSON explanations. Requests without access
requirements keep their existing behavior. (#160) -
Require declared license eligibility in placement plans. Requests can
declarerequired_licenses, and targets can recordlicenses. Every required
label must match before completion-time ranking; missing or unknown eligibility
rejects a target with an explanation. Text and JSON retain the requirements,
and unconstrained requests keep their existing behavior. (#160) -
vq remote submit|status|logs|fetch|cancelis a client for the submission gateway (#163, slice 5). It reads the URL and key from an owner-only credentials file, uploads in resumable chunks and retries safely (a dropped response never creates a second job). It follows logs and fetches results, checking each file's SHA-256. -
vq gateway(#163, slice 1) adds the identity layer of the HTTPS submission gateway on the broker host:- the opt-in
[gateway]and[gateway.projects.NAME]config; vq gateway key create|list|revoke|rotate, with per-project API keys shown once and stored only as an HMAC under a private pepper;vq gateway serve, which refuses a plaintext non-loopback bind, plus/v1/healthzand/v1/whoamiand an owner-only audit trail.
Submission routes follow in later slices. The gateway is for a trusted group only (
SECURITY.md). - the opt-in
-
The submission gateway (#163, slice 3) pages an owned job's stdout or stderr by byte offset (
GET /v1/jobs/{id}/logs,readscope). It also cancels it (POST /v1/jobs/{id}/cancel,cancelscope): at the broker, and on its host once placed. A transfer in doubt is reported as not stopped. -
The submission gateway (#163, slice 4) collects a finished job's workspace once (
GET /v1/jobs/{id}/results,downloadscope). It keeps only regular files under safe paths and returns a SHA-256 manifest, and serves files by exact path with their digest. Results count against the storage quota, and placed jobs count as active until collected.vq gateway pruneremoves results and uploads past their project's retention. -
The submission gateway (#163, slice 2) accepts resumable, digest-checked tar uploads (
/v1/uploads) and submissions (POST /v1/jobs) through the broker's keyed intake. Submissions are idempotent per client key, checked against project policy (programs, CPU, memory and wall-time caps, waiting-job limit) and owned by the submitting principal. Jobs are listed and shown withGET /v1/jobsandGET /v1/jobs/{id}, without the command, payload or fleet host. -
vq admin reconcile-submit JOBID --fence-unstarted(preview, then--apply --plan-digest) settles a scheduler attempt leftsubmit_outcome_unknownwithout guessing whether it was admitted. Every new single-job attempt carries a daemon-issued start token. Its job script first claims_vq/attempt-<token>.claimexclusively and exits without running if the attempt was fenced. The fenced job endsFAILEDwith scheduler statesubmit_fenced_unstarted, is never retried automatically, and nothing is submitted or cancelled. Attempts from before this change cannot be fenced (#179).
Changed
- Check the QVF queue protocol against toolkit 0.5.0. Its format-version-1 schema and existing job lifecycle remain unchanged.
Fixed
-
Submit fresh jobs for a new benchmark campaign. Fresh run directories get
a persisted campaign identity, so running the same plan again cannot reuse
an earlier campaign's jobs. Restarts and directory moves retain their keys,
and older journals preserve their original keys for safe recovery. (#162) -
Keep benchmark runs isolated after an unresolved error. A run with a
problem continues to block later runs on its node class across runner
passes. An unknown job state can no longer let another benchmark start
alongside a job that may still be running. Independent classes continue,
and restarting checks the existing accepted job before proceeding. (#162) -
Filter fleet calculations by their execution state. Waiting and unconfirmed
scheduler jobs no longer appear under the running filter. The state column,
sorting, refresh and jobs API use the cached effective state, with the raw vq
lifecycle retained for diagnosis. Last-known states remain selectable, and
selections survive disappearing jobs. (#168) -
Keep fleet program filters available during host outages. The program
selector includes last-known jobs and retains the selected program when
matching jobs disappear, preventing later filter submissions from silently
dropping that selection. (#168)
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.
Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/
vibe-queue v0.26.43 - 2026-10-17 - Raymond's Bazaar
Validation scope
This patch collects placement account and MPI-layout handling, conservative
command/OOM memory evidence, checkpointed benchmark recovery and host-scoped
failed-runtime retry checks. Candidate qualification runs the full Queue suite,
lint and documentation gates. Inclusion records source delivery; it does not
claim broader feature completion or scientific acceptance for queued work.
Added
- Honor MPI rank and thread layouts in placement explanations. Requests
can declarempi_ranksandthreads_per_rankalongside their total CPU
count. Whole ranks must fit each node before targets are ranked, so a small
per-node CPU average cannot hide an oversized rank layout. Text and JSON
preserve the layout, while requests without one keep their existing behavior.
(#160)
Fixed
-
Estimate evaluation checks command outcomes in usage receipts. A case
marked completed can no longer count a failed command as successful memory
or runtime evidence. Malformed or contradictory command-status fields are
reported with the case and receipt before a report is produced. Nonzero exits
remain valid evidence for censored OOM, time-limit and kill outcomes. (#159) -
Respect memory scope when evaluating OOM kills. A job-wide memory limit
can no longer refute a per-node, per-rank or unknown-scope estimate without
a measured excess. Evaluation cases can declare job, node or rank limits;
omitted scope retains the total-job meaning, and undecided cases widen the
reported underprediction bounds. (#159) -
Require known account permissions for placement.
vq placement explain
rejects targets with missing or unknown account capabilities when an account
is explicitly requested, and holds the request when none have a recorded
match. Text and JSON explain the account constraint and preserve other
rejection reasons. Requests without an account constraint keep their
existing behavior. (#160) -
Preserve benchmark samples across interrupted checkpoints. The runner
saves result rows before marking their runs recorded in the journal.
Restarting after either write is interrupted collects the accepted job
again as needed and retains one result row, without repeating its execution.
(#162) -
Exclude remote scheduler-only jobs from local failed-runtime retry protection while retaining exact local references and ambiguous or local jobs.
-
Forward failed-marker acknowledgement through every update route. Scheduler runtime and helper updates, environment batches, host fan-out and staged drivers now honor
--acknowledge-failed-markerwithin the resolved update scope. Live or unobservable writers and durable managed/pause receipts remain protected; acknowledgement never implies--force. (#187)
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.
Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/
vibe-queue v0.26.42
Added
- Placement explanations enforce operating-system requirements. Requests
and targets can declare an OS label. A different or unknown target OS rejects
an explicit requirement before completion times are ranked, with every other
rejection retained. Text and JSON preserve the requested OS; requests that
omit it keep their behavior. (#160)
Fixed
-
Document the implemented opt-in broker and runtime workflows, correct obsolete broker status in the specification and distribution naming, and bind manual versions to their source tree.
-
Placement respects the job's execution time limit. The planner rejects
targets whose predicted execution upper bound exceeds the requested walltime,
even when the target accepts that reservation or has no walltime cap. It can
select a faster busy target or explain a hold when none fit. Equal limits fit;
queue wait, staging and result transfer remain outside execution walltime.
(#160) -
Benchmark cost recovery uses uncertainty in paired time savings. The
conservative break-even count now uses a Student-t lower bound on mean paired
core-second savings, rather than scaling the relative-speedup bound by mean
baseline time. It cannot be more optimistic than the point count. A
non-positive saving bound leaves the conservative count unavailable, with an
explanation in the text and JSON reports. (#162) -
Bound large scheduler orphan inventories without slowing ordinary commands.
Workspace and transfer-archive listing use one directory traversal when Python
is available, with a portable shell fallback, a 180-second observation budget
and bounded output. Incomplete observations report their operation and limit
and refuse cleanup. Archive removal no longer retries an ambiguous mutation.
Existing driver, age, local-job and scheduler protection still applies. (#185) -
Immutable replacement builds now disable the product installer's optional
queue pause for the build subprocess and its children. Running calculations
keep using their protected generation while the replacement compiles. Direct
in-place updates and required service pause scopes retain their existing
behavior.
Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.
Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/