Skip to content

Releases: vibyverse/vibe-scheduler

vibe-queue v1.0.2 - source release

Choose a tag to compare

@mpei mpei released this 30 Oct 03:45

Public source release of vibe-queue v1.0.2.

Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers. Follow the included installation instructions. Managed Git-based installers require a clone checked out at this tag.

Desktop availability is listed separately at https://vibe-qc.com/download/.

vibe-queue v1.0.1 - 2026-10-28 - Meyer's Contract

Choose a tag to compare

@mpei mpei released this 29 Oct 14:33

Added

  • Preserve contained native library file aliases during qualified slot seeding, rejecting escaping, cyclic, dangling and directory aliases before copying.

  • A broker job can carry its calibrated runtime estimate, and the completion-time policy plans with it. vq submit BROKER --estimate CALIBRATED sends the estimate's runtime interval (the receipt's lower bound, the calibrated upper bound) as vq broker accept --runtime-seconds LOW HIGH, recorded on the intent as runtime_seconds. With [broker] policy = "completion-time", a host where the predicted execution could overrun the job's time limit or the host's walltime limit is then refused with that reason. A raw receipt or an uncalibrated runtime is refused, and --mem-from-estimate/--time-from-estimate stay refused for broker jobs. The field is written only when set, so other broker records and intent digests are unchanged; a vq without it reports a record that carries it as corrupt, so use the option only when every host reading the broker store knows it. (#159)

  • vq broker show names the figures a completion-time blocker rests on. The broker job view (vq broker show, vq broker list --json and the broker part of vq status BROKER b-...) now includes wall_time_seconds and runtime_seconds (null when unset), and vq broker show prints wall_time_seconds: and runtime estimate: lines when they are set. (#159)

  • A gateway submission can carry its calibrated runtime estimate. POST /v1/jobs accepts an optional runtime_seconds: [low, high], checked like vq broker accept --runtime-seconds and recorded on the broker intent for the completion-time policy to rank by. vq remote submit --estimate CALIBRATED sends it from a calibrated estimate and refuses a raw receipt or an uncalibrated runtime before uploading. Relay connectors receive the field only when it is set. (#159)

  • The broker can place PBS and Slurm jobs through a driver daemon on another host. A broker scheduler lane's scheduler_driver may now name a daemon host reached over SSH. The broker and the driver each digest the lane's definition (lane_sha256: scheduler, dialect, submit_extra, scratch roots, job/CPU/walltime caps and runtime deployments) from their own config, leaving out per-machine routing such as SSH aliases and the driver's name. A lane whose definition differs, a driver that does not drive the lane itself, or a driver whose vq reports no lane digest gives no evidence. The payload is staged in the driver's home for the keyed submit, removed once the driver answers and kept while its answer is unknown. A driver on the broker machine is unchanged. (#160)

  • A runtime bundle build can run on a remote build host. vq runtime-bundle build --host NAME -d SRC ... checks the paths, builder and description locally, stages a tar of the source tree and the description in that host's home, and runs the host's own vq runtime-bundle build -a TAR .... The staging is removed afterwards, or kept with a clear error when SSH cannot tell whether the build was accepted; a rerun is then refused by the build record already there. vq runtime-bundle build-finish --host NAME RECORD packs on that host and relays its answer and exit code. build -a/--archive takes a source archive in place of -d. (#161)

  • A benchmark plan variant can take its identity from a portable runtime bundle. vq benchmark variant BUNDLE --name NAME checks a packed or published #161 bundle's archive against its manifest and prints a plan variants entry. Its runtime_digest is the artifact ID, which vq benchmark measure --runtime-root reports for a deployment of it. build_flags, the toolchain and dependencies libraries and storage_bytes come from the bundle; --library and the build and deployment costs are added as given. Plan and result schemas are unchanged. (#162)

  • New guides: installing vq on a workstation, a server or a PBS/Slurm cluster without root access, what changes for a job on each kind of host, registering programs (binaries, Python environments and cluster-only programs with their scheduler hooks), running vibe-qc calculations through the queue, and using vq from vibe-sketch and vibe-view. Four tutorials walk from a first job to a pinned vibe-qc calculation and a headless vibe-view figure. The starter configuration config/public-example.toml now shows a daemon host, PBS and Slurm login nodes and a registered vibe-qc program. Two more pages cover running ORCA and CRYSTAL with the shipped contrib/ wrappers, on a daemon host and on a cluster, and troubleshooting by symptom: unreachable hosts, refused submissions, pending gates, terminal states and fetch failures. A further page works through arrays, dependencies, chains and reruns with a parameter sweep and its analysis, including what each job can see of another's files. Another page covers asking for CPUs, memory and time: what a job sees, how the queue orders and admits it, and which limits stop a job on Linux and on macOS, where only the time limit is enforced. A page on managing jobs covers names and tags, pausing, killing, resubmitting, archiving and deleting finished jobs, and usage totals.

  • Consume product-owned analytical memory allocations with domains and scopes in estimate receipts; incomplete route models remain unknown instead of falling back to a heuristic scalar.
    Preserve resolved zero occupied/virtual dimensions in receipts and calibration ranges.

  • Estimate calibrations now record, per group, the observed range of each numeric feature (Nbf, Naux, occupied/virtual, roots, k-points, ranks, threads). vq estimate --calibration and vq estimate-evaluate --calibration leave an estimate outside those ranges, or missing such a feature, uncalibrated and say why, instead of extrapolating the multiplier.

  • Preserve explicit runtime-bound empirical memory-baseline provenance separately from analytical allocations, and reject inconsistent identities, domains or residual evidence.

  • vq estimate-evaluate reports gain route_groups, keyed by method, algorithm and hardware class. The new --thresholds option reads a vq.estimate-thresholds/1 document and gives each route group a pass, fail, undecided or insufficient verdict from its underprediction bounds.

  • Consume product-observed input digests and resolved resource features, refusing mismatched input bytes and conflicting operator overrides while preserving legacy receipt serialization.

  • vq submit HOST -- CMD with no -d or -c now runs CMD in an empty workspace, so a command that needs no files, such as the -- hostname canary, no longer needs a directory. Without -- the positionals still name one input file, and -- input.py keeps the single-file form. The refusals for a command given without -- say how to write it.

  • Add opt-in managed local Docker execution with immutable image/container bindings, authenticated worker accounting, durable cancellation and restart recovery. Unknown outcomes retain capacity and refuse resubmission or cleanup. Run shell payloads inside this managed route; arbitrary scripts that launch daemon-owned containers remain unsupported.

Changed

  • Reconcile the roadmap with the published 1.0.0 release and accepted article demonstrations, retaining outstanding qualification limits and identifying 1.0.x maintenance as the next target.

  • vq submit --help now describes how --mem-mb is enforced: by cgroups or the watchdog on Linux, by the scheduler on PBS and Slurm, and not at all on macOS.

Fixed

  • A broker log or result stream through a local scheduler driver no longer reads the caller's stdin. The driver helper that broker_target.stream() starts for a driver on the broker machine now gets /dev/null as stdin, like every other vq subprocess (#118), so a vq logs/vq fetch or gateway result collection through such a lane cannot consume input meant for the caller. (#195)

  • Portable release-check launchers. Release checks use the executing Python for the real driver helper and explicitly restore catchable signals before testing scheduler finalization, including when CI ignores SIGHUP. The 1.0 codename is registered as "Meyer's Contract" for its stable interface and compatibility boundary. (#199)

  • vq estimate now prints Job memory: unknown instead of unknown bytes when the job's memory figure is unknown.

  • Accept matching method labels with display capitalization in product dry-run evidence while still rejecting conflicting methods.

  • Add explicit campaign recovery through authenticated retained managed runtimes,
    including packed archive and launcher identities, without changing campaign
    bytes or submission keys after an active-runtime rollover.

  • vq status, vq logs, vq output and vq events now refuse a malformed job id with a one-line usage error instead of a Python traceback.

  • vq status labels a job's parent as (resubmitted from) instead of (auto-resumed from), which was wrong for vq resubmit and vq kill --resubmit.

  • Submit examples in the README and guides that ran a command without a workspace (vq submit HOST -- CMD), which vq refuses, now give one with -d, and the refusal says to. The vq submit --help watchdog note no longer says macOS falls back to a process-group walk: there the watchdog reads no process usage.

  • Scheduler polls recognize standalone OpenSSH address/UNKNOWN connection-close diagnostics and use the existing bounded refusal cooldown. Authentication and host-key errors remain visible even when a jump host appends connection-close text. Scheduler submission and staging replay behavior is unchanged.

  • vq submit HOST --program NAME input.py for a venv program now runs input.py with that program's interpreter, on the daemon host and on a remote host, including one with remote_python set; it used vq's own Python or remote_python while recording the program and its revision on the job. A...

Read more

vibe-queue v1.0.0 - source release

Choose a tag to compare

@mpei mpei released this 24 Oct 03:59

Public source release of vibe-queue v1.0.0.

Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers. Follow the included installation instructions. Managed Git-based installers require a clone checked out at this tag.

Desktop availability is listed separately at https://vibe-qc.com/download/.

vibe-queue v0.26.48 - 2026-10-22 - Raymond's Bazaar

Choose a tag to compare

@mpei mpei released this 29 Oct 14:32

This patch extends placement planning: fleet files built from live daemon capacity, Slurm and Torque/PBS lane shapes and recorded queue waits, GPU-aware explanations, estimate-completed requests, deployed portable runtimes as placement runtimes, an opt-in completion-time broker ordering with a read-only comparison, and candidate hosts in the console. Multi-user daemons can opt in to GPU device isolation. Daemon GPU jobs now report live and final GPU use, including across daemon restarts, the GPU sampler no longer outlives its job, remote registry migration exchanges bounded stdin and output, and the gateway again shows and cancels jobs a broker pass left held. The default capacity placement policy is unchanged; completion-time ordering is opt-in and site acceptance remains limited to its recorded scope.

Added

  • The broker can order its placements by predicted completion time (opt-in). With [broker] policy = "completion-time" and an absolute placement_fleet (a vq.placement-fleet/1 record of the daemon targets), each placement still admits only the hosts the existing capacity gates admit: open admission, a ready queue, and room now after outstanding charges. It then orders those hosts with the placement planner, using this pass's evidence laid over the record, so a faster target is preferred.

    • Never extra hosts. The policy never adds a host that the default policy would refuse.
    • Blockers. A host missing from the record, or rejected by the planner, becomes a completion-time: blocker.
    • Unreadable record. An unreadable record holds jobs rather than falling back.
    • Default. The default policy = "capacity" leaves broker placement unchanged. (#160)
  • vq broker capacity reports GPUs. Its answer gains gpus and free_gpus, taken from the daemon's own admission snapshot: the devices the daemon may hand to --gpus jobs (0 when it lists none) and how many are free now. A daemon older than GPU admission gives null for both, meaning unknown, not none. The text form shows gpus=FREE (of TOTAL). The figures are evidence for placement plans; the broker's own placement is unchanged. (#160)

  • The fleet console shows a broker job's candidate hosts. Under "Broker jobs" on /fleet/jobs, a job with a placement attempt now lists the hosts its decision ranked, best first, with the target it went to in bold. vq broker show lists them too, as candidates. A refusal by the completion-time broker policy now shows as its own completion-time reason code, instead of the generic placement-blocked. Its detail stays in vq broker show. (#160)

  • vq placement fleet --scheduler-shape also reads Torque/PBS lanes. On a scheduler = "pbs" host, it reads the lane's queue with qstat -Qf, which supplies its walltime and node-count maxima. An enabled or started that is false marks the lane under maintenance. The node shape comes from pbsnodes -a, using the smallest usable node the queue routes to: by the properties its resources_default.neednodes names on Torque, or by each node's queue on PBS Pro. A queue that routes to no node keeps the base node record rather than guessing. (#160)

  • vq placement compare shows what the completion-time broker policy would change. Given a fleet file, it places each of this host's recent finished local jobs twice: by the broker's default capacity ordering, and by [broker] policy = "completion-time" applied to that decision. Both run over capacity snapshots built from the fleet's daemon targets. Per job it shows both choices and their blockers, plus totals of same, different, held and skipped. Jobs without a declared --mem-mb are skipped. It is read-only: it places nothing and contacts no host. (#160)

  • vq placement explain --estimate completes a request from a calibrated estimate. Given a vq estimate --calibration ... --json document, it fills two request fields that are left unset:

    • runtime_seconds, from the receipt's lower bound and the calibrated upper bound;
    • mem_mb_per_node, from the calibrated peak in MiB, when that figure is per node: node scope, or job scope on a single-node request.

    Explicit request values win. The explanation gains an estimate section that says per field whether it was filled, kept or not filled, and why. A raw receipt is refused. Without the option, explanations are unchanged. (#160)

  • vq placement fleet --collect NAME asks daemons for their capacity directly. Each named daemon target, which must be a configured host, is asked for its vq broker capacity --json over SSH, the same way the broker asks. Only the programs in its base record are checked. Its figures are dated by this host's clock less the daemon's snapshot age. A daemon that gives no answer is recorded as unreachable now, with nothing free, and the build continues. Collecting a scheduler lane, an unknown or unconfigured target, or a target that also has --capacity is refused before any host is contacted. (#160)

  • vq placement fleet adds live daemon capacity to a fleet file. Run vq placement fleet BASE --capacity NAME=ANSWER.json ... [-o OUT], giving it your base vq.placement-fleet/1 record and each daemon's vq broker capacity --json answer. For each answered daemon it sets:

    • when the daemon was observed and whether it is reachable;
    • its free CPUs, memory and GPUs, only when its admission is open and its queue ready;
    • its GPUs per node.

    It also removes programs the daemon reports unavailable. Scheduler lanes and daemons without an answer keep their base record. Malformed or misdirected answers refuse the build, and the base file is never overwritten. It contacts no host. (#160)

  • Placement explanations can require GPUs. A vq placement explain request may declare gpus_per_node, whole GPUs per node, which for a single-node vq job is its --gpus. Targets record node.gpus, where 0 means none and a missing value is unknown, and a daemon may record free.gpus, its devices not assigned now. A target with too few GPUs, or an unknown count, is rejected with the reason. A daemon counts as fitting its free capacity now only when enough GPUs are free. Requests without GPUs are explained exactly as before. (#160)

  • Scheduler lanes' queue waits can come from recorded history. The driver now records when the scheduler accepted each job (scheduler_submitted_at) and when its poll first saw the job running (scheduler_started_at). vq placement fleet --queue-history NAME[=TARGET] sets a scheduler lane's queue_wait_seconds from the waits that this host's recent jobs on that target recorded. It uses the nearest-rank 10th to 90th percentile of up to --history-jobs jobs from the last --history-days. With fewer than --history-min waits, the lane keeps its base record. (#160)

  • vq placement fleet --scheduler-shape NAME[=HOST] reads a Slurm lane's shape. It asks the lane's Slurm host, read-only, for sinfo on the lane's partition, its queue. From the answer it records the smallest node group's CPUs, memory and GPUs, the tightest walltime limit (or unlimited), and a finite job-size limit as max_nodes. A host that gives no usable answer leaves the lane unreachable. A lane without a queue, a host that is not a configured Slurm host, or a non-scheduler lane is refused before any host is contacted. (#160)

  • A running daemon GPU job shows its GPUs now. Each daemon tick reads the end of a running GPU job's local nvidia-smi samples, only when the file has changed and never through a link. It keeps the newest complete sample on the job, as the scheduler poll does for a cluster GPU job. vq status then shows its gpu now: line, and the console shows its live GPU figures. Reading is telemetry only: a missing or invalid file never affects the job. (#168)

  • A finished daemon GPU job reports what its devices did. A job that a daemon gave GPU devices to (--gpu-devices) is now sampled by the same nvidia-smi sampler as a scheduler GPU job, limited to its own devices with nvidia-smi -i. The sampler runs outside the resource collector, so the job's resource receipt still measures only its command. When the job finishes, vq status shows its gpu use: line and vq status --json its gpu_usage. In single-user mode the daemon also writes the summary that listings and the fleet table read; a multi-user daemon leaves the user's workspace alone, and vq status summarises the samples when read. (#168)

  • A daemon's deployed portable runtimes can feed its placement record. vq runtime-bundle generations --root ROOT [--json] lists a deployment root's generations, read-only, with each one's program and whether it is verified. vq placement fleet --runtime-root NAME=ROOT reads that listing from daemon target NAME over SSH. It records each program's verified generations as the target's runtimes, current one first, so a request pinned to a --runtime-artifact is planned only where that generation is deployed. A target that gives no listing keeps its base record. (#160, #161)

  • A multi-user daemon can isolate GPUs between jobs (opt-in). With [daemon] gpu_isolation = true, every job's system scope gets DevicePolicy=closed. A GPU job may then open only its own /dev/nvidiaN and the shared NVIDIA control nodes, and a job without GPUs may open none. Each job may also open the standard pseudo-devices and whatever [daemon] gpu_isolation_allow lists: /dev/... nodes or char-/block- classes, for example char-infiniband_verbs for MPI. systemd can only allow, not deny, so other devices a job needs must be listed. The daemon refuses the setting when it is single-user, because a user manager cannot enforce device policy, or when gpu_devices holds non-numeric IDs. (#168)

Fixed

  • A daemon GPU job keeps its GPU use across a daemon restart. A job that a restarted daemon re-attaches now gets its live gpu now: sample. When it finishes,...
Read more

vibe-queue v0.26.47 - 2026-10-21 - Raymond's Bazaar

Choose a tag to compare

@mpei mpei released this 29 Oct 14:32

This patch adds calibrated memory and runtime estimates with explicit opt-in submission limits, queue-constrained placement, signed portable runtime bundles and protected generation pruning. It improves direct-host migration and retained scheduler-runtime admission, and adds GPU admission and usage reporting. Site-specific functionality remains limited to its recorded acceptance; candidate CI does not provision new GPU capacity or official platform channels.

Added

  • A calibration can be applied to the estimate a job is submitted with. vq estimate --calibration CALIBRATION --hardware-class LABEL --json prints a new vq.calibrated-estimate/1. It wraps the unchanged vq.resource-estimate/1 receipt with its digest, the calibration's id, method and training set, the matching group's multiplier (and whether it is a lower bound), and the calibrated peak (raw peak times multiplier, rounded up). When nothing applies it is uncalibrated with the reason, and the text output adds a Calibrated memory: line. vq submit --estimate accepts either document. A calibrated one is checked, its receipt is checked against the payload as before and recorded as resource_estimate, and the calibration is recorded in a new JobSpec field, resource_estimate_calibration. vq status shows the calibrated figure, and resubmission, reboot recovery and the idempotency intent keep it. Raw receipts are unchanged and nothing sets resources or placement from either figure. (#159)

  • Peak-memory calibration, fitted on a training set and judged on held-out cases. vq estimate-calibrate TRAINING_SET -o CALIBRATION [--min-cases N] writes a vq.estimate-calibration/1 document, with one multiplier per estimator, program version, method, algorithm, memory scope and hardware class. Each multiplier is the worst observed/predicted peak ratio among decided training cases, never below 1. Those are exact ratios where the measurement compares (a rank-scope estimate or a single process) and lower bounds otherwise, including OOM kills at a comparable limit. A lower-bound worst case is flagged, and a group with fewer than --min-cases (default 5) decided cases stays uncalibrated. vq estimate-evaluate HELD_OUT --calibration CALIBRATION adds, beside the raw verdicts, each case's calibrated verdict and the calibrated underprediction bounds. It refuses the training set and any of its cases. Without --calibration the report is unchanged, and no receipt, admission or placement uses a calibration. (#159)

  • vq submit --mem-from-estimate takes a job's memory limit from its calibrated estimate. With a calibrated, job-scope vq.calibrated-estimate/1 as --estimate, the job's mem_mb becomes the calibrated peak rounded up to MiB, exactly as if given with --mem-mb. It is derived before placement, so vq submit auto fits by it. It is refused with an explicit --mem-mb (which stays authoritative), a raw or uncalibrated estimate, or another memory scope. The job records mem_mb_source: "calibrated-estimate", checked again where it is created and shown by vq status as a mem_mb: line. Resubmission keeps it unless the limit is overridden, and reboot recovery keeps it. Without the flag no estimate changes a limit. (#159)

  • Estimate calibrations also fit runtime upper bounds. vq estimate-calibrate writes runtime_groups beside the peak-memory groups, keyed the same way but without the memory scope. Each group's multiplier is the worst observed wall time over the estimate's runtime upper bound, never below 1. A completed run counts exactly; a run stopped at its time limit, or one that passed the bound before it was stopped, counts as a lower bound. vq estimate-evaluate --calibration adds a calibrated_runtime verdict to each held-out case, calibrated coverage bounds to every summary, and a CALIBRATED COVERED column. Only the upper bound is scaled, and applying a calibration to a submitted estimate still uses only its memory groups. Calibrations fitted earlier load with no runtime groups. (#159)

  • A job's wall-time limit can come from a calibrated estimate. vq estimate --calibration now also applies the matching runtime group. Its vq.calibrated-estimate/1 records the calibrated runtime upper bound (calibrated_runtime_upper_seconds), or, when the runtime cannot be calibrated, why not. vq submit --estimate calibrated.json --time-from-estimate sets the job's wall-time limit to that bound, rounded up to whole minutes, exactly as if it had been given with --time-limit. It is refused with --time-limit or --wall-time-seconds, and refused for a raw estimate or an uncalibrated runtime. The job records wall_time_source: "calibrated-estimate", which is checked where the job is created, shown by vq status, and kept on resubmission (unless the limit is overridden) and reboot recovery. Without the flag, no estimate changes a limit. (#159)

  • Placement explanations honor an explicit scheduler queue or partition.
    Requests may declare queue, which must match a scheduler lane's recorded
    PBS queue or Slurm partition exactly before completion-time ranking. Unknown
    or different queues and daemon targets are rejected with reasons; no matching
    eligible lane produces a hold. Text and JSON preserve the constraint, and
    requests that omit it keep their existing behavior. (#160)

  • Old portable runtime generations can be pruned. vq runtime-bundle prune --root ROOT [--keep ID ...] [--apply] lists, and with --apply removes, the deployed generations nothing keeps. A generation is kept if it is the current or previous selection, if a queued or running job in this host's queue (multi-user included) is bound to it, or if it is named with --keep. The default is a read-only plan with sizes. --apply takes the deployment lock, rereads the bindings, and renames each generation out of place before removing it without following links, so a racing job fails closed. It also finishes an interrupted prune and leaves non-generation entries alone. The artifact store keeps the bundle, so deploy restores a pruned generation. Generations that scheduler jobs bind from another driver's queue are not visible to a prune on the cluster; protect them with --keep. (#161)

  • Portable runtime bundles can be trusted by their publisher's OpenSSH signature. A vq.runtime-trust/2 policy can list publisher public keys under signers, as an alternative to pinning every artifact digest.

    • Signing. vq runtime-bundle sign BUNDLE --key KEY checks the archive, then signs the canonical manifest with ssh-keygen -Y sign, writing manifest.json.sig. The key may be a private key file or a key held in ssh-agent.
    • Trust. Publish, deploy and HTTPS/SSH fetch trust an unpinned manifest only when ssh-keygen -Y verify (OpenSSH 8.1+) confirms its signature with a key listed for its publisher. A deploy trusted that way records the key's fingerprint as signer.
    • Fetch. An unpinned fetch verifies the signature before it requests the archive. A pinned fetch requests no signature.
    • Compatibility. Schema 1 policies are unchanged, and earlier versions refuse schema 2 policies. (#161)
  • Daemon hosts can admit GPU jobs. Start a daemon with vq daemon run --gpu-devices 0,1 (or [daemon] gpu_devices = ["0", "1"]) to list the GPUs it may hand out.

    • Admission. A local vq submit --gpus N job waits until N of those devices are free, then gets the lowest free ones. It sees them in CUDA_VISIBLE_DEVICES and VQ_GPU_DEVICES.
    • Records. The devices are recorded on the job as gpu_devices, so a restarted daemon keeps them busy. Killed jobs whose processes survive hold theirs too.
    • Submit. --gpus to a daemon host is accepted only when that host's running daemon lists enough devices. An older or stopped daemon refuses it, rather than oversubscribe. Brokered and auto-placed submits still refuse it.
    • Limits. Devices are assigned, not isolated, and a daemon job's GPU use is not sampled. Scheduler targets are unchanged. (#168)
  • A running scheduler GPU job shows its GPUs now. The driver's scheduler poll reads the end of each running GPU job's nvidia-smi samples, with one bounded tail per 32 jobs. It keeps the newest complete sample on the job as scheduler_gpu_live. vq status shows a gpu now: line and the console's job detail page a gpu now section, both with the sample's node-clock time. They follow the same coverage rule as the finished summary and give device-level utilisation and device memory, plus the memory of the job's own processes. Only running jobs that requested GPUs are read, so a scheduler target without one makes no extra remote call, and a failed read never fails the poll. vq status --json gains the nullable scheduler_gpu_live key. (#168)

  • GPU processes of a Slurm job's srun steps count as the job's. An srun step's tasks are slurmstepd's children, not the job script's, so the GPU sampler did not attribute their memory to the job. Now a GPU process whose /proc/<pid>/cgroup names this job, job_<SLURM_JOB_ID> matched exactly (cgroup v1 or v2), is attributed as well, and counted once. Without Slurm's cgroup containment, or on Torque/PBS, only descendants of the job script are attributed, as before. (#168)

  • The fleet jobs table shows a GPU job's measured GPU use. Below its +N gpu allocation, a running GPU job shows the mean utilisation over its listed devices and their memory in use out of their total. These come from its newest nvidia-smi sample, with the sample's age, and are marked stale past 10 minutes. A finished job shows its mean utilisation and the largest device's peak memory, marked peak. Both are device-level, follow the detail page's coverage rule and are never added into the allocation tile. The owning host's vq queue --json rows carry a finished GPU job's gpu_usage, and scheduler projections keep the driver's reading. ...

Read more

vibe-queue v0.26.46 - 2026-10-20 - Raymond's Bazaar

Choose a tag to compare

@mpei mpei released this 22 Oct 17:31

This patch adds optional per-target scheduler job caps and submission pacing, binds scheduler jobs and campaigns to verified runtime generations, and retains submission-time estimates with stable directory-input fingerprints. Site-specific limits remain operator configuration. Candidate CI qualifies the shipped source; configured remote-site acceptance and official platform delivery remain separate.

Added

  • A job can keep the estimate it was submitted with, and finished jobs become an evaluation set. vq submit --estimate RECEIPT attaches a vq estimate --json receipt to the job as resource_estimate. The submission is refused unless every input file the receipt fingerprinted is in the payload by content: the input file, a regular file under --dir, or a --compressed archive member. The estimate changes no resource, placement or admission. vq status shows an estimate: line, and resubmission and reboot recovery keep the receipt. A remote host checks it again, and a broker refuses the option. vq estimate-collect OUT_DIR --hardware-class LABEL writes a vq.estimate-evaluation-set/1 from the local queue's finished jobs that carry an estimate, pairing each receipt with the job's resource-usage.json, outcome and deciding limit, for vq estimate-evaluate. (#159)

  • vq estimate --input accepts a directory. A directory stands for every regular file under it, in relative-path order, each opened without following a link. Links and special files are left out and counted in a warning, because a vq submit -d payload carries a link as a link. The existing bounds apply after expansion (64 files of 64 MiB each), a walk visits at most 10,000 entries, and an empty or partly unreadable directory is an error. An estimate of a directory is therefore accepted by vq submit --estimate RECEIPT -d DIR for that directory, and refused once one of its files changes. Receipts for file inputs are unchanged. (#159)

  • A scheduler job can be bound to a portable runtime generation, verified on its node. vq submit TARGET --runtime-root ROOT --runtime-artifact ID records the named generation for a scheduler target, where ROOT is the vq runtime-bundle deploy target on the cluster; the identity is required because the driver does not read the cluster's selection. Before the command runs, the job script checks the generation on the compute node. Nothing may be a link, the SHA256 of the canonical manifest.json must be the identity, the deployment receipt must record it verified, and the payload must not be group or world writable. The node needs sha256sum or shasum. If the check fails, the command never starts and the job fails with exit code 126, giving the reason in stderr.log; it never runs another generation. The job gets VQ_RUNTIME_PAYLOAD and VQ_RUNTIME_ARTIFACT as on daemon hosts. Unbound jobs' scripts are unchanged. (#161)

  • A campaign node can bind its declared runtime generation on a scheduler target. vq campaign run no longer refuses a node that declares runtime_digest and runtime_root on a configured scheduler target. It submits with --runtime-root/--runtime-artifact as on a daemon host, and the job script verifies the generation on the compute node before the command (#161). Preflight checks the root by the rule vq submit applies to a cluster path, a plain absolute path, before anything is submitted. A scheduler job's start is recorded before its node checks the generation, so the controller takes the generation as run only once the job completed with exit code 0. While the job runs, the attempt records no generation, and the missing record is not taken as a mismatch. A job whose node refused the generation exits 126 without starting its command. It becomes a failed attempt with no generation recorded and nothing published, and the controller logs the node's reason. A failed scheduler job records no generation even if its command ran. (#165)

  • Scheduler targets can set scheduler_max_jobs and
    scheduler_submit_interval_seconds to bound this driver's outstanding
    allocations and pace submission attempts. Canonical targets and their aliases
    share these limits without holding unrelated targets or local work. Site-wide
    account quotas remain the scheduler's authority.

Changed

  • Track the bounded v0.27 feature release with a GitLab milestone and acceptance issue, require weekly roadmap reconciliation, and distinguish v0.26.45 delivery from newer main changes.

Fixed

  • Document HTTPS gateway and outbound relay access alongside SSH in introductory and integration guides. Reconcile idempotency and backpressure wording with the separate gateway contract. Reconcile roadmap release status with v0.26.45.

  • Reconcile the roadmap with current development. Distinguish v0.26.44 from main-only increments, replace obsolete closed-issue gates with the eight active feature boundaries, and frame v0.27 as a bounded feature release. Correct related gateway, campaign and historical planning references without claiming pending live qualification has passed.


Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.

Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/

vibe-queue v0.26.45 - 2026-10-19 - Raymond's Bazaar

Choose a tag to compare

@mpei mpei released this 23 Oct 11:58

This patch adds outbound relay cancellation, logs, verified result transfer and backpressure; binds daemon-host jobs and campaigns to portable runtime generations; improves scheduler GPU requests and allocation visibility; and includes an Apple Silicon app builder. Candidate CI qualifies the shipped source. Real site gateway and packaged-app acceptance remain separate from the source tests.

Added

  • Add an Apple Silicon macOS app, DMG and ZIP builder containing Python, the
    Queue CLI and web console dependencies. Installed commands work without an
    external Python environment and keep configuration and jobs outside the app.

  • A daemon-host job can be bound to a deployed portable runtime generation. Use vq submit HOST --runtime-root /abs/runtime [--runtime-artifact SHA256]. At submit, the execution host resolves the deployment's current selection, or the named generation, and accepts it only as vq runtime-bundle deploy left it. That means a matching manifest identity, a verification receipt that says it verified, and an operator-owned, non-writable payload. The job records it as runtime_bundle. Before dispatch the daemon checks that same generation again and runs the job with VQ_RUNTIME_PAYLOAD and VQ_RUNTIME_ARTIFACT. A later deploy or rollback changes only later submissions, and a generation that no longer verifies fails the job rather than letting it run on another. vq status shows a runtime: line, the console's software column names the generation, vq resubmit keeps it, and it is part of an --idempotency-key submission's identity. Scheduler targets refuse the option for now. Without it, submissions, their hashes and the remote wire are unchanged. (#161)

  • The relay connector (#164, slice 5) reports its free spool ([connector] spool_mb, default 4096). Work that would not fit stays at the gateway, and clients see connector_backpressure. The new connection-loss acceptance harness covers an outbound-only coordinator and interruptions at every stage with restarts.

  • Cancellation through a relay gateway (#164, slice 2). A job no connector has claimed is cancelled at the gateway and never delivered. For an admitted job the cancel is a durable request: the connector kills it on the broker host and acknowledges stopped or not-stopped, and clients see requested until then.

  • Relay mode for the submission gateway (#164, slice 1). With [gateway] mode = "relay" the reachable gateway holds admitted jobs (IDs g-…) and needs no broker, SSH keys or fleet inventory. vq connector run on the broker host makes only outbound HTTPS requests: it claims the project's jobs with fencing epochs, admits each to its broker under the same scoped idempotency key and reports their state. Connectors enroll with a one-use token (vq gateway connector enroll, vq connector enroll), rotate online and are revoked by the operator, one per project.

  • Logs through a relay gateway (#164, slice 3). A client's logs request asks the connector for that job's output for 10 minutes. The connector reads it on the broker host and pushes new bytes by offset into a bounded per-job buffer (16 MiB per stream). Clients page through what has arrived, with its time.

  • Results through a relay gateway (#164, slice 4). A client's results request asks the connector to collect the finished job on the broker host. The connector pushes each file by offset within the remaining storage quota, and the gateway publishes them only after every file matches the manifest's size and SHA-256. Downloads then work as in local mode.

  • Campaign attempts record the runtime their job actually ran with. Until a declared runtime_digest can be bound to a submission, the controller archives what each job's queue resolved at dispatch: the node's program and version, the git SHA authenticated immediately before dispatch, the immutable artifact identity (such as a scheduler deployment's per-SHA launcher) and the runtime slot. It copies these from the job's vq status --json into the attempt's journal record as runtime, and every published output's manifest carries them as executed_runtime. vq campaign status lists them under Ran with and in its JSON, and /fleet/campaigns shows them beside each attempt. This is a record, not a binding: a declared runtime_digest is still refused, and a job without a registered program, or not yet dispatched, records nothing. Journals and manifests without a recorded runtime are written and hashed exactly as before. Ones that record a runtime are refused by older controllers. (#165)

  • A campaign node's runtime_digest now binds its job to that portable runtime generation. The node names runtime_root, the vq runtime-bundle deploy target on its daemon host (a root needs a digest), and the controller submits with --runtime-root and --runtime-artifact. The queue then binds exactly that generation, whatever the deployment currently selects, or refuses the job. Once the job has started, the controller compares the generation in the job's own record with the declaration. A job that ran anything else, or that records none, is reported, and nothing it produced is published. The attempt's executed runtime and each output's manifest carry the generation, and the manifest's runtime_digest is now the enforced one. A digest without a root, or one on a scheduler target, is still refused before anything is submitted. Campaign files that declare no root keep their digest, so existing journals stay valid. (#165)

  • The fleet console's refresh cost is measured and documented. scripts/measure_fleet_console.py runs the console's real sweep and jobs table against a synthetic fleet. Only the SSH transport is replaced: it answers with the JSON the real commands print, after a simulated round trip. The script reports remote calls per sweep, sweep durations, table render times and sizes, the machine and its load. docs/fleet_console.md gains a "What a refresh costs" section with three named workloads. They show one queue listing and one overview per host per refresh, plus one vq top for a host running a direct job, independent of the number of jobs, and no remote call while pages render. The figures are labelled as those workloads on that machine. (#168)

  • The fleet console shows which nodes a distributed scheduler job holds. The execution host string the scheduler reports is now read node by node. That covers PBS/Torque exec_host (core ranges such as node07/0-7+node08/0-7, Torque's one-entry-per-core form, and PBS Pro's node07/0*8) and Slurm node lists (node[01-04,07],gpu[1-2]-ib). The jobs table's nodes column reads N nodes for a multi-node job, with each node, and for PBS/Torque its allocated cores, on hover. The job drill-down gains a "nodes" section listing them. Slurm's node list names nodes only, so their cores are shown as not reported rather than split evenly. Expansion is bounded, and a string that does not parse is shown as reported, as before. This is the scheduler's allocation, not a per-node measurement. (#168)

  • Scheduler jobs can request whole GPUs with vq submit --gpus N. The count is a job total, recorded on the job as gpus. Torque renders it as #PBS -l nodes=1:ppn=CPUS:gpus=N (vq's Torque jobs are single-node) and Slurm as #SBATCH --gpus=N, so the scheduler allocates and enforces it. vq status shows a gpus: line, queue tables add a GPUS column when a listed job has one, the job sees VQ_GPUS, and vq resubmit keeps it. In the fleet jobs table the cpus column shows +N gpu, and a gpus allocated tile sums what running jobs requested. A daemon host has no GPU admission or device assignment, so --gpus is refused there rather than ignored. PBS Pro's ngpus and GPU types stay with the host's submit_extra, and GPU use is not measured yet. Without --gpus, submissions, their idempotency hashes and the remote wire are unchanged. (#168)

Fixed

  • Detect lost benchmark measurements before resuming. The runner now
    requires a results row for each run the journal marks recorded. A missing
    results file or missing row stops with an actionable error, preserving saved
    files and job identities instead of reporting success with lost samples. (#162)

  • Reject benchmark results from another plan before resuming. The runner
    checks the saved results' plan digest before contacting the queue or writing
    a checkpoint, preventing further jobs from being submitted under mismatched
    results. The error asks for the campaign's matching results. (#162)

  • Use the verified runtime launcher for QVF scheduler submissions. The managed
    Python launcher is already verified in the target runtime pin, without requiring
    an extra site wrapper hook that
    could conflict with the existing interpreter mapping. Missing or wrong-target
    launcher evidence and conflicting legacy hooks are refused before staging;
    runtime SHA checks remain in force. (#188)


Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.

Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/

vibe-queue v0.26.44 - 2026-10-18 - Raymond's Bazaar

Choose a tag to compare

@mpei mpei released this 23 Oct 11:58

Validation scope

This patch updates QVF protocol checks to toolkit 0.5.0, adds gateway client and placement constraints, and strengthens scheduler start fencing, benchmark isolation and fleet filters. The exact candidate runs the full Queue suite, lint and documentation gates. Broader placement, dashboard and benchmark qualification and a real TLS gateway calculation remain pending; historical scheduler attempts without start tokens stay protected.

Added

  • Require data access before ranking placement targets. Read-only placement
    plans can declare required access labels and match them against each target's
    recorded capabilities. Unknown or missing access rejects that target, with all
    reasons retained in text and JSON explanations. Requests without access
    requirements keep their existing behavior. (#160)

  • Require declared license eligibility in placement plans. Requests can
    declare required_licenses, and targets can record licenses. Every required
    label must match before completion-time ranking; missing or unknown eligibility
    rejects a target with an explanation. Text and JSON retain the requirements,
    and unconstrained requests keep their existing behavior. (#160)

  • vq remote submit|status|logs|fetch|cancel is a client for the submission gateway (#163, slice 5). It reads the URL and key from an owner-only credentials file, uploads in resumable chunks and retries safely (a dropped response never creates a second job). It follows logs and fetches results, checking each file's SHA-256.

  • vq gateway (#163, slice 1) adds the identity layer of the HTTPS submission gateway on the broker host:

    • the opt-in [gateway] and [gateway.projects.NAME] config;
    • vq gateway key create|list|revoke|rotate, with per-project API keys shown once and stored only as an HMAC under a private pepper;
    • vq gateway serve, which refuses a plaintext non-loopback bind, plus /v1/healthz and /v1/whoami and an owner-only audit trail.

    Submission routes follow in later slices. The gateway is for a trusted group only (SECURITY.md).

  • The submission gateway (#163, slice 3) pages an owned job's stdout or stderr by byte offset (GET /v1/jobs/{id}/logs, read scope). It also cancels it (POST /v1/jobs/{id}/cancel, cancel scope): at the broker, and on its host once placed. A transfer in doubt is reported as not stopped.

  • The submission gateway (#163, slice 4) collects a finished job's workspace once (GET /v1/jobs/{id}/results, download scope). It keeps only regular files under safe paths and returns a SHA-256 manifest, and serves files by exact path with their digest. Results count against the storage quota, and placed jobs count as active until collected. vq gateway prune removes results and uploads past their project's retention.

  • The submission gateway (#163, slice 2) accepts resumable, digest-checked tar uploads (/v1/uploads) and submissions (POST /v1/jobs) through the broker's keyed intake. Submissions are idempotent per client key, checked against project policy (programs, CPU, memory and wall-time caps, waiting-job limit) and owned by the submitting principal. Jobs are listed and shown with GET /v1/jobs and GET /v1/jobs/{id}, without the command, payload or fleet host.

  • vq admin reconcile-submit JOBID --fence-unstarted (preview, then --apply --plan-digest) settles a scheduler attempt left submit_outcome_unknown without guessing whether it was admitted. Every new single-job attempt carries a daemon-issued start token. Its job script first claims _vq/attempt-<token>.claim exclusively and exits without running if the attempt was fenced. The fenced job ends FAILED with scheduler state submit_fenced_unstarted, is never retried automatically, and nothing is submitted or cancelled. Attempts from before this change cannot be fenced (#179).

Changed

  • Check the QVF queue protocol against toolkit 0.5.0. Its format-version-1 schema and existing job lifecycle remain unchanged.

Fixed

  • Submit fresh jobs for a new benchmark campaign. Fresh run directories get
    a persisted campaign identity, so running the same plan again cannot reuse
    an earlier campaign's jobs. Restarts and directory moves retain their keys,
    and older journals preserve their original keys for safe recovery. (#162)

  • Keep benchmark runs isolated after an unresolved error. A run with a
    problem continues to block later runs on its node class across runner
    passes. An unknown job state can no longer let another benchmark start
    alongside a job that may still be running. Independent classes continue,
    and restarting checks the existing accepted job before proceeding. (#162)

  • Filter fleet calculations by their execution state. Waiting and unconfirmed
    scheduler jobs no longer appear under the running filter. The state column,
    sorting, refresh and jobs API use the cached effective state, with the raw vq
    lifecycle retained for diagnosis. Last-known states remain selectable, and
    selections survive disappearing jobs. (#168)

  • Keep fleet program filters available during host outages. The program
    selector includes last-known jobs and retains the selected program when
    matching jobs disappear, preventing later filter submissions from silently
    dropping that selection. (#168)


Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.

Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/

vibe-queue v0.26.43 - 2026-10-17 - Raymond's Bazaar

Choose a tag to compare

@mpei mpei released this 23 Oct 11:58

Validation scope

This patch collects placement account and MPI-layout handling, conservative
command/OOM memory evidence, checkpointed benchmark recovery and host-scoped
failed-runtime retry checks. Candidate qualification runs the full Queue suite,
lint and documentation gates. Inclusion records source delivery; it does not
claim broader feature completion or scientific acceptance for queued work.

Added

  • Honor MPI rank and thread layouts in placement explanations. Requests
    can declare mpi_ranks and threads_per_rank alongside their total CPU
    count. Whole ranks must fit each node before targets are ranked, so a small
    per-node CPU average cannot hide an oversized rank layout. Text and JSON
    preserve the layout, while requests without one keep their existing behavior.
    (#160)

Fixed

  • Estimate evaluation checks command outcomes in usage receipts. A case
    marked completed can no longer count a failed command as successful memory
    or runtime evidence. Malformed or contradictory command-status fields are
    reported with the case and receipt before a report is produced. Nonzero exits
    remain valid evidence for censored OOM, time-limit and kill outcomes. (#159)

  • Respect memory scope when evaluating OOM kills. A job-wide memory limit
    can no longer refute a per-node, per-rank or unknown-scope estimate without
    a measured excess. Evaluation cases can declare job, node or rank limits;
    omitted scope retains the total-job meaning, and undecided cases widen the
    reported underprediction bounds. (#159)

  • Require known account permissions for placement. vq placement explain
    rejects targets with missing or unknown account capabilities when an account
    is explicitly requested, and holds the request when none have a recorded
    match. Text and JSON explain the account constraint and preserve other
    rejection reasons. Requests without an account constraint keep their
    existing behavior. (#160)

  • Preserve benchmark samples across interrupted checkpoints. The runner
    saves result rows before marking their runs recorded in the journal.
    Restarting after either write is interrupted collects the accepted job
    again as needed and retains one result row, without repeating its execution.
    (#162)

  • Exclude remote scheduler-only jobs from local failed-runtime retry protection while retaining exact local references and ambiguous or local jobs.

  • Forward failed-marker acknowledgement through every update route. Scheduler runtime and helper updates, environment batches, host fan-out and staged drivers now honor --acknowledge-failed-marker within the resolved update scope. Live or unobservable writers and durable managed/pause receipts remain protected; acknowledgement never implies --force. (#187)


Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.

Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/

vibe-queue v0.26.42

Choose a tag to compare

@mpei mpei released this 23 Oct 11:58

Added

  • Placement explanations enforce operating-system requirements. Requests
    and targets can declare an OS label. A different or unknown target OS rejects
    an explicit requirement before completion times are ranked, with every other
    rejection retained. Text and JSON preserve the requested OS; requests that
    omit it keep their behavior. (#160)

Fixed

  • Document the implemented opt-in broker and runtime workflows, correct obsolete broker status in the specification and distribution naming, and bind manual versions to their source tree.

  • Placement respects the job's execution time limit. The planner rejects
    targets whose predicted execution upper bound exceeds the requested walltime,
    even when the target accepts that reservation or has no walltime cap. It can
    select a faster busy target or explain a hold when none fit. Equal limits fit;
    queue wait, staging and result transfer remain outside execution walltime.
    (#160)

  • Benchmark cost recovery uses uncertainty in paired time savings. The
    conservative break-even count now uses a Student-t lower bound on mean paired
    core-second savings, rather than scaling the relative-speedup bound by mean
    baseline time. It cannot be more optimistic than the point count. A
    non-positive saving bound leaves the conservative count unavailable, with an
    explanation in the text and JSON reports. (#162)

  • Bound large scheduler orphan inventories without slowing ordinary commands.
    Workspace and transfer-archive listing use one directory traversal when Python
    is available, with a portable shell fallback, a 180-second observation budget
    and bounded output. Incomplete observations report their operation and limit
    and refuse cleanup. Archive removal no longer retries an ambiguous mutation.
    Existing driver, age, local-job and scheduler protection still applies. (#185)

  • Immutable replacement builds now disable the product installer's optional
    queue pause for the build subprocess and its children. Running calculations
    keep using their protected generation while the replacement compiles. Direct
    in-place updates and required service pause scopes retain their existing
    behavior.


Download Source code (zip) or Source code (tar.gz) under Assets. These are source archives, not prebuilt installers; follow the included installation instructions. Managed Git-based installers need a clone checked out at this tag.

Documentation: https://vibe-qc.com/ · Desktop downloads: https://vibe-qc.com/download/