3.2.0

3.2.0 — shared GPUs, one-shot workers, studies with budgets, DDP, provenance

3.2.0 completes the control-plane / worker split that 3.1.0 started (#764). A worker's GPU can be shared by memory budget, one-shot workers come up inside SLURM jobs and ssh commands, and worker runs can keep their files in an S3 or MinIO bucket, record their outputs for the Teaching Inspector, and outlive a control-plane restart. Studies enforce their budgets on the server, show in a Studies panel, and plugins can submit them as durable jobs. TrainingLoop can train one model across the GPUs of a worker run (strategy: ddp). A run can record which dataset files and model weights it read, and a worker run records the environment that executed it. The eval pack scores free text with BLEU and ROUGE-L and evaluates visual question answering. Everything new is off by default: a run with no target executes exactly as in 3.1.1, the exported Python script is unchanged, and the single-node-parity check holds both to that.

The full entry-by-entry notes are in CHANGELOG.md. New guides: Studies, Evaluation pipelines and Execution design notes.

Added

• Shared GPUs by memory budget (#788): a card set to shared admits runs while their declared budgets fit, caps each run to its budget, and refuses a run without one.
• One-shot workers (#790, #845): cdui agent start --one-shot takes one run and exits, for SLURM jobs and ssh commands; one enrollment serves a whole SLURM array.
• S3 / MinIO artifacts (#791): a worker without the shared folder keeps its files in a bucket (CODEFYUI_S3_*).
• Worker runs record outputs (#789) and the Teaching Inspector reads them from the worker.
• Adoption after a restart (#793): a restarted control plane takes over the runs still executing on its workers.
• Memory estimates (#794): the Queue on... dialog offers a static estimate for a graph no run has measured.
• Studies with budgets (#343): server-enforced max_runs / max_active_seconds with a halted state and resume; a Studies panel; plugin-scoped durable jobs under /api/plugins/{id}/jobs.
• TrainingLoop strategy: ddp (#792): one model across the GPUs of a worker run, loopback-only rendezvous, falls back to one process with a ddp_not_available warning.
• Run provenance and worker run environments (#344): run option provenance (identity / content) records dataset and model inputs per node with sha256 content hashes; the environment fingerprint moves to schema 2 with pack content identity, and a worker's run subprocess reports its own fingerprint.
• eval pack (#139): Eval-BLEU, Eval-ROUGE, VQA evaluation, and Eval-BatchInference generation settings taken from the model.

Changed

• A study that uses up budget.max_runs ends halted; queue reasons and worker messages for the new features carry codes (#841).

Fixed

• Two workers offering the same GPU no longer run two runs on it at a limit of 1 (#842).
• A worker run cancelled while its subprocess starts reports finishing on Windows (#856).
• A worker's node catalog is no longer compared against a stale cached catalog (#863).
• SSH workers stop retrying failures a retry cannot fix; a removed worker's agent stops (#840); the Workers panel, Add worker and Queue on... confirm and explain what they will do (#840).
• Run devices are checked against this server and device fallbacks show in the run log (#839).
• Sweep Final metrics show the objective's branch; Runs panel queue reasons and layout; New sweep dialog layout; toolbar wrap at 1440 px (#841).
• VectorIndex.save writes a whole index or leaves the old one (#139).

---

Full changelog: 3.1.1...3.2.0