Skip to content

0.8.0

Latest

Choose a tag to compare

@fab2s fab2s released this 09 Aug 20:13
· 89 commits to main since this release

Upgrading from 0.7.0? Most of the surface moved by rename with a
deprecated alias left behind, so existing code keeps compiling and
starts warning. Two changes do not warn: the Rust floor is now 1.91,
and flodl-hf needs owner-qualified Hub repo ids (bert-base-uncased
becomes google-bert/bert-base-uncased) because hf-hub 1.0 no longer
follows the Hub's redirect. See UPGRADE.md for the
step-by-step migration.

Added

  • AMD (ROCm) GPU support. flodl-sys compiles and links against a hipified libtorch. The vendor is a build-time property (libtorch ships one backend per build) and is reconciled in flodl-sys/gpu_compat.h, which maps the shim's CUDA spelling onto c10/hip + ATen/hip and routes streams through HIPStreamMasqueradingAsCUDA. Not yet validated on AMD hardware: this covers compile, link, provisioning and detection only. Provisioning is vendor-plural throughout, including the parts that only a second vendor makes visible: cluster fan-out counts a worker's GPUs for both vendors in one round trip (nvidia-smi for NVIDIA, the KFD topology's vendor_id 4098 for AMD, which is mask-proof) and reduces them with that host's declared arch:, since one libtorch build serves one vendor and that field already names which; an undeclared host reporting both vendors is refused by name rather than guessed at, because the controller cannot know which build it will load and a guess assigns ranks to devices nobody addresses.

  • flodl-hw, a dependency-free hardware-detection crate shared by flodl and flodl-cli (flodl::sys and flodl_cli::util::system are facades over it). AMD devices are read from the kernel's KFD topology, and reported as usable devices only when a ROCm userspace runtime is present; otherwise they surface as a finding naming the arch and what to install.

  • fdl libtorch download --rocm 7.0 and --rocm 7.1. --path, --no-activate and --dry-run are now listed in fdl libtorch -h. The two ROCm builds cover exactly the same gfx targets, so the choice is about the runtime, not the hardware: the HIP ordering rule puts the host's own ROCm ahead of the bundled one, and within a major version that ABI only grows, so a bundle older than the host loads while a newer one can fail on a symbol the host does not have. Auto-detection therefore routes to 7.0, which serves every 7.x host; a box that wants the exact match asks for it by version. Picking the newest bundle the detected host runtime can satisfy is the better rule and waits on ROCm version resolution. The covered gfx list is read off the published archive's own index rather than inferred: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1102, gfx1150, gfx1151, gfx1200 and gfx1201 ship rocBLAS kernels, while gfx900 and gfx906 (Vega 10 and Vega 20, i.e. MI50 and Radeon VII) carry MIOpen tuning databases and no kernels at all, so they count as uncovered instead of being installed and left to fail at the first matmul. Both ROCm buckets carry the same set. Upstream publishes ROCm builds for Linux only, so --rocm is refused on Windows and macOS with that reason.

  • RHEL-family install guidance. The missing-toolkit message from flodl-sys's build script now prints a RHEL/Fedora: sudo dnf install … line beside the Ubuntu one, and fdl probe/fdl setup pick the dnf spelling on RHEL-family hosts (detected via /etc/os-release). The rpm names are a mechanical transform of the Debian ones (-dev → -devel, g++ → gcc-c++), every spelling verified against the vendors' rhel9 repositories: NVIDIA's cuda-rhel9 repo and AMD's rocm/rhel9 repo, whose setup one-liners the CI toolkit phase now exercises.

  • fdl probe reports a libtorch variant this host's dynamic linker cannot load. Upstream does not build every archive against the same C library baseline: measured on 2.10.0, the cpu and cu128 trees need GLIBC_2.29 while the rocm7.0 tree needs GLIBC_2.35. RHEL 9 ships glibc 2.34 and cannot move past it (libstdc++ is upgradable through gcc-toolset, glibc is not, since the whole userland links against it), so that pair downloads, compiles, links, and then dies in the loader quoting symbol versions. The probe asks ldd and names the unmet versions with what they mean, before any of that is spent. Found by the OS matrix's rocky leg on its ROCm rotation; Dockerfile.rhel documents the baselines as a buildable reference.

  • fdl probe and fdl setup report missing prerequisites with the command to install them: host tools (curl or wget, unzip, a C++ compiler) and the vendor toolkit headers for the active libtorch variant. Presence is decided by capability, not by a list of known paths: when the header is in neither the toolkit root nor the default system include directories, the check asks the compiler that will do the build, with the same -I it will get, so a box carrying its headers somewhere unusual (or reaching them through CPATH or a multiarch directory) is not told to install what it already has. flodl-sys's build gate shares the behaviour, so a missing header now names the packages to install instead of failing inside the C++ compile (libtorch bundles the runtime libraries, so headers are the only gap). The command is rendered per family throughout, so a RHEL host is told sudo dnf install hip-devel … rather than apt with Debian package names. The probe runs only for a header already believed missing, so an ordinary build spawns nothing, and "no compiler to ask" stays a distinct answer from "missing". Setup reports the native prerequisites only when the build is native.

  • Per-OS CI matrix (ci/os-matrix.sh, .github/workflows/os.yml) covering Linux x86_64 and aarch64, macOS arm64 and Windows x86_64. Each host installs libtorch through fdl and builds against it; the Linux leg alternates CUDA and ROCm.

  • A RHEL-family box you can actually open (Dockerfile.rhel, the rhel compose service, commented rhel-* commands in fdl.yml). Now that the tool gives RHEL-specific advice, a RHEL question should be answerable on a RHEL userland rather than by reasoning about one: the dnf package names, rrsync shipping as non-executable documentation, curl-minimal conflicting with curl, and which libtorch baselines EL9 can load. Deliberately not what CI uses — the rocky legs run a stock image and install prerequisites the way the docs tell a user to, which is the step a pre-baked image would skip.

  • ci/rig-ladder.sh, a multi-GPU validation ladder that cannot report a false pass. The NCCL suites are device-gated (require_multi_gpu() early-returns), so on a one-GPU box the whole suite reports ok in a tenth of a second having tested nothing. The ladder answers what a rig actually EXECUTED: every rung reports RAN or SKIPPED with its reason, a pass requires a zero exit AND the trainer's done: line AND no libtorch [W] warnings, and a ladder where nothing ran exits non-zero rather than looking green. Three rungs: single-host process-per-rank, heterogeneous cross-host fan-out, and a walk-in dialing a guardrailed door (controller here, worker joins, fdl start fires the freeze).

  • CI runs what it used to only compile. The CUDA and ROCm jobs now RUN flodl's lib suite on their GPU-less runners (the driverless deployment a --features cuda/rocm build meets on any CPU-only box, and it immediately caught two driver-assuming tests); the CPU job asserts the model-sig probe wire contract (the flodl-model-sig: line, exit before training, identical hex across runs) and feeds fdl join-config --cloud-init's artifact through a real YAML parser; and the OS matrix gained a scaffold smoke: fdl init --native, the registry-pin regression guard, then an env-scrubbed ./fdl build and ./fdl run against the checkout's flodl, proving the scaffold's printed next steps on every compiling host. A rockylinux:9 job (host + docker run, because a container: job cannot free host disk, and the rocm rotation needs ~30 GB) runs the same script on the OPPOSITE rotation phase from the ubuntu leg, so every push exercises both vendors, one per distro family; FDL_CI_VARIANT gained cpu and rotate-alt arms. The matrix covers both ends of the supported range rather than only one: Ubuntu 24.04 (the base releases are built against, so the floor users have) alongside 26.04, and two EL releases, each given only variants it can actually run — EL9's glibc 2.34 cannot load the rocm archive at all, so it takes the cpu leg while EL10 (glibc 2.39) carries the RHEL-family vendor rotation. Scheduling a combination that cannot work is not a test, it is a failure that never heals.

  • fdl coverage and fdl coverage-all, a coverage report that names what it could not run. coverage is the CPU one-liner and therefore a FLOOR, scoring every GPU-reachable region as missed; coverage-all (ci/coverage.sh) drives the phases itself across the containers they each need, and prints a RAN / SKIPPED / FAILED roster rather than one number. The roster is the point, and it is the same hazard the rig ladder answers: a hardware- or tool-gated suite returns early and reports ok, so a single percentage can be built entirely out of suites that executed nothing. Coverage here is a number to look at, deliberately never a threshold to clear: the I/O glue in flodl-cli is uncovered on purpose (mocking Command / fs / tty is testing the mock), so a floor would push toward exactly the tests this project decided not to write. --skip leakcheck is required rather than tidy: those tests assert RSS growth stays under a bound, and the instrumentation's own counters allocate.

  • fdl fmt and fdl fmt-check. Formatting became a CI gate with the rustfmt adoption below, and a gate with no local command means contributors meet it for the first time as a red build. fmt writes, fmt-check is byte-for-byte the check CI runs.

  • ci/release/11-semver-checks.sh: public-API breakage checked against what is actually published, per crate, automating a step that was being done by eye. It catches a class the test suite structurally cannot, since all 3000 tests compile against the current source and none of them can notice that a name a downstream crate imports has gone. It runs after 02-version-sync.sh deliberately: the question is whether the DECLARED version covers the changes made, so running it before the bump correctly fails. Needs no libtorch (DOCS_RS=1, and rustdoc never links); absent tooling prints UNVERIFIED rather than passing quietly.

  • Discovery windows: a cluster can now form with no roster at all, because controller.join.discovery: true makes the join window itself define the world. workers: may be empty; whoever walks in during the window (see fdl join below) IS the topology, ranks assigned in admission order exactly as dial-in membership always did (the cloud shape, where worker addresses do not exist before the VMs boot). Discovery requires an explicit min_rank_start (there is no configured capacity to derive a quorum from) and leaves the early-close target unset unless asked, so the window stays open its full length for stragglers. Fan-out and discovery compose: an enumerated rig fans out as usual while walk-ins self-register into the same window. Two trust knobs round out the admission story: join.token is a pre-shared session salt (hex) replacing the per-run generated one, so a credential injected at fleet-create time can be presented by walk-ins; it forces credential-checked admission even behind sshd (the guardrails and the token are layers, not alternatives) and contradicts open_admission: true loudly; join.tunnel_only binds the controller loopback-only with no roster to infer that from, making an sshd-carried forward the ONLY road in: reachability itself is the authentication (CPU averaging modes only, like any tunneled worker). Walk-ins admitted through a loopback bind synthesize tunnel: true so their rank children dial the loopback end of the forward rather than an unreachable config address.

  • The staging hold: controller.join.start: manual (or hybrid) turns the join window into an operator surface, and fdl start fires it. Auto stays the default: the clock closes the window (target / expiry) as always. Under manual, quorum met puts the run in a new staging phase (visible in fdl status, which renders "roster startable, fire with fdl start"): the roster is held, window expiry keeps holding, and only the operator's start (or the hard cap, loudly, with a raise-the-cap hint) ends the hold; target_ranks is refused up front since a closing target contradicts an operator-owned close. Hybrid keeps both: the clock still closes, and the operator may fire earlier. This is the scavenged-credit shape: launch instances until the money runs out, watch fdl status, start when the roster looks full enough. The switch itself is an authenticated POST /start on the same single training port (the mux routes the request line's POST like a channel magic): from the controller box or through the sshd tunnel the loopback peer address is the credential, from anywhere else ?token= must match the run's session salt. Refusals name their reason (auto mode, quorum not met with counts, window already closed), and arming below quorum is refused rather than queued; the verdict loop stays the authority (quorum is re-checked at the fire poll).

  • fdl join: the worker-side walk-in. fdl join [<controller>] --token <hex> --bin <training-binary> -- <its args> dials a window, offers the box's GPUs (--devices to scope), and runs the binary in agent role: it joins, then spawns and supervises the host's relay and rank children itself, byte-identical to the fan-out path from there. --ssh [user@]host[:port] brings up a local ssh -L forward of the controller port (fresh per attempt, ExitOnForwardFailure, never a password prompt) and dials through it, the guardrailed-sshd trust path; the agent bootstrap spec rides the child's environment, never argv, the same salt hygiene as fan-out. Every flag defaults from a top-level join: block in fdl.yml so a golden image boots into bare fdl join, and --persist re-dials with backoff (5s doubling to 60s) whenever the agent exits (no window yet, run finished, controller rebooted), so a fleet can sit ready before the operator ever launches. When fdl runs inside a project, the active libtorch's lib/ rides LD_LIBRARY_PATH onto the binary automatically (FDL_LIBTORCH_CASE honored) and its variant label rides the join hello. Two structural pieces landed with it: worker agents now rewrite the controller address in their formed-world artifacts (envelope + relay spec) with the address their own join provably reached, because a walk-in's ephemeral-port tunnel or a NAT'd controller makes the controller-authored address plain wrong on that host, and the relay dying on it used to take the whole host down; and the launcher learned that walk-in workers are nobody's process children, so run completion now waits on every host control link's EOF (a discovery cohort can have ZERO process children, and supervision returning instantly used to tear the world down moments after forming it), while a failure verdict actively sends each connected host an Abort down that link instead of leaving agents nobody told to die.

  • The join sshd guardrail recipe (docs/ddp/02-cluster-guide.md): what to put in front of a controller that accepts walk-ins from outside the private network. One authorized_keys line, restrict,port-forwarding,permitopen="127.0.0.1:1337",command="/usr/sbin/nologin", reduces a compromised join key to exactly one capability: a port forward to the controller mux and nothing else (no shell, no pty, no agent forwarding, no other destination; verified live: the same reachable port answers directly and returns nothing through the restricted forward). The forced command covers subsystem requests too, so that line also refuses sftp and rsync while leaving ssh -N untouched, so the recipe now spells out what a key must carry when the same box also serves the walk-in's data mount (internal-sftp -R, read-only but scoped only by the user unless chrooted) or a source pull (rrsync -ro, directory-scoped), and that one key cannot do both. For permanent setups the same restrictions repeat at the daemon level under a dedicated no-shell user (Match User block) so a mistake in either layer is caught by the other; the ephemeral variant is a dedicated sshd instance whose lifetime brackets the run: own config, own port, an authorized_keys carrying only join keys, no root needed above 1024. One asymmetry the recipe spells out: established forwards ARE the tunneled workers' transport, so bounding the window means disabling new admissions after formation, and tearing the sshd down only once the run is over.

  • fdl join prepares the box before it dials, and classes failures by what retrying can fix. Preparation runs per attempt, strictly before the tunnel, because admission starts a window deadline: a GPU gate (no usable device means no dial: the agent's own check comes only after admission, when the host is already counted into a quorum and its failure takes the cohort's formation with it), the dataset source root (join.data_path, and join.data_source: sshfs://[user@]host[:port]/abs/path to mount it there, READ-ONLY, so the kernel enforces the ranks-never-write-the-source-root invariant instead of flodl merely obeying it; an already-mounted root is reused), and write-proofs of the node-local directories the data plane uses (the ~/.flodl/data cache and the temp stage, proven by writing: permissions, ACLs, a full disk and a read-only mount all present identically to a write). Transient failures (controller not up, mount attempt, tunnel) exit 1 and re-dial under --persist; permanent ones (no GPU, an unparseable spec, a missing tool, an unwritable directory) exit 2 and do not, so a misprovisioned instance stops instead of hot-looping. fdl never powers a box off itself: the shipped systemd recipe pairs 2 with RestartPreventExitStatus=2 + FailureAction=poweroff, and the recipe is written for --persist (one-shot mode passes the agent's own exit code through, so 2 is reserved there).

  • A walk-in can build its training binary instead of being handed one (join.source: + join.libtorch:). The tree always lands on LOCAL disk first (cargo stats every source file on every invocation, so a mount build pays that latency thousands of times, and the attribute caching that would hide it makes cargo serve a stale binary), with mtimes preserved so a re-dial costs the changed files and nothing else; the build links against the exact libtorch the box holds, which makes the ABI match by construction rather than by manifest discipline. Three transports, and the scheme names the tool: file:///abs/path, rsync://[user@]host[:port]:/abs/path (the one that carries UNCOMMITTED work, which a training crate living in no repo needs), and git+https:// / git+ssh:// with a required #<tag|branch|sha> (a default branch floats, and a floating ref is not a pin; a bare sha needs server-side allowReachableSHA1InWant and errors by name rather than falling back to a full clone). join.libtorch: auto|cpu|cu126|cu128|rocm7.0 acquires into ~/.flodl/libtorch/ and activates it there, never the project tree (often a read-only mount on a walk-in); auto routes on the box's own devices, so one golden image serves NVIDIA and AMD instances. bin: and source: are mutually exclusive, loudly, and a declared bin: is the escape hatch that lets a borrowed box keep the last word on what it compiles. There is no toolchain field on purpose: the tree carries its own rust-toolchain.toml and lockfile, and RUSTUP_TOOLCHAIN set by fdl would silently override the operator's pin. Source that does not compile is TRANSIENT (the fix is a push away at the source; the box picks it up on its next dial), with one exception: a build that failed while the vendor's toolkit headers are demonstrably missing goes permanent, because re-dialing cannot install a package, and the error names the apt line.

  • fdl publish: the controller is the authority for what a run is, with no protocol. It resolves a source spec into <served>/tree (default ~/.flodl/run/tree), builds it once as a gate, and writes the run manifest .fdl-run.yml at the tree root, so one worker fetch carries source and recipe together: a worker pointed at a published tree needs nothing but the pointer, and cwd / build / bin / args come from the manifest. args is the sharp case and the reason this is correctness rather than ergonomics: rank children re-enter the binary with them, so a standing fleet holding its own copy trains the next run with the previous one's hyperparameters. The manifest's PRESENCE is the commit point: publish removes it before touching the tree and writes it only after the gate passes, so a box dialing mid-publish (or after a failed gate) finds none and waits, and "no manifest and no local artifact" is transient on the worker side. The build is validation, not an artifact: every worker still compiles its own (a controller producing binaries for N variants is the build matrix this design deleted), and the gate needs no GPU libtorch (fdl libtorch download --cpu is enough, and --features "$FDL_GPU_FEATURE" expands empty on a CPU variant, so one recipe line serves the gate and both worker vendors). It proves the controller's variant only: a break that exists solely under --features rocm still lands on a worker. Chaining runs on a standing fleet is one command: publish again, and every box picks the new run up on its next dial with nothing to edit anywhere. The report prints the worker source.from in BOTH spellings, labelled by the key that serves them, because rrsync re-roots every requested path under its own directory: following an absolute tree path and an rrsync -ro guardrail line at the same time double-roots and fails, so the recipe has to say which spelling pairs with which key.

  • Datasets route through a declared source root, and a rank never writes it. cluster.workers[].data_path (and a walk-in's join.data_path) now reaches every rank on that host and supplies the training binary's data directory, so declaring it changes a run, not just the pre-flight checks; undeclared ships nothing and the binary keeps its own default. Underneath, flodl::data::host_cache is the across-run tier that owns the invariant: the source root is read-only, anything missing is acquired into the node-local cache (~/.flodl/data) through a caller-supplied closure (flodl never learns a URL) and published by rename (resolve_cached, data_cache_dir, publish_atomically); ranks read the effective root via distributed::cluster_data_path.

  • A data pass can be sliced into several training events: epoch_splits on DdpRunConfig / Trainer::builder().with_epoch_splits(n) runs num_epochs × n shorter events over the same epoch permutation and LR schedule, so cadence, eval election and checkpoint boundaries land at sub-pass granularity without changing what a pass covers. Checkpoint format v5 records it (CoverageBlock.epoch_splits) and a resume whose slicing changed is refused up front instead of silently re-slicing coverage. ddp-bench --epoch-splits exposes it.

  • Mixed GPU vendors are refused at the join window when the data plane is NCCL/RCCL. The two libraries export the same symbols and the same 128-byte unique-id format, so nothing structural rejects a mixed cohort: it hangs (or dies opaquely) inside ncclCommInitRank at formation, after the window deadline was spent. The hello's libtorch variant label already said the vendor, so admission now checks it: the first classifiable member seeds the cohort's vendor (the same first-member pattern as the dataset signature) and a mismatch is rejected naming both labels and both fixes (a CPU ElChe mode, which genuinely works cross-vendor and therefore disables the gate, or a one-vendor fleet). Labels that classify to no vendor gate nothing, and the check deliberately does not seed from the controller's own build: a CUDA-built coordinator-only controller in front of an all-AMD RCCL cohort is a legitimate shape.

  • Run identity, NCCL version, and arch coverage join the coherence checks: the "arch/libtorch coherence + per-host prechecks" the membership design promised, landed where each fact actually lives. fdl publish stamps a fresh nonce (run: in .fdl-run.yml) every publish; it rides each worker's join hello, and the window refuses a cohort whose members hold different ids, because two boxes that fetched across a publish boundary would train two different runs (different args at minimum, and rank children re-enter the binary with them) as one world. A nonce rather than a content hash on purpose: the common re-publish is "same args, new code", which a manifest hash would call identical. The hello also carries the (major, minor, patch) of the NCCL/RCCL library the worker's binary actually LOADS (ncclGetVersion through the shim, LD_PRELOAD-aware, no CUDA context touched), and admission refuses major.minor skew under an NCCL data plane: skew refuses the handshake at formation, after the window was spent, and a walk-in fleet has no roster for fdl probe's skew scan to sweep, so the window is the only place the check can live. Both are first-member-seeded consistency among joiners like the vendor gate and the dataset signature; boxes carrying no id or no version (a --bin box, a CPU build) gate nothing. The third check is worker-side, in fdl join's prepare: the resolved libtorch's .arch coverage is tested against the cards this box OFFERS, so a Pascal-class box holding a cu128-only build is refused before the dial instead of dying at its first GPU op (no kernel image is available) after admission counted it into a quorum, scoped to the variant's own vendor and to --devices, and a variant without metadata gates nothing (probe owns that complaint). Controller-side per-host prechecks were deliberately NOT built: the controller cannot reach back through a NAT'd tunnel, and the worker has the filesystem. CONTROL_PROTOCOL_VERSION 2 → 3 (the hello grew fields and bincode is not self-describing; the bump moves a mixed-version cohort's failure to the handshake, named).

  • A mixed-model formation is refused by name instead of hanging at the first collective. Every rank hashes the model it actually constructed (parameter and buffer names, shapes and dtypes, in declaration order, since averaging is position-addressed) and carries the signature in its control-channel handshake; the per-host relay forwards them, and the coordinator compares across the cohort before acknowledging. On a mismatch, formation aborts naming both (host, rank) pairs and the usual causes (a stale source tree, a wrong bin: on one box), where previously NCCL would hang on mismatched buffer sizes and CPU averaging would garble frames, after the join window was already spent. The signature is a within-cohort equality token, never persisted, deliberately not a hash of the binary (which legitimately differs per host: libtorch variant, GPU arch, rustc) nor of a source tree (only meaningful when every box built the same one). CONTROL_PROTOCOL_VERSION 3 → 4.

  • Walk-ins are checked for the same model signature at admission, where a mismatch costs only their own dial. fdl join probes the resolved binary before the tunnel (a short CPU-only re-run: with FLODL_INTERNAL_MODEL_SIG_PROBE set, Trainer::run prints the signature and exits before auto-promote, touching no CUDA); the signature rides the join hello, and the window refuses a box building a different model the way it refuses a dataset or run-identity mismatch, so the box re-dials under --persist once fixed, instead of the formed cohort dying at the backstop above. The ledger's expectation is seeded by the controller's own CPU-built model (piggybacking on the schema capture it already does; its model IS the run's truth, so the very first mismatching walk-in is refused, no first-member luck), with first-member seeding as the fallback. The probe is best-effort by design: skipped (--no-sig-probe / join.sig_probe: false), failed, or timed out (a binary built against a flodl that predates the probe runs its whole main; a 120s ceiling bounds it) all join without a signature, loudly, and gate nothing. Its OUTCOME is cached across --persist re-dials, keyed on a digest of the probe recipe (binary path, mtime, size, the run's args), so an idle box re-dialing at the backoff cap pays the probe (or its timeout) once per actual change, not once per tick; a rebuild or a re-publish invalidates the key by construction. A binary that exits non-zero under the probe earns the loudest warning: rank children re-enter it with the same arguments after admission, so that failure usually takes the cohort's formation with it, so the probe surfaces it before the dial. Exit 0 with no signature has two very different causes — a binary older than the contract, or one that failed before reaching the trainer while exiting 0 anyway, which a read-only project directory produces on any binary that creates an output directory in main — so the probe names both and quotes the binary's last lines rather than guessing.

  • docs/windows-wsl.md (WSL2 setup, CUDA and the honest state of AMD under WSL) with its site page, and rocm-capture.sh: a one-pass support dump of an AMD host's GPU stack (KFD topology, ROCm userspace, device permissions, both sides of the container boundary), for the box that detection cannot explain from afar.

  • gpu_ram_share is declarable in cluster config: the integrated-GPU host-RAM share (the fraction of MemTotal an APU's aperture claims) was code-only (TrainerConfig / DdpRunConfig / DataLoaderBuilder), so on a cluster every host got whatever single value the binary hardcoded. It is host-hardware truth, so it travels the data_path road, not the manifest: per-host cluster.workers[].gpu_ram_share, a cluster-scope cluster.gpu_ram_share default (the one-line answer for a farm of identical APU boxes, legal on mixed fleets because discrete-GPU hosts ignore it), and join.gpu_ram_share / --gpu-ram-share for a walk-in. Each later road knows the box better and wins. The value fills the training binary's config only when that left the knob unset; explicit code keeps the last word, like a passed --data-dir. Resolution happens at ONE point (the slim-envelope build), a walk-in's own declaration rewrites it at envelope localization beside data_path, and ranks read it through LocalCluster::gpu_ram_share(). ClusterBuilder mirrors both scopes. Values above 1.0 stay legal in yml exactly as in code (a platform can under-state the aperture); negatives and non-numbers are refused at config load, named.

  • A standing publish: block in fdl.yml: the controller's answers for fdl publish (source / bin / cwd / build / to / identity / args), so re-publishing a run is one bare command. Flags win field by field; a -- tail replaces the block's args: outright, even empty ("explicitly none" must be sayable, since the args belong to the run). --no-build has no block field on purpose: a standing config that skips the gate would ship every future typo to the fleet.

  • fdl publish --json: the report as JSON on stdout (notes stay on stderr): tree, served dir, built flag, toolchain, args, run id, and BOTH worker source-spec spellings keyed by the serving key they pair with. One report value feeds both renderers, so the machine twin cannot drift from the human text.

  • fdl publish --gate <variant>: repeatable extra check-builds against named libtorch variants (e.g. --gate precompiled/rocm70 on a CUDA controller), closing the "the gate proves only the controller's variant" hole: a --features rocm-only break now dies on the controller instead of on a worker. Each gate runs the same recipe under its own CARGO_TARGET_DIR (per-variant incremental caches; success is the verdict, no artifact convention). Needs no GPU; a flodl-linking crate still needs that vendor's dev headers on the controller, and a gate without them fails loudly with the exact package line.

  • fdl join-config, the once-per-farm wizard. Everything the guardrail recipe asks an operator to assemble by hand, produced in one pass on the controller: the farm overlay fdl.<label>.yml (discovery window, start: manual, admission token stamped; a farm IS an env overlay, so fdl @<label> <cmd> targets it with the machinery that already exists), an ed25519 join key born per farm in ./.fdl/<label>/ (which gitignores itself, since it holds keys; an identity referenced from outside the farm dir draws the cross-farm-reuse warning), the composed authorized_keys line for the chosen door (--door b rrsync source pull / a sftp data mount / nologin) plus the sshd Match hardening block, the paste-ready worker yml in that door's dialect, a publish recipe derived from the training crate's own manifest (a path dep on flodl walks source: up to the dep root with cwd: pointing back down; a registry dep ships the crate dir alone; --features "$FDL_GPU_FEATURE" only when the manifest declares the vendor features; a workspace above the crate earns an explicit bin: caveat instead of a silent wrong guess), and a lockfile-vs-source freshness report. On an existing overlay only the token: line is ever touched, byte-preservingly; a user-authored overlay without one is never edited. The scaffold also wires a commands: entry, because a join window only opens for a command running in launcher mode and cluster: true is what puts it there: without one, fdl @<label> <cmd> resolves the base command and trains locally, silently, since training on that box is a legitimate thing to do (the entry is named after the training crate when one is readable, commented when it would be a guess, and a non-cluster command run under a discovery overlay warns for the same reason). It also guides the setup rather than only producing artifacts: a preflight names what THIS box is still missing for the chosen door before anything is written (an ssh daemon, something listening on the door port, rrsync for door b or an sftp server for door a, the served directory, and the two traps that produce no useful error on their own — Debian's ssh.socket owning the listener so sshd_config's Port is ignored, and SELinux refusing a non-standard ssh port), each with the fix spelled in this platform's own commands (apt/dnf/brew, ssh vs sshd units, ufw vs firewalld, semanage). It reports which environment it inspected, because that changes what a fix MEANS: inside a container the findings describe the container and a package installed into a running one is gone after the next --rm, so the durable answer is the image; on the host, a project whose commands carry docker: builds and serves THERE, so the fix is offered in its docker compose exec <service> form with the services named. It writes a ready-to-install sshd_config.d drop-in into the farm dir and renders the whole setup as numbered steps whose commands READ those files rather than repeating their contents, ending with a door self-test that is specific to the door (every door must refuse a shell and permit the forward; only b must also list the served tree, only a must open sftp). The drop-in scopes its guardrail with Match LocalPort, not Match User: binding it to the exposed port confines every key that arrives there, including ones added later, while leaving port 22 and the operator's own logins alone — and it emits ForceCommand only for the tunnel-only door, since doors a and b carry their command in the key line and a daemon-level one would silently override it. Re-runs reuse credentials AND the farm's shape: door and controller are recovered from the farm's own worker yml rather than falling back to flag defaults, so reprinting the authorized_keys line cannot re-render the farm around it; naming a flag is how you intend a change. --regen rotates key and token together. --json is the machine twin, secrets as file paths only. The token draws from OS entropy via getrandom, flodl-cli's one non-serde dependency, added because an entropy binding is the dependency class you never hand-roll (the previous /dev/urandom read fell back to time-xor-pid SILENTLY, which on a box without the device (native Windows) meant a guessable admission token as the norm); a credential mint with no entropy now refuses loudly instead of degrading, while the publish nonce keeps its fallback (its bar is uniqueness, not secrecy).

  • The wizard installs its own authorized_keys line, consent-gated. The composed guardrail line is the artifact most likely to be mangled by hand, by exactly the audience least equipped to debug an sshd refusal. Consent is explicit only: the tty prompt or --install-key; --yes deliberately does not count (it accepts ordinary defaults, and a security-relevant mutation is not one). Only the wizard's own line is ever touched: identity is the public key material (quote-aware against command="…" options), foreign lines are preserved byte for byte, an existing install with different options is shown and replaced on confirm, permissions (~/.ssh 700, authorized_keys 600) are reported with the exact chmod and fixed on confirm, the rewrite is atomic, a symlinked authorized_keys is refused by name, and /etc/ssh is never edited. After installing, the wizard checks something is listening where workers will dial and says so if not (macOS: Remote Login hint). --authorized-keys <path> names a door the default cannot reach at all: an sshd in a container reads a bind-mounted key file, writable only from the host while the in-container run sees a read-only filesystem, so neither identity can install it; a host using AuthorizedKeysFile /etc/ssh/authorized_keys.d/%u has the same shape. Every other rule holds, /etc/ssh is still refused by name, and the named file's parent directory is left exactly as found, since the 0700 rule belongs to a ~/.ssh the wizard may have to create rather than to a layout the operator already owns.

  • fdl join-config --cloud-init: a user-data file embedding the worker yml, the private key and the systemd recipe (Restart=always + RestartPreventExitStatus=2 + FailureAction=poweroff, the failure taxonomy end to end), so a cloud instance boots straight into a persistent fdl join. It assumes only a shell, systemd and a network: fdl is fetched on first boot before the unit it enables can start, and the declared door's own tooling comes with it (door b builds a source tree on the box, so a compiler and the transports; door a mounts the data root, so sshfs, whose absence prepare classes permanent, which under this very unit means exit 2 and a halt over a missing package). Every step is guarded by command -v, so anything baked into the image wins. A Rust toolchain installs as the service user rather than system-wide, since cargo writes its registry cache into CARGO_HOME, and the unit carries the matching PATH. --cloud-init-user root is wired for the images that log in as root (DigitalOcean and the AMD Developer Cloud on top of it), including root's real home at /root. The file states what the halt does and does not buy: where a provider bills powered-off instances, FailureAction=poweroff stops the work but not the meter, and only destroying the instance stops the bill. A SECRET artifact from the moment it exists; written 0600 into the farm dir, never printed.

  • inherit-from: grew ~ expansion, refuses remote parents by name, and is finally user-documented. inherit-from: ~/.flodl/farms/b300.yml now resolves against the invoking user's home (it previously mis-resolved relative to the declaring file), which is what lets an ephemeral working directory carry a two-line farm file while the shared definition lives wherever the operator keeps it. A ://-shaped parent is refused loudly as reserved grammar: a remote parent is config that could change under a standing fleet between two invocations, so it waits for pinning to be designed. The mechanism itself (chains, declaring-file-relative paths, cycle detection, fdl config show provenance) had shipped earlier with design-doc-only coverage; docs/cli/05-manifest.md now documents it.

Changed

  • Formatting is rustfmt's now, enforced in CI. Adopted for a concrete reason rather than conformity: cargo clippy --fix applies span-based text edits and leaves layout to rustfmt by design, so on a hand-formatted tree it produces correct code at the wrong indentation, and every mechanical fix needs a manual pass afterwards. That surfaced when correcting the MSRV unmasked 171 collapsible_if suggestions (the lint is MSRV-gated and had been silent while the declared floor forbade let-chains). One reformat, 385 files, and the 3026-test suite passes unchanged, which is what makes it a formatting change rather than a rewrite. Doc comments are untouched: rustfmt does not reflow comments by default. The commit is listed in .git-blame-ignore-revs.

  • bincode replaced by bincode-next, and supply-chain checks added. bincode 2 is unmaintained (RUSTSEC-2025-0141) and its 3.0.0 on crates.io is a tombstone: a README plus a lib.rs containing only a compiler error, published because crates.io cannot archive a crate. bincode-next is the continuation, and the swap changes nothing observable: the same API, and a byte-identical wire encoding that each version decodes from the other (verified across u32, String, Vec<i64> and Option<f32>), so CONTROL_PROTOCOL_VERSION is unchanged and no cohort can split on it. bincode is used for the distributed wire protocol only, never for anything on disk. A new deny CI job runs cargo-deny over advisories, licenses, sources and duplicate versions; the policy and the reasoning behind each choice live in deny.toml, including why MPL-2.0 is allowed for one dependency and why unmaintained advisories fire for direct dependencies only.

  • flodl-hf's HuggingFace stack moved to hf-hub 1.0, safetensors 0.8 and tokenizers 0.23 (from 0.4 / 0.4 / 0.20). The two visible consequences are worth reading before upgrading. Hub repo ids must be owner-qualified. The Hub answers a bare legacy name (the pre-2024 short form, no owner segment) with a 307 to its owner-qualified home on the metadata endpoint; hf-hub 0.4's ureq transport followed that redirect and 1.0 does not, reporting Repository not found instead. BertModel::from_pretrained("bert-base-uncased") therefore needs "google-bert/bert-base-uncased", and likewise FacebookAI/roberta-base, FacebookAI/xlm-roberta-base, distilbert/distilbert-base-uncased, albert/albert-base-v2. Owner-qualified is also the form the Hub itself now documents, and every id in this crate, its examples, its docs and its test matrix moved to it. The MSRV consequence is not optional: hf-hub 1.0 depends on hf-xet unconditionally (no feature gates it), which reaches xet-core-structures and its 1.91 floor, so the whole workspace floor moves with it. Reverting to 0.4 was the alternative considered and declined: it is the unmaintained line, and paying the floor once is cheaper than porting away from a stale Hub client later.

  • Every workspace dependency is declared once, in the root [workspace.dependencies]. Members say dep.workspace = true and add only what is theirs. It guards one specific drift, which had already happened: two members declaring the same dependency at different versions, flodl asking for flodl-hw = "0.7.0" while flodl-cli asked for "=0.7.0", two contracts for one crate. It deliberately does NOT guard a direct dependency drifting from a transitive one (cargo tree -d and cargo-deny's duplicate warnings are what catch that), and the table is full rather than only the shared entries, because a partial one needs a judgement call about which deps qualify and that judgement is itself the drift risk. This changes the release procedure: the internal =X.Y.Z pins that used to be six hardcoded companions across three member manifests are now five lines in one table, and docs/release.md is updated accordingly.

  • The CLI and DDP references are split into chaptered directories. docs/cli.md (1568 lines) became docs/cli/01-install.md through 06-source-checkout.md, and docs/ddp.md became docs/ddp/01-reference.md plus a cluster guide, an internals chapter and a troubleshooting chapter; docs/README.md is the new index and docs/distributed/architecture.md was rewritten. Both single files had passed the point where a reader could find anything in them, and the cluster surface added since 0.7.0 would have doubled them again. Direct links to docs/cli.md and docs/ddp.md no longer resolve; the site permalinks are unchanged, and ci/release/03-lint-docs.sh now validates every link, heading anchor and /guide/ URL in the tree, which is what makes a split of this size safe to do again.

  • The cuda-* fdl commands are gpu-*: fdl gpu-test, gpu-test-all, gpu-test-nccl, gpu-test-serial, gpu-build, gpu-clippy, gpu-shell. The cargo feature they pass derives from the ACTIVE libtorch variant (cu…/sm… builds --features cuda, rocm…/gfx… builds --features rocm) via $FDL_GPU_FEATURE, so the names say gpu because that is what they now mean. Anything calling fdl cuda-test (CI, scripts, muscle memory) needs the new spelling. The ddp-bench and hf-ddp vehicle manifests moved with it: docker: gpu + --features "$FDL_GPU_FEATURE" instead of a hardcoded cuda. The variable reaches every place a recipe can run: native (non-docker) commands get the libtorch env exactly as the compose passthrough provides it in docker mode, all six scaffolded compose services list it under environment: (compose passes only what is listed), and it expands empty on a CPU variant so one quoted --features "$FDL_GPU_FEATURE" line serves a CPU gate and both worker vendors.

  • The GPU API is vendor-neutral: cuda_* to gpu_*, Cuda* to Gpu*, C ABI flodl_cuda_* to flodl_gpu_*. Deprecated aliases for all 33 renamed items ship in flodl::compat, re-exported from the crate root and from tensor/nn, so existing paths keep resolving. flodl_nccl_* (RCCL exports the nccl names verbatim), set_cudnn_benchmark (routed to MIOpen on ROCm) and cuda_compute_capability (an NVIDIA-only concept, and an error on ROCm) keep their names.

  • Feature flags name the axis they select: gpu is the vendor-neutral gate, cuda and rocm select a backend and imply it. --features cuda behaves as before. build.rs rejects both vendors at once and gpu without one.

  • detect_gpus() returns only devices the running build can address. detect_gpus_physical() ignores the vendor filter and answers the provisioning question (which libtorch variant covers this box).

  • Integrated GPUs no longer double-count memory. An APU's aperture is carved out of host RAM, so the data plane reserves it once, subtracting unrealized headroom only. gpu_ram_share on DataLoaderBuilder, TrainerConfig and DdpRunConfig overrides the fraction, and is required on a multi-socket APU, where each package carries its own aperture while /proc/meminfo is system-wide and the kernel exposes no per-package equivalent to correct with. That case is refused as an ordinary error from DataLoaderBuilder::build / rank-worker construction, not from inside the per-epoch arithmetic: every input to the verdict (the knob, the socket count, whether the device is integrated) is static for the process, so it is decided once where a Result already exists. The gate counts CPU packages, not NUMA nodes: NPS and Sub-NUMA Clustering split one socket's memory into several nodes without multiplying the aperture, so a single-socket machine keeps sizing itself whatever its NUMA layout.

  • The system ROCm runtime precedes libtorch's bundled copy on LD_LIBRARY_PATH in cluster pre-build, remote dispatch, the container images, and the native-build recipe fdl setup prints (which honors $ROCM_PATH, defaulting to /opt/rocm). libtorch-rocm bundles the whole userspace ROCm stack, and letting it win over the host's segfaults at the first GPU op, so a recipe printed the other way round is that configuration handed to the user to paste.

  • fdl diagnose reports GPUs by vendor and prints the sweep's findings. The section was headed CUDA and said "No CUDA devices available" on a box whose cards are AMD; an AMD GPU present with no ROCm runtime installed, a device the sweep deliberately does not count, now surfaces there as well as in fdl probe.

  • Lint gates deny instead of warn. -W clippy::all is a no-op, since clippy::all is warn-by-default: every clippy step in CI, in fdl clippy / fdl gpu-clippy, and in the fdl init scaffold could only fail on a compile error the build already covered. They now pass -D warnings (-D clippy::all in the scaffold, so a half-written function does not block the lint run), and clippy runs ahead of the build in all three CI jobs, which costs nothing because the two share dependency rlibs and compile the workspace twice whichever order they run in.

  • The pinned consensus decode is now RAM-neutral: the reply decodes back into the snapshot staging the rank just streamed from, instead of a second locked model copy. The 0.7.0 pinned decode bought a truly-async H2D writeback at the cost of one page-locked f32 model copy per rank, measured on a RAM-tight host as a net regression (locked pages are unswappable, so the kernel evicted the per-window working set instead: +6–8% epoch wall, all of it swap thrash inside the reduce windows), which the MemAvailable gate could only answer by giving that host up entirely. The observation that removes the trade-off: the reply's frame mirrors the schema of the snapshot the rank just sent, and the snapshot staging's bytes are dead the moment the streamed encode has read them, so the client now accepts the request tensors themselves as decode destinations (CpuReduceClient::set_decode_into_request / arm_decode_into), and the consensus lands in the same pinned storage load_averaged then reads, under the same single-consumer fence chain the staging already carried. Zero marginal locked memory, no affordability gate: RAM-tight hosts get the async writeback back instead of the fresh-alloc fallback (rig-verified at the motivating A/B's anchor invocation: epoch wall 289.2s vs 290.2–292.8s baseline and 308.5–315.9s for the gated predecessor's ungated arm, with swap growth collapsing from +2.0GB to +0.8MB). Two hazards this surfaced are guarded: a zero-mass round's reply (meaningless zeros the caller keep-locals over) must not clobber the very tensors keep-local returns (the realized-work weight rides the wire ahead of the payloads precisely so the decode can fall back to fresh allocs before touching the staging), and a degenerate snapshot fallback shipping non-CPU or dtype-mismatched tensors is refused per payload rather than written through. Under bf16_wire the reply lands VERBATIM in the bf16 snapshot staging (no information loss: the controller folds in f32 and the wire is bf16 either way) and the writeback stages through a flat device-side bf16 bounce, because libtorch's copy_ casts on the source device for cross-device + cross-dtype copies: a direct f32-param ← bf16-pinned copy would materialize a pageable f32 temp on the host and silently degrade to a synchronous bounce. Net effect for bf16 cohorts: half the writeback H2D bytes on top of the zero-locked-RAM property; the weight-divergence triple keeps full f32 precision via a per-tensor path (never a model-sized upcast transient). The slot-based decode staging, its DECODE_SLOT_* tags, and the gate's decode role are retired (pinned_decode_affordable stays, documented as historical: the reasoning applies to the snapshot staging a future RAM-tight host may need to interrogate).

  • Structural zeros never cross the cluster wire anymore: a payload can declare itself all-zeros (nbytes = 0, schema intact) and every tier folds it without materializing it. The 0.7.0 formation-broadcast fix stopped materializing the non-root zeros model (~762MB of RAM per rank at 190M params) but still shipped the zero bytes: every non-root rank pushed a full model of zeros up the wire, HMAC-SHA256'd on both ends, decoded and accumulated by the relay fold: pure overhead scaling with model size × (N−1) ranks, at formation time on every multi-host run, and worst exactly where the CPU plane aims (slow inter-node links). Three senders know their payload is structurally zeros without scanning anything, and all three now declare it instead of shipping it: the formation broadcast's non-root ranks, an idle rank's zero-mass window contribution (γ-mass 0 fused as scale-by-zero, a full model of ±0.0 per idle window before), and the controller's degenerate zero-mass scatter. Elision is a member of the fold monoid: zeros + zeros = zeros, so an all-elided round stays elided through the relay fold (seed→sums promotion is now lazy, firing only at the first non-elided deposit), through sum_frames, and back out the scatter: it composes through fold tiers the same way the mass algebra does, hierarchical-relay-ready. The fold reads elided as "adds nothing", which is IEEE-exact, with one deliberate semantic improvement: a NaN-poisoned idle rank can no longer forward 0.0 × NaN = NaN into the consensus, because an elided contribution is true zeros. Receivers decode elided payloads to zeros of the declared shape (covering the zero-mass scatter), and the checkpoint forge skips zero-mass rounds the same way ranks do, since the consensus checkpoint is a consensus consumer too. PROTOCOL_VERSION 2 → 3: a v2 reader would reject an elided frame loudly anyway (schema error at the fold), but the bump moves a mixed-version cohort's failure to the handshake (first contact, named cause) instead of the first idle window.

Fixed

  • The declared minimum Rust version was wrong, and nothing checked it. rust-version said 1.85 while the source has used let-chains since before 0.7.0, which stabilised in 1.88; the transitive icu_* crates independently require 1.86. Cargo trusts that field, so a user on 1.85 got past the version check and then hit error[E0658]: let expressions in this position are unstable in someone else's crate rather than a refusal naming the version they needed. It is now 1.91, measured rather than assumed, and three floors stack to get there: 1.88 is our own source's (let-chains), bincode-next raises it to 1.90, and xet-core-structures to 1.91. A new msrv CI job compiles the workspace on exactly the toolchain Cargo.toml declares, reading the value from the manifest so the two cannot drift, and it is what caught the third one: xet-core-structures arrives through hf-hub 1.0's mandatory hf-xet dependency, declares no rust-version of its own, and calls str::floor_char_boundary, so cargo could neither warn nor resolve around it, and MSRV-aware resolution had nothing to act on. It simply failed to compile, in a transitive crate, on the declared floor. Note this floor tracks the ecosystem for the same reason: with no committed lockfile, consumers resolve fresh, so a transitive dependency raising its own rust-version raises this one.

  • A native project built on macOS and then refused to start. fdl composed LD_LIBRARY_PATH for native commands, which macOS's loader ignores, so a scaffolded project linked and then died at launch with dyld: Library not loaded: @rpath/libtorch.dylib. Exporting DYLD_LIBRARY_PATH beside it does not fix this: run lines execute through sh -c, /bin/sh is SIP restricted, and dyld purges every DYLD_* from a restricted process before the shell execs anything. An rpath from flodl-sys's build script does not reach the consuming binary either, since cargo:rustc-link-arg does not propagate to dependents. So macOS native commands now pass RUSTFLAGS=-C link-arg=-Wl,-rpath,<variant>/lib, which SIP does not touch and which applies to the final link: the binary locates libtorch itself, and keeps working when launched directly rather than through fdl run. Filled only when absent, so a caller's own RUSTFLAGS still wins, and Linux is untouched.

    Behind that sat a second layer, which the first fix exposed by getting far enough to reach it: upstream's libtorch-macos-arm64 archive BUNDLES lib/libomp.dylib and then has libtorch_cpu.dylib depend on it by absolute Homebrew path, so the load failed on any box without that formula while the library it wanted sat in the same directory as the dylib asking for it. brew install libomp is the wrong answer there: it installs a second, possibly ABI-divergent copy of something already present. fdl libtorch download (and fdl setup through it) now repoints those dependencies at @loader_path/libomp.dylib after extracting, and re-signs, since editing a Mach-O invalidates its signature and arm64 will not load an unsigned one. @loader_path rather than @rpath because the bundled copy is a sibling of every dylib referencing it, so it resolves without depending on the referrer carrying a correct LC_RPATH. The three tools this needs are checked BEFORE anything is modified and the step is advisory, so a box without the command line tools is told to run xcode-select --install rather than left holding a patched library it cannot re-sign.

  • The bootstrap's source fallback could not run where it was most needed. ./fdl verifies a downloaded release binary by running it, and replaces it from source when it cannot run — which is exactly what happens on a distribution whose glibc is older than the build base. But the fallback did cargo build -p flodl-cli in the script's own directory, so it only worked inside a checkout: in the curl-and-run shape, in a scaffolded project, or on a cloud instance that fetched the script and nothing else, it failed with could not find Cargo.toml. It now installs from crates.io when there is no local workspace, which needs nothing but cargo. Verified end to end on a RHEL 9 userland, where the released binary cannot run at all: bare directory, one script, working fdl seven seconds later.

  • The released fdl binaries were built on a floating base. release-cli.yml used ubuntu-latest for the Linux targets, and a dynamically linked binary cannot run where glibc is older than the one it was built against — so the base is a floor on who can use the release, and that floor moved whenever GitHub rotated the image, with no code change and no release note. The Linux runners are pinned, and a gate now reads the artifact's own maximum GLIBC_ requirement and fails the build if it rises above the declared floor: raising it drops users, so it should be a decision rather than a side effect.

  • Every fdl init scaffold since crates.io tightened its API policy carried a floating git dependency. The scaffold prefers the latest published flodl version, but its crates.io query sent no identifying User-Agent, which crates.io's data-access policy rejects, so the "registry unavailable" fallback (flodl = { git = … }, default branch, floating) had silently become the norm on every box, and fdl add flodl-hf then refused the very scaffold fdl init produced ("needs a pinnable crates.io version"). Found by finally running the stranger's path end to end: a throwaway crate on a box that never built flodl. The query now goes through the update check's probe (correct User-Agent, --fail, 2s cap), and the offline fallback pins fdl's own version (fdl and flodl are workspace-versioned twins), so the scaffold is always a pinnable registry dep, never a git one.

  • The native scaffold's printed next steps did not work. ./fdl libtorch download --cpu then ./fdl build failed at flodl-sys's build script: fdl downloaded libtorch but never exported LIBTORCH_PATH for native commands (docker services get it from their compose env; native got a comment telling the user to export it by hand), and a successful build then failed at ./fdl run with libtorch_cpu.so: cannot open shared object file for the same reason on the LD_LIBRARY_PATH side. Native commands now fill both from the active variant (inherited values kept as the override), so download, build and run work with zero hand exports, on the registry crate, verified end to end on two boxes (one with a cold rustup).

  • fdl libtorch download built an invalid URL on Windows. Windows archives carry a -win- infix that the Linux ones do not, and nothing had ever evaluated that arm on a Windows host, so the 404 shipped unnoticed.

  • fdl setup fetched an x86_64 libtorch on Apple Silicon. A Docker-mounted project needs the container's libtorch, which upstream publishes for Linux x86_64 but not for Linux aarch64. Intel Macs still force the Linux build; Apple Silicon gets the host build and a pointer to the wheel-extraction steps, instead of one that cannot load in a linux/arm64 container.

  • A GPU could be reported as covered by a libtorch variant that ships no kernel for it. The compute-capability check substring-searched the raw archs= list, so a bare major matched the minor of an unrelated entry: a Maxwell sm_50/sm_52 card tested its 5 against cu128's 7.0 7.5 …, hit the 5 inside 7.5, and fdl diagnose printed OK for a build whose first kernel launch fails with no kernel image is available for execution on the device. The list is tokenized before matching now. A +PTX suffix on a list entry is understood.

  • fdl setup could finish with no libtorch/.active. The CPU download deliberately does not claim the pointer so a GPU variant fetched after it can, but on a host where no GPU variant follows, nothing ever wrote it, leaving fdl diagnose reporting "no active variant" immediately after a successful setup. CPU now claims the pointer if it is still unclaimed.

  • A record-path collision no longer drops a rank out of the monitor tree: build_tree descends by path segment, and a node claimed as a leaf by one producer and as an interior by another (two producers disagreeing on tiering, which cohort_tiering exists to prevent) had its terminal entry filtered out of the descent while the interior branch was taken anyway. In debug a debug_assert! named it; in release that rank's numbers simply vanished from the tree, with no warning and with the parent's roll-up averaging over the survivors as if the missing rank had never reported: a wrong number presented with full confidence, the same failure class as the absent-is-not-zero defects above. The collision now warns once and the stray's mass folds into its parent's aggregate, so the work-weighted numbers stay whole. It deliberately gets no record of its own: a record's path is its file path in the record log, so emitting it as a child of its own parent would append its lines into the parent's file, so the anomaly is surfaced as a warning rather than papered over with a synthetic path segment that would materialize a real file. Promoting the assert to a release panic was rejected for the reason the record log already swallows its I/O errors: a monitoring artifact must never be able to kill a training run. The sibling case that carried no assert at all is covered too: two leaves claiming one path (the "metrics feed splits in half" hazard named on cohort_tiering) now warn instead of silently first-winning.

  • --record-log says where it actually writes: a relative path resolves against the launcher's cwd rather than the directory it was typed in, which is how ddp-bench/runs/x silently becomes ddp-bench/ddp-bench/runs/x; nothing echoed the destination, so the doubling was found by hunting for the files. The launcher now prints the resolved directory at startup. It joins the cwd rather than canonicalizing, because the directory is created lazily at the first record and canonicalizing a path that does not exist yet fails exactly when the message is most wanted.

  • wire: full zero-elision, structural zeros ship as schema, fold-through by @fab2s in #44
  • ddp: RAM-neutral consensus decode. the reply lands in the snapshot st… by @fab2s in #45
  • distributed: discovery window - roster-free formation by @fab2s in #46
  • distributed: staging hold + fdl start switch by @fab2s in #47
  • distributed: fdl join user-facing command by @fab2s in #48
  • distributed: sshd guardrail recipe + rig validation by @fab2s in #49
  • doc: Correctness and refactor by @fab2s in #50
  • monitor: a colliding record path folds into its parent instead of van… by @fab2s in #51
  • document join yaml by @fab2s in #52
  • Multi vendor support by @fab2s in #53
  • ci: semver-checks + llvm-cov by @fab2s in #54
  • release-prep: changelog the last PR, and fix two defects the docs rev… by @fab2s in #55
  • ci: the rocky cache never once saved, and target had no business in it by @fab2s in #56

Full Changelog: 0.7.0...0.8.0