Repository navigation
Upgrading from 0.7.0? Most of the surface moved by rename with a
deprecated alias left behind, so existing code keeps compiling and
starts warning. Two changes do not warn: the Rust floor is now 1.91,
andflodl-hfneeds owner-qualified Hub repo ids (bert-base-uncased
becomesgoogle-bert/bert-base-uncased) because hf-hub 1.0 no longer
follows the Hub's redirect. See UPGRADE.md for the
step-by-step migration.
Added
-
AMD (ROCm) GPU support.
flodl-syscompiles and links against a hipified libtorch. The vendor is a build-time property (libtorch ships one backend per build) and is reconciled inflodl-sys/gpu_compat.h, which maps the shim's CUDA spelling ontoc10/hip+ATen/hipand routes streams throughHIPStreamMasqueradingAsCUDA. Not yet validated on AMD hardware: this covers compile, link, provisioning and detection only. Provisioning is vendor-plural throughout, including the parts that only a second vendor makes visible: cluster fan-out counts a worker's GPUs for both vendors in one round trip (nvidia-smifor NVIDIA, the KFD topology'svendor_id 4098for AMD, which is mask-proof) and reduces them with that host's declaredarch:, since one libtorch build serves one vendor and that field already names which; an undeclared host reporting both vendors is refused by name rather than guessed at, because the controller cannot know which build it will load and a guess assigns ranks to devices nobody addresses. -
flodl-hw, a dependency-free hardware-detection crate shared byflodlandflodl-cli(flodl::sysandflodl_cli::util::systemare facades over it). AMD devices are read from the kernel's KFD topology, and reported as usable devices only when a ROCm userspace runtime is present; otherwise they surface as a finding naming the arch and what to install. -
fdl libtorch download --rocm 7.0and--rocm 7.1.--path,--no-activateand--dry-runare now listed infdl libtorch -h. The two ROCm builds cover exactly the same gfx targets, so the choice is about the runtime, not the hardware: the HIP ordering rule puts the host's own ROCm ahead of the bundled one, and within a major version that ABI only grows, so a bundle older than the host loads while a newer one can fail on a symbol the host does not have. Auto-detection therefore routes to 7.0, which serves every 7.x host; a box that wants the exact match asks for it by version. Picking the newest bundle the detected host runtime can satisfy is the better rule and waits on ROCm version resolution. The covered gfx list is read off the published archive's own index rather than inferred:gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1150,gfx1151,gfx1200andgfx1201ship rocBLAS kernels, whilegfx900andgfx906(Vega 10 and Vega 20, i.e. MI50 and Radeon VII) carry MIOpen tuning databases and no kernels at all, so they count as uncovered instead of being installed and left to fail at the first matmul. Both ROCm buckets carry the same set. Upstream publishes ROCm builds for Linux only, so--rocmis refused on Windows and macOS with that reason. -
RHEL-family install guidance. The missing-toolkit message from
flodl-sys's build script now prints aRHEL/Fedora: sudo dnf install …line beside the Ubuntu one, andfdl probe/fdl setuppick the dnf spelling on RHEL-family hosts (detected via/etc/os-release). The rpm names are a mechanical transform of the Debian ones (-dev→-devel,g++→gcc-c++), every spelling verified against the vendors' rhel9 repositories: NVIDIA'scuda-rhel9repo and AMD'srocm/rhel9repo, whose setup one-liners the CI toolkit phase now exercises. -
fdl probereports a libtorch variant this host's dynamic linker cannot load. Upstream does not build every archive against the same C library baseline: measured on 2.10.0, the cpu and cu128 trees needGLIBC_2.29while the rocm7.0 tree needsGLIBC_2.35. RHEL 9 ships glibc 2.34 and cannot move past it (libstdc++ is upgradable throughgcc-toolset, glibc is not, since the whole userland links against it), so that pair downloads, compiles, links, and then dies in the loader quoting symbol versions. The probe askslddand names the unmet versions with what they mean, before any of that is spent. Found by the OS matrix's rocky leg on its ROCm rotation;Dockerfile.rheldocuments the baselines as a buildable reference. -
fdl probeandfdl setupreport missing prerequisites with the command to install them: host tools (curl or wget, unzip, a C++ compiler) and the vendor toolkit headers for the active libtorch variant. Presence is decided by capability, not by a list of known paths: when the header is in neither the toolkit root nor the default system include directories, the check asks the compiler that will do the build, with the same-Iit will get, so a box carrying its headers somewhere unusual (or reaching them throughCPATHor a multiarch directory) is not told to install what it already has.flodl-sys's build gate shares the behaviour, so a missing header now names the packages to install instead of failing inside the C++ compile (libtorch bundles the runtime libraries, so headers are the only gap). The command is rendered per family throughout, so a RHEL host is toldsudo dnf install hip-devel …rather than apt with Debian package names. The probe runs only for a header already believed missing, so an ordinary build spawns nothing, and "no compiler to ask" stays a distinct answer from "missing". Setup reports the native prerequisites only when the build is native. -
Per-OS CI matrix (
ci/os-matrix.sh,.github/workflows/os.yml) covering Linux x86_64 and aarch64, macOS arm64 and Windows x86_64. Each host installs libtorch throughfdland builds against it; the Linux leg alternates CUDA and ROCm. -
A RHEL-family box you can actually open (
Dockerfile.rhel, therhelcompose service, commentedrhel-*commands infdl.yml). Now that the tool gives RHEL-specific advice, a RHEL question should be answerable on a RHEL userland rather than by reasoning about one: the dnf package names,rrsyncshipping as non-executable documentation,curl-minimalconflicting withcurl, and which libtorch baselines EL9 can load. Deliberately not what CI uses — the rocky legs run a stock image and install prerequisites the way the docs tell a user to, which is the step a pre-baked image would skip. -
ci/rig-ladder.sh, a multi-GPU validation ladder that cannot report a false pass. The NCCL suites are device-gated (require_multi_gpu()early-returns), so on a one-GPU box the whole suite reportsokin a tenth of a second having tested nothing. The ladder answers what a rig actually EXECUTED: every rung reports RAN or SKIPPED with its reason, a pass requires a zero exit AND the trainer'sdone:line AND no libtorch[W]warnings, and a ladder where nothing ran exits non-zero rather than looking green. Three rungs: single-host process-per-rank, heterogeneous cross-host fan-out, and a walk-in dialing a guardrailed door (controller here, worker joins,fdl startfires the freeze). -
CI runs what it used to only compile. The CUDA and ROCm jobs now RUN flodl's lib suite on their GPU-less runners (the driverless deployment a
--features cuda/rocmbuild meets on any CPU-only box, and it immediately caught two driver-assuming tests); the CPU job asserts the model-sig probe wire contract (theflodl-model-sig:line, exit before training, identical hex across runs) and feedsfdl join-config --cloud-init's artifact through a real YAML parser; and the OS matrix gained a scaffold smoke:fdl init --native, the registry-pin regression guard, then an env-scrubbed./fdl buildand./fdl runagainst the checkout's flodl, proving the scaffold's printed next steps on every compiling host. A rockylinux:9 job (host +docker run, because acontainer:job cannot free host disk, and the rocm rotation needs ~30 GB) runs the same script on the OPPOSITE rotation phase from the ubuntu leg, so every push exercises both vendors, one per distro family;FDL_CI_VARIANTgainedcpuandrotate-altarms. The matrix covers both ends of the supported range rather than only one: Ubuntu 24.04 (the base releases are built against, so the floor users have) alongside 26.04, and two EL releases, each given only variants it can actually run — EL9's glibc 2.34 cannot load the rocm archive at all, so it takes the cpu leg while EL10 (glibc 2.39) carries the RHEL-family vendor rotation. Scheduling a combination that cannot work is not a test, it is a failure that never heals. -
fdl coverageandfdl coverage-all, a coverage report that names what it could not run.coverageis the CPU one-liner and therefore a FLOOR, scoring every GPU-reachable region as missed;coverage-all(ci/coverage.sh) drives the phases itself across the containers they each need, and prints aRAN/SKIPPED/FAILEDroster rather than one number. The roster is the point, and it is the same hazard the rig ladder answers: a hardware- or tool-gated suite returns early and reportsok, so a single percentage can be built entirely out of suites that executed nothing. Coverage here is a number to look at, deliberately never a threshold to clear: the I/O glue inflodl-cliis uncovered on purpose (mockingCommand/ fs / tty is testing the mock), so a floor would push toward exactly the tests this project decided not to write.--skip leakcheckis required rather than tidy: those tests assert RSS growth stays under a bound, and the instrumentation's own counters allocate. -
fdl fmtandfdl fmt-check. Formatting became a CI gate with the rustfmt adoption below, and a gate with no local command means contributors meet it for the first time as a red build.fmtwrites,fmt-checkis byte-for-byte the check CI runs. -
ci/release/11-semver-checks.sh: public-API breakage checked against what is actually published, per crate, automating a step that was being done by eye. It catches a class the test suite structurally cannot, since all 3000 tests compile against the current source and none of them can notice that a name a downstream crate imports has gone. It runs after02-version-sync.shdeliberately: the question is whether the DECLARED version covers the changes made, so running it before the bump correctly fails. Needs no libtorch (DOCS_RS=1, and rustdoc never links); absent tooling printsUNVERIFIEDrather than passing quietly. -
Discovery windows: a cluster can now form with no roster at all, because
controller.join.discovery: truemakes the join window itself define the world.workers:may be empty; whoever walks in during the window (seefdl joinbelow) IS the topology, ranks assigned in admission order exactly as dial-in membership always did (the cloud shape, where worker addresses do not exist before the VMs boot). Discovery requires an explicitmin_rank_start(there is no configured capacity to derive a quorum from) and leaves the early-close target unset unless asked, so the window stays open its full length for stragglers. Fan-out and discovery compose: an enumerated rig fans out as usual while walk-ins self-register into the same window. Two trust knobs round out the admission story:join.tokenis a pre-shared session salt (hex) replacing the per-run generated one, so a credential injected at fleet-create time can be presented by walk-ins; it forces credential-checked admission even behind sshd (the guardrails and the token are layers, not alternatives) and contradictsopen_admission: trueloudly;join.tunnel_onlybinds the controller loopback-only with no roster to infer that from, making an sshd-carried forward the ONLY road in: reachability itself is the authentication (CPU averaging modes only, like any tunneled worker). Walk-ins admitted through a loopback bind synthesizetunnel: trueso their rank children dial the loopback end of the forward rather than an unreachable config address. -
The staging hold:
controller.join.start: manual(orhybrid) turns the join window into an operator surface, andfdl startfires it. Auto stays the default: the clock closes the window (target / expiry) as always. Under manual, quorum met puts the run in a newstagingphase (visible infdl status, which renders "roster startable, fire withfdl start"): the roster is held, window expiry keeps holding, and only the operator's start (or the hard cap, loudly, with a raise-the-cap hint) ends the hold;target_ranksis refused up front since a closing target contradicts an operator-owned close. Hybrid keeps both: the clock still closes, and the operator may fire earlier. This is the scavenged-credit shape: launch instances until the money runs out, watchfdl status, start when the roster looks full enough. The switch itself is an authenticatedPOST /starton the same single training port (the mux routes the request line'sPOSTlike a channel magic): from the controller box or through the sshd tunnel the loopback peer address is the credential, from anywhere else?token=must match the run's session salt. Refusals name their reason (auto mode, quorum not met with counts, window already closed), and arming below quorum is refused rather than queued; the verdict loop stays the authority (quorum is re-checked at the fire poll). -
fdl join: the worker-side walk-in.fdl join [<controller>] --token <hex> --bin <training-binary> -- <its args>dials a window, offers the box's GPUs (--devicesto scope), and runs the binary in agent role: it joins, then spawns and supervises the host's relay and rank children itself, byte-identical to the fan-out path from there.--ssh [user@]host[:port]brings up a localssh -Lforward of the controller port (fresh per attempt,ExitOnForwardFailure, never a password prompt) and dials through it, the guardrailed-sshd trust path; the agent bootstrap spec rides the child's environment, never argv, the same salt hygiene as fan-out. Every flag defaults from a top-leveljoin:block in fdl.yml so a golden image boots into barefdl join, and--persistre-dials with backoff (5s doubling to 60s) whenever the agent exits (no window yet, run finished, controller rebooted), so a fleet can sit ready before the operator ever launches. When fdl runs inside a project, the active libtorch'slib/ridesLD_LIBRARY_PATHonto the binary automatically (FDL_LIBTORCH_CASEhonored) and its variant label rides the join hello. Two structural pieces landed with it: worker agents now rewrite the controller address in their formed-world artifacts (envelope + relay spec) with the address their own join provably reached, because a walk-in's ephemeral-port tunnel or a NAT'd controller makes the controller-authored address plain wrong on that host, and the relay dying on it used to take the whole host down; and the launcher learned that walk-in workers are nobody's process children, so run completion now waits on every host control link's EOF (a discovery cohort can have ZERO process children, and supervision returning instantly used to tear the world down moments after forming it), while a failure verdict actively sends each connected host an Abort down that link instead of leaving agents nobody told to die. -
The join sshd guardrail recipe (docs/ddp/02-cluster-guide.md): what to put in front of a controller that accepts walk-ins from outside the private network. One authorized_keys line,
restrict,port-forwarding,permitopen="127.0.0.1:1337",command="/usr/sbin/nologin", reduces a compromised join key to exactly one capability: a port forward to the controller mux and nothing else (no shell, no pty, no agent forwarding, no other destination; verified live: the same reachable port answers directly and returns nothing through the restricted forward). The forced command covers subsystem requests too, so that line also refuses sftp and rsync while leavingssh -Nuntouched, so the recipe now spells out what a key must carry when the same box also serves the walk-in's data mount (internal-sftp -R, read-only but scoped only by the user unless chrooted) or a source pull (rrsync -ro, directory-scoped), and that one key cannot do both. For permanent setups the same restrictions repeat at the daemon level under a dedicated no-shell user (Match Userblock) so a mistake in either layer is caught by the other; the ephemeral variant is a dedicated sshd instance whose lifetime brackets the run: own config, own port, an authorized_keys carrying only join keys, no root needed above 1024. One asymmetry the recipe spells out: established forwards ARE the tunneled workers' transport, so bounding the window means disabling new admissions after formation, and tearing the sshd down only once the run is over. -
fdl joinprepares the box before it dials, and classes failures by what retrying can fix. Preparation runs per attempt, strictly before the tunnel, because admission starts a window deadline: a GPU gate (no usable device means no dial: the agent's own check comes only after admission, when the host is already counted into a quorum and its failure takes the cohort's formation with it), the dataset source root (join.data_path, andjoin.data_source: sshfs://[user@]host[:port]/abs/pathto mount it there, READ-ONLY, so the kernel enforces the ranks-never-write-the-source-root invariant instead of flodl merely obeying it; an already-mounted root is reused), and write-proofs of the node-local directories the data plane uses (the~/.flodl/datacache and the temp stage, proven by writing: permissions, ACLs, a full disk and a read-only mount all present identically to a write). Transient failures (controller not up, mount attempt, tunnel) exit 1 and re-dial under--persist; permanent ones (no GPU, an unparseable spec, a missing tool, an unwritable directory) exit 2 and do not, so a misprovisioned instance stops instead of hot-looping. fdl never powers a box off itself: the shipped systemd recipe pairs 2 withRestartPreventExitStatus=2+FailureAction=poweroff, and the recipe is written for--persist(one-shot mode passes the agent's own exit code through, so 2 is reserved there). -
A walk-in can build its training binary instead of being handed one (
join.source:+join.libtorch:). The tree always lands on LOCAL disk first (cargo stats every source file on every invocation, so a mount build pays that latency thousands of times, and the attribute caching that would hide it makes cargo serve a stale binary), with mtimes preserved so a re-dial costs the changed files and nothing else; the build links against the exact libtorch the box holds, which makes the ABI match by construction rather than by manifest discipline. Three transports, and the scheme names the tool:file:///abs/path,rsync://[user@]host[:port]:/abs/path(the one that carries UNCOMMITTED work, which a training crate living in no repo needs), andgit+https:///git+ssh://with a required#<tag|branch|sha>(a default branch floats, and a floating ref is not a pin; a bare sha needs server-sideallowReachableSHA1InWantand errors by name rather than falling back to a full clone).join.libtorch: auto|cpu|cu126|cu128|rocm7.0acquires into~/.flodl/libtorch/and activates it there, never the project tree (often a read-only mount on a walk-in);autoroutes on the box's own devices, so one golden image serves NVIDIA and AMD instances.bin:andsource:are mutually exclusive, loudly, and a declaredbin:is the escape hatch that lets a borrowed box keep the last word on what it compiles. There is no toolchain field on purpose: the tree carries its ownrust-toolchain.tomland lockfile, andRUSTUP_TOOLCHAINset by fdl would silently override the operator's pin. Source that does not compile is TRANSIENT (the fix is a push away at the source; the box picks it up on its next dial), with one exception: a build that failed while the vendor's toolkit headers are demonstrably missing goes permanent, because re-dialing cannot install a package, and the error names theaptline. -
fdl publish: the controller is the authority for what a run is, with no protocol. It resolves a source spec into<served>/tree(default~/.flodl/run/tree), builds it once as a gate, and writes the run manifest.fdl-run.ymlat the tree root, so one worker fetch carries source and recipe together: a worker pointed at a published tree needs nothing but the pointer, andcwd/build/bin/argscome from the manifest.argsis the sharp case and the reason this is correctness rather than ergonomics: rank children re-enter the binary with them, so a standing fleet holding its own copy trains the next run with the previous one's hyperparameters. The manifest's PRESENCE is the commit point: publish removes it before touching the tree and writes it only after the gate passes, so a box dialing mid-publish (or after a failed gate) finds none and waits, and "no manifest and no local artifact" is transient on the worker side. The build is validation, not an artifact: every worker still compiles its own (a controller producing binaries for N variants is the build matrix this design deleted), and the gate needs no GPU libtorch (fdl libtorch download --cpuis enough, and--features "$FDL_GPU_FEATURE"expands empty on a CPU variant, so one recipe line serves the gate and both worker vendors). It proves the controller's variant only: a break that exists solely under--features rocmstill lands on a worker. Chaining runs on a standing fleet is one command: publish again, and every box picks the new run up on its next dial with nothing to edit anywhere. The report prints the workersource.fromin BOTH spellings, labelled by the key that serves them, because rrsync re-roots every requested path under its own directory: following an absolute tree path and anrrsync -roguardrail line at the same time double-roots and fails, so the recipe has to say which spelling pairs with which key. -
Datasets route through a declared source root, and a rank never writes it.
cluster.workers[].data_path(and a walk-in'sjoin.data_path) now reaches every rank on that host and supplies the training binary's data directory, so declaring it changes a run, not just the pre-flight checks; undeclared ships nothing and the binary keeps its own default. Underneath,flodl::data::host_cacheis the across-run tier that owns the invariant: the source root is read-only, anything missing is acquired into the node-local cache (~/.flodl/data) through a caller-supplied closure (flodl never learns a URL) and published by rename (resolve_cached,data_cache_dir,publish_atomically); ranks read the effective root viadistributed::cluster_data_path. -
A data pass can be sliced into several training events:
epoch_splitsonDdpRunConfig/Trainer::builder().with_epoch_splits(n)runsnum_epochs × nshorter events over the same epoch permutation and LR schedule, so cadence, eval election and checkpoint boundaries land at sub-pass granularity without changing what a pass covers. Checkpoint format v5 records it (CoverageBlock.epoch_splits) and a resume whose slicing changed is refused up front instead of silently re-slicing coverage.ddp-bench --epoch-splitsexposes it. -
Mixed GPU vendors are refused at the join window when the data plane is NCCL/RCCL. The two libraries export the same symbols and the same 128-byte unique-id format, so nothing structural rejects a mixed cohort: it hangs (or dies opaquely) inside
ncclCommInitRankat formation, after the window deadline was spent. The hello's libtorch variant label already said the vendor, so admission now checks it: the first classifiable member seeds the cohort's vendor (the same first-member pattern as the dataset signature) and a mismatch is rejected naming both labels and both fixes (a CPU ElChe mode, which genuinely works cross-vendor and therefore disables the gate, or a one-vendor fleet). Labels that classify to no vendor gate nothing, and the check deliberately does not seed from the controller's own build: a CUDA-built coordinator-only controller in front of an all-AMD RCCL cohort is a legitimate shape. -
Run identity, NCCL version, and arch coverage join the coherence checks: the "arch/libtorch coherence + per-host prechecks" the membership design promised, landed where each fact actually lives.
fdl publishstamps a fresh nonce (run:in.fdl-run.yml) every publish; it rides each worker's join hello, and the window refuses a cohort whose members hold different ids, because two boxes that fetched across a publish boundary would train two different runs (different args at minimum, and rank children re-enter the binary with them) as one world. A nonce rather than a content hash on purpose: the common re-publish is "same args, new code", which a manifest hash would call identical. The hello also carries the(major, minor, patch)of the NCCL/RCCL library the worker's binary actually LOADS (ncclGetVersionthrough the shim, LD_PRELOAD-aware, no CUDA context touched), and admission refuses major.minor skew under an NCCL data plane: skew refuses the handshake at formation, after the window was spent, and a walk-in fleet has no roster forfdl probe's skew scan to sweep, so the window is the only place the check can live. Both are first-member-seeded consistency among joiners like the vendor gate and the dataset signature; boxes carrying no id or no version (a--binbox, a CPU build) gate nothing. The third check is worker-side, infdl join's prepare: the resolved libtorch's.archcoverage is tested against the cards this box OFFERS, so a Pascal-class box holding a cu128-only build is refused before the dial instead of dying at its first GPU op (no kernel image is available) after admission counted it into a quorum, scoped to the variant's own vendor and to--devices, and a variant without metadata gates nothing (probe owns that complaint). Controller-side per-host prechecks were deliberately NOT built: the controller cannot reach back through a NAT'd tunnel, and the worker has the filesystem.CONTROL_PROTOCOL_VERSION2 → 3 (the hello grew fields and bincode is not self-describing; the bump moves a mixed-version cohort's failure to the handshake, named). -
A mixed-model formation is refused by name instead of hanging at the first collective. Every rank hashes the model it actually constructed (parameter and buffer names, shapes and dtypes, in declaration order, since averaging is position-addressed) and carries the signature in its control-channel handshake; the per-host relay forwards them, and the coordinator compares across the cohort before acknowledging. On a mismatch, formation aborts naming both (host, rank) pairs and the usual causes (a stale source tree, a wrong
bin:on one box), where previously NCCL would hang on mismatched buffer sizes and CPU averaging would garble frames, after the join window was already spent. The signature is a within-cohort equality token, never persisted, deliberately not a hash of the binary (which legitimately differs per host: libtorch variant, GPU arch, rustc) nor of a source tree (only meaningful when every box built the same one).CONTROL_PROTOCOL_VERSION3 → 4. -
Walk-ins are checked for the same model signature at admission, where a mismatch costs only their own dial.
fdl joinprobes the resolved binary before the tunnel (a short CPU-only re-run: withFLODL_INTERNAL_MODEL_SIG_PROBEset,Trainer::runprints the signature and exits before auto-promote, touching no CUDA); the signature rides the join hello, and the window refuses a box building a different model the way it refuses a dataset or run-identity mismatch, so the box re-dials under--persistonce fixed, instead of the formed cohort dying at the backstop above. The ledger's expectation is seeded by the controller's own CPU-built model (piggybacking on the schema capture it already does; its model IS the run's truth, so the very first mismatching walk-in is refused, no first-member luck), with first-member seeding as the fallback. The probe is best-effort by design: skipped (--no-sig-probe/join.sig_probe: false), failed, or timed out (a binary built against a flodl that predates the probe runs its whole main; a 120s ceiling bounds it) all join without a signature, loudly, and gate nothing. Its OUTCOME is cached across--persistre-dials, keyed on a digest of the probe recipe (binary path, mtime, size, the run's args), so an idle box re-dialing at the backoff cap pays the probe (or its timeout) once per actual change, not once per tick; a rebuild or a re-publish invalidates the key by construction. A binary that exits non-zero under the probe earns the loudest warning: rank children re-enter it with the same arguments after admission, so that failure usually takes the cohort's formation with it, so the probe surfaces it before the dial. Exit 0 with no signature has two very different causes — a binary older than the contract, or one that failed before reaching the trainer while exiting 0 anyway, which a read-only project directory produces on any binary that creates an output directory inmain— so the probe names both and quotes the binary's last lines rather than guessing. -
docs/windows-wsl.md(WSL2 setup, CUDA and the honest state of AMD under WSL) with its site page, androcm-capture.sh: a one-pass support dump of an AMD host's GPU stack (KFD topology, ROCm userspace, device permissions, both sides of the container boundary), for the box that detection cannot explain from afar. -
gpu_ram_shareis declarable in cluster config: the integrated-GPU host-RAM share (the fraction ofMemTotalan APU's aperture claims) was code-only (TrainerConfig/DdpRunConfig/DataLoaderBuilder), so on a cluster every host got whatever single value the binary hardcoded. It is host-hardware truth, so it travels thedata_pathroad, not the manifest: per-hostcluster.workers[].gpu_ram_share, a cluster-scopecluster.gpu_ram_sharedefault (the one-line answer for a farm of identical APU boxes, legal on mixed fleets because discrete-GPU hosts ignore it), andjoin.gpu_ram_share/--gpu-ram-sharefor a walk-in. Each later road knows the box better and wins. The value fills the training binary's config only when that left the knob unset; explicit code keeps the last word, like a passed--data-dir. Resolution happens at ONE point (the slim-envelope build), a walk-in's own declaration rewrites it at envelope localization besidedata_path, and ranks read it throughLocalCluster::gpu_ram_share().ClusterBuildermirrors both scopes. Values above1.0stay legal in yml exactly as in code (a platform can under-state the aperture); negatives and non-numbers are refused at config load, named. -
A standing
publish:block in fdl.yml: the controller's answers forfdl publish(source / bin / cwd / build / to / identity / args), so re-publishing a run is one bare command. Flags win field by field; a--tail replaces the block'sargs:outright, even empty ("explicitly none" must be sayable, since the args belong to the run).--no-buildhas no block field on purpose: a standing config that skips the gate would ship every future typo to the fleet. -
fdl publish --json: the report as JSON on stdout (notes stay on stderr): tree, served dir, built flag, toolchain, args, run id, and BOTH worker source-spec spellings keyed by the serving key they pair with. One report value feeds both renderers, so the machine twin cannot drift from the human text. -
fdl publish --gate <variant>: repeatable extra check-builds against named libtorch variants (e.g.--gate precompiled/rocm70on a CUDA controller), closing the "the gate proves only the controller's variant" hole: a--features rocm-only break now dies on the controller instead of on a worker. Each gate runs the same recipe under its ownCARGO_TARGET_DIR(per-variant incremental caches; success is the verdict, no artifact convention). Needs no GPU; a flodl-linking crate still needs that vendor's dev headers on the controller, and a gate without them fails loudly with the exact package line. -
fdl join-config, the once-per-farm wizard. Everything the guardrail recipe asks an operator to assemble by hand, produced in one pass on the controller: the farm overlayfdl.<label>.yml(discovery window,start: manual, admission token stamped; a farm IS an env overlay, sofdl @<label> <cmd>targets it with the machinery that already exists), an ed25519 join key born per farm in./.fdl/<label>/(which gitignores itself, since it holds keys; an identity referenced from outside the farm dir draws the cross-farm-reuse warning), the composedauthorized_keysline for the chosen door (--door brrsync source pull /asftp data mount /nologin) plus the sshdMatchhardening block, the paste-ready worker yml in that door's dialect, a publish recipe derived from the training crate's own manifest (a path dep on flodl walkssource:up to the dep root withcwd:pointing back down; a registry dep ships the crate dir alone;--features "$FDL_GPU_FEATURE"only when the manifest declares the vendor features; a workspace above the crate earns an explicitbin:caveat instead of a silent wrong guess), and a lockfile-vs-source freshness report. On an existing overlay only thetoken:line is ever touched, byte-preservingly; a user-authored overlay without one is never edited. The scaffold also wires acommands:entry, because a join window only opens for a command running in launcher mode andcluster: trueis what puts it there: without one,fdl @<label> <cmd>resolves the base command and trains locally, silently, since training on that box is a legitimate thing to do (the entry is named after the training crate when one is readable, commented when it would be a guess, and a non-cluster command run under a discovery overlay warns for the same reason). It also guides the setup rather than only producing artifacts: a preflight names what THIS box is still missing for the chosen door before anything is written (an ssh daemon, something listening on the door port,rrsyncfor doorbor an sftp server for doora, the served directory, and the two traps that produce no useful error on their own — Debian'sssh.socketowning the listener sosshd_config'sPortis ignored, and SELinux refusing a non-standard ssh port), each with the fix spelled in this platform's own commands (apt/dnf/brew,sshvssshdunits, ufw vs firewalld,semanage). It reports which environment it inspected, because that changes what a fix MEANS: inside a container the findings describe the container and a package installed into a running one is gone after the next--rm, so the durable answer is the image; on the host, a project whose commands carrydocker:builds and serves THERE, so the fix is offered in itsdocker compose exec <service>form with the services named. It writes a ready-to-installsshd_config.ddrop-in into the farm dir and renders the whole setup as numbered steps whose commands READ those files rather than repeating their contents, ending with a door self-test that is specific to the door (every door must refuse a shell and permit the forward; onlybmust also list the served tree, onlyamust open sftp). The drop-in scopes its guardrail withMatch LocalPort, notMatch User: binding it to the exposed port confines every key that arrives there, including ones added later, while leaving port 22 and the operator's own logins alone — and it emitsForceCommandonly for the tunnel-only door, since doorsaandbcarry their command in the key line and a daemon-level one would silently override it. Re-runs reuse credentials AND the farm's shape: door and controller are recovered from the farm's own worker yml rather than falling back to flag defaults, so reprinting theauthorized_keysline cannot re-render the farm around it; naming a flag is how you intend a change.--regenrotates key and token together.--jsonis the machine twin, secrets as file paths only. The token draws from OS entropy viagetrandom, flodl-cli's one non-serde dependency, added because an entropy binding is the dependency class you never hand-roll (the previous/dev/urandomread fell back to time-xor-pid SILENTLY, which on a box without the device (native Windows) meant a guessable admission token as the norm); a credential mint with no entropy now refuses loudly instead of degrading, while the publish nonce keeps its fallback (its bar is uniqueness, not secrecy). -
The wizard installs its own
authorized_keysline, consent-gated. The composed guardrail line is the artifact most likely to be mangled by hand, by exactly the audience least equipped to debug an sshd refusal. Consent is explicit only: the tty prompt or--install-key;--yesdeliberately does not count (it accepts ordinary defaults, and a security-relevant mutation is not one). Only the wizard's own line is ever touched: identity is the public key material (quote-aware againstcommand="…"options), foreign lines are preserved byte for byte, an existing install with different options is shown and replaced on confirm, permissions (~/.ssh700,authorized_keys600) are reported with the exactchmodand fixed on confirm, the rewrite is atomic, a symlinkedauthorized_keysis refused by name, and/etc/sshis never edited. After installing, the wizard checks something is listening where workers will dial and says so if not (macOS: Remote Login hint).--authorized-keys <path>names a door the default cannot reach at all: an sshd in a container reads a bind-mounted key file, writable only from the host while the in-container run sees a read-only filesystem, so neither identity can install it; a host usingAuthorizedKeysFile /etc/ssh/authorized_keys.d/%uhas the same shape. Every other rule holds,/etc/sshis still refused by name, and the named file's parent directory is left exactly as found, since the 0700 rule belongs to a~/.sshthe wizard may have to create rather than to a layout the operator already owns. -
fdl join-config --cloud-init: a user-data file embedding the worker yml, the private key and the systemd recipe (Restart=always+RestartPreventExitStatus=2+FailureAction=poweroff, the failure taxonomy end to end), so a cloud instance boots straight into a persistentfdl join. It assumes only a shell, systemd and a network:fdlis fetched on first boot before the unit it enables can start, and the declared door's own tooling comes with it (doorbbuilds a source tree on the box, so a compiler and the transports; dooramounts the data root, so sshfs, whose absence prepare classes permanent, which under this very unit means exit 2 and a halt over a missing package). Every step is guarded bycommand -v, so anything baked into the image wins. A Rust toolchain installs as the service user rather than system-wide, since cargo writes its registry cache intoCARGO_HOME, and the unit carries the matchingPATH.--cloud-init-user rootis wired for the images that log in as root (DigitalOcean and the AMD Developer Cloud on top of it), including root's real home at/root. The file states what the halt does and does not buy: where a provider bills powered-off instances,FailureAction=poweroffstops the work but not the meter, and only destroying the instance stops the bill. A SECRET artifact from the moment it exists; written 0600 into the farm dir, never printed. -
inherit-from:grew~expansion, refuses remote parents by name, and is finally user-documented.inherit-from: ~/.flodl/farms/b300.ymlnow resolves against the invoking user's home (it previously mis-resolved relative to the declaring file), which is what lets an ephemeral working directory carry a two-line farm file while the shared definition lives wherever the operator keeps it. A://-shaped parent is refused loudly as reserved grammar: a remote parent is config that could change under a standing fleet between two invocations, so it waits for pinning to be designed. The mechanism itself (chains, declaring-file-relative paths, cycle detection,fdl config showprovenance) had shipped earlier with design-doc-only coverage;docs/cli/05-manifest.mdnow documents it.
Changed
-
Formatting is rustfmt's now, enforced in CI. Adopted for a concrete reason rather than conformity:
cargo clippy --fixapplies span-based text edits and leaves layout to rustfmt by design, so on a hand-formatted tree it produces correct code at the wrong indentation, and every mechanical fix needs a manual pass afterwards. That surfaced when correcting the MSRV unmasked 171collapsible_ifsuggestions (the lint is MSRV-gated and had been silent while the declared floor forbade let-chains). One reformat, 385 files, and the 3026-test suite passes unchanged, which is what makes it a formatting change rather than a rewrite. Doc comments are untouched: rustfmt does not reflow comments by default. The commit is listed in.git-blame-ignore-revs. -
bincodereplaced bybincode-next, and supply-chain checks added. bincode 2 is unmaintained (RUSTSEC-2025-0141) and its3.0.0on crates.io is a tombstone: a README plus alib.rscontaining only a compiler error, published because crates.io cannot archive a crate.bincode-nextis the continuation, and the swap changes nothing observable: the same API, and a byte-identical wire encoding that each version decodes from the other (verified acrossu32,String,Vec<i64>andOption<f32>), soCONTROL_PROTOCOL_VERSIONis unchanged and no cohort can split on it. bincode is used for the distributed wire protocol only, never for anything on disk. A newdenyCI job runscargo-denyover advisories, licenses, sources and duplicate versions; the policy and the reasoning behind each choice live indeny.toml, including whyMPL-2.0is allowed for one dependency and why unmaintained advisories fire for direct dependencies only. -
flodl-hf's HuggingFace stack moved tohf-hub1.0,safetensors0.8 andtokenizers0.23 (from 0.4 / 0.4 / 0.20). The two visible consequences are worth reading before upgrading. Hub repo ids must be owner-qualified. The Hub answers a bare legacy name (the pre-2024 short form, no owner segment) with a 307 to its owner-qualified home on the metadata endpoint;hf-hub0.4'sureqtransport followed that redirect and 1.0 does not, reportingRepository not foundinstead.BertModel::from_pretrained("bert-base-uncased")therefore needs"google-bert/bert-base-uncased", and likewiseFacebookAI/roberta-base,FacebookAI/xlm-roberta-base,distilbert/distilbert-base-uncased,albert/albert-base-v2. Owner-qualified is also the form the Hub itself now documents, and every id in this crate, its examples, its docs and its test matrix moved to it. The MSRV consequence is not optional:hf-hub1.0 depends onhf-xetunconditionally (no feature gates it), which reachesxet-core-structuresand its 1.91 floor, so the whole workspace floor moves with it. Reverting to 0.4 was the alternative considered and declined: it is the unmaintained line, and paying the floor once is cheaper than porting away from a stale Hub client later. -
Every workspace dependency is declared once, in the root
[workspace.dependencies]. Members saydep.workspace = trueand add only what is theirs. It guards one specific drift, which had already happened: two members declaring the same dependency at different versions,flodlasking forflodl-hw = "0.7.0"whileflodl-cliasked for"=0.7.0", two contracts for one crate. It deliberately does NOT guard a direct dependency drifting from a transitive one (cargo tree -dand cargo-deny's duplicate warnings are what catch that), and the table is full rather than only the shared entries, because a partial one needs a judgement call about which deps qualify and that judgement is itself the drift risk. This changes the release procedure: the internal=X.Y.Zpins that used to be six hardcoded companions across three member manifests are now five lines in one table, anddocs/release.mdis updated accordingly. -
The CLI and DDP references are split into chaptered directories.
docs/cli.md(1568 lines) becamedocs/cli/01-install.mdthrough06-source-checkout.md, anddocs/ddp.mdbecamedocs/ddp/01-reference.mdplus a cluster guide, an internals chapter and a troubleshooting chapter;docs/README.mdis the new index anddocs/distributed/architecture.mdwas rewritten. Both single files had passed the point where a reader could find anything in them, and the cluster surface added since 0.7.0 would have doubled them again. Direct links todocs/cli.mdanddocs/ddp.mdno longer resolve; the site permalinks are unchanged, andci/release/03-lint-docs.shnow validates every link, heading anchor and/guide/URL in the tree, which is what makes a split of this size safe to do again. -
The
cuda-*fdl commands aregpu-*:fdl gpu-test,gpu-test-all,gpu-test-nccl,gpu-test-serial,gpu-build,gpu-clippy,gpu-shell. The cargo feature they pass derives from the ACTIVE libtorch variant (cu…/sm…builds--features cuda,rocm…/gfx…builds--features rocm) via$FDL_GPU_FEATURE, so the names say gpu because that is what they now mean. Anything callingfdl cuda-test(CI, scripts, muscle memory) needs the new spelling. Theddp-benchandhf-ddpvehicle manifests moved with it:docker: gpu+--features "$FDL_GPU_FEATURE"instead of a hardcodedcuda. The variable reaches every place a recipe can run: native (non-docker) commands get the libtorch env exactly as the compose passthrough provides it in docker mode, all six scaffolded compose services list it underenvironment:(compose passes only what is listed), and it expands empty on a CPU variant so one quoted--features "$FDL_GPU_FEATURE"line serves a CPU gate and both worker vendors. -
The GPU API is vendor-neutral:
cuda_*togpu_*,Cuda*toGpu*, C ABIflodl_cuda_*toflodl_gpu_*. Deprecated aliases for all 33 renamed items ship inflodl::compat, re-exported from the crate root and fromtensor/nn, so existing paths keep resolving.flodl_nccl_*(RCCL exports the nccl names verbatim),set_cudnn_benchmark(routed to MIOpen on ROCm) andcuda_compute_capability(an NVIDIA-only concept, and an error on ROCm) keep their names. -
Feature flags name the axis they select:
gpuis the vendor-neutral gate,cudaandrocmselect a backend and imply it.--features cudabehaves as before.build.rsrejects both vendors at once andgpuwithout one. -
detect_gpus()returns only devices the running build can address.detect_gpus_physical()ignores the vendor filter and answers the provisioning question (which libtorch variant covers this box). -
Integrated GPUs no longer double-count memory. An APU's aperture is carved out of host RAM, so the data plane reserves it once, subtracting unrealized headroom only.
gpu_ram_shareonDataLoaderBuilder,TrainerConfigandDdpRunConfigoverrides the fraction, and is required on a multi-socket APU, where each package carries its own aperture while/proc/meminfois system-wide and the kernel exposes no per-package equivalent to correct with. That case is refused as an ordinary error fromDataLoaderBuilder::build/ rank-worker construction, not from inside the per-epoch arithmetic: every input to the verdict (the knob, the socket count, whether the device is integrated) is static for the process, so it is decided once where aResultalready exists. The gate counts CPU packages, not NUMA nodes: NPS and Sub-NUMA Clustering split one socket's memory into several nodes without multiplying the aperture, so a single-socket machine keeps sizing itself whatever its NUMA layout. -
The system ROCm runtime precedes libtorch's bundled copy on
LD_LIBRARY_PATHin cluster pre-build, remote dispatch, the container images, and the native-build recipefdl setupprints (which honors$ROCM_PATH, defaulting to/opt/rocm). libtorch-rocm bundles the whole userspace ROCm stack, and letting it win over the host's segfaults at the first GPU op, so a recipe printed the other way round is that configuration handed to the user to paste. -
fdl diagnosereports GPUs by vendor and prints the sweep's findings. The section was headed CUDA and said "No CUDA devices available" on a box whose cards are AMD; an AMD GPU present with no ROCm runtime installed, a device the sweep deliberately does not count, now surfaces there as well as infdl probe. -
Lint gates deny instead of warn.
-W clippy::allis a no-op, sinceclippy::allis warn-by-default: every clippy step in CI, infdl clippy/fdl gpu-clippy, and in thefdl initscaffold could only fail on a compile error the build already covered. They now pass-D warnings(-D clippy::allin the scaffold, so a half-written function does not block the lint run), and clippy runs ahead of the build in all three CI jobs, which costs nothing because the two share dependency rlibs and compile the workspace twice whichever order they run in. -
The pinned consensus decode is now RAM-neutral: the reply decodes back into the snapshot staging the rank just streamed from, instead of a second locked model copy. The 0.7.0 pinned decode bought a truly-async H2D writeback at the cost of one page-locked f32 model copy per rank, measured on a RAM-tight host as a net regression (locked pages are unswappable, so the kernel evicted the per-window working set instead: +6–8% epoch wall, all of it swap thrash inside the reduce windows), which the MemAvailable gate could only answer by giving that host up entirely. The observation that removes the trade-off: the reply's frame mirrors the schema of the snapshot the rank just sent, and the snapshot staging's bytes are dead the moment the streamed encode has read them, so the client now accepts the request tensors themselves as decode destinations (
CpuReduceClient::set_decode_into_request/arm_decode_into), and the consensus lands in the same pinned storageload_averagedthen reads, under the same single-consumer fence chain the staging already carried. Zero marginal locked memory, no affordability gate: RAM-tight hosts get the async writeback back instead of the fresh-alloc fallback (rig-verified at the motivating A/B's anchor invocation: epoch wall 289.2s vs 290.2–292.8s baseline and 308.5–315.9s for the gated predecessor's ungated arm, with swap growth collapsing from +2.0GB to +0.8MB). Two hazards this surfaced are guarded: a zero-mass round's reply (meaningless zeros the caller keep-locals over) must not clobber the very tensors keep-local returns (the realized-work weight rides the wire ahead of the payloads precisely so the decode can fall back to fresh allocs before touching the staging), and a degenerate snapshot fallback shipping non-CPU or dtype-mismatched tensors is refused per payload rather than written through. Underbf16_wirethe reply lands VERBATIM in the bf16 snapshot staging (no information loss: the controller folds in f32 and the wire is bf16 either way) and the writeback stages through a flat device-side bf16 bounce, because libtorch'scopy_casts on the source device for cross-device + cross-dtype copies: a directf32-param ← bf16-pinnedcopy would materialize a pageable f32 temp on the host and silently degrade to a synchronous bounce. Net effect for bf16 cohorts: half the writeback H2D bytes on top of the zero-locked-RAM property; the weight-divergence triple keeps full f32 precision via a per-tensor path (never a model-sized upcast transient). The slot-based decode staging, itsDECODE_SLOT_*tags, and the gate's decode role are retired (pinned_decode_affordablestays, documented as historical: the reasoning applies to the snapshot staging a future RAM-tight host may need to interrogate). -
Structural zeros never cross the cluster wire anymore: a payload can declare itself all-zeros (
nbytes = 0, schema intact) and every tier folds it without materializing it. The 0.7.0 formation-broadcast fix stopped materializing the non-root zeros model (~762MB of RAM per rank at 190M params) but still shipped the zero bytes: every non-root rank pushed a full model of zeros up the wire, HMAC-SHA256'd on both ends, decoded and accumulated by the relay fold: pure overhead scaling with model size × (N−1) ranks, at formation time on every multi-host run, and worst exactly where the CPU plane aims (slow inter-node links). Three senders know their payload is structurally zeros without scanning anything, and all three now declare it instead of shipping it: the formation broadcast's non-root ranks, an idle rank's zero-mass window contribution (γ-mass 0 fused as scale-by-zero, a full model of ±0.0 per idle window before), and the controller's degenerate zero-mass scatter. Elision is a member of the fold monoid: zeros + zeros = zeros, so an all-elided round stays elided through the relay fold (seed→sums promotion is now lazy, firing only at the first non-elided deposit), throughsum_frames, and back out the scatter: it composes through fold tiers the same way the mass algebra does, hierarchical-relay-ready. The fold reads elided as "adds nothing", which is IEEE-exact, with one deliberate semantic improvement: a NaN-poisoned idle rank can no longer forward0.0 × NaN = NaNinto the consensus, because an elided contribution is true zeros. Receivers decode elided payloads to zeros of the declared shape (covering the zero-mass scatter), and the checkpoint forge skips zero-mass rounds the same way ranks do, since the consensus checkpoint is a consensus consumer too.PROTOCOL_VERSION2 → 3: a v2 reader would reject an elided frame loudly anyway (schema error at the fold), but the bump moves a mixed-version cohort's failure to the handshake (first contact, named cause) instead of the first idle window.
Fixed
-
The declared minimum Rust version was wrong, and nothing checked it.
rust-versionsaid1.85while the source has used let-chains since before 0.7.0, which stabilised in 1.88; the transitiveicu_*crates independently require 1.86. Cargo trusts that field, so a user on 1.85 got past the version check and then hiterror[E0658]: let expressions in this position are unstablein someone else's crate rather than a refusal naming the version they needed. It is now 1.91, measured rather than assumed, and three floors stack to get there: 1.88 is our own source's (let-chains),bincode-nextraises it to 1.90, andxet-core-structuresto 1.91. A newmsrvCI job compiles the workspace on exactly the toolchainCargo.tomldeclares, reading the value from the manifest so the two cannot drift, and it is what caught the third one:xet-core-structuresarrives through hf-hub 1.0's mandatoryhf-xetdependency, declares norust-versionof its own, and callsstr::floor_char_boundary, so cargo could neither warn nor resolve around it, and MSRV-aware resolution had nothing to act on. It simply failed to compile, in a transitive crate, on the declared floor. Note this floor tracks the ecosystem for the same reason: with no committed lockfile, consumers resolve fresh, so a transitive dependency raising its ownrust-versionraises this one. -
A native project built on macOS and then refused to start.
fdlcomposedLD_LIBRARY_PATHfor native commands, which macOS's loader ignores, so a scaffolded project linked and then died at launch withdyld: Library not loaded: @rpath/libtorch.dylib. ExportingDYLD_LIBRARY_PATHbeside it does not fix this: run lines execute throughsh -c,/bin/shis SIP restricted, and dyld purges everyDYLD_*from a restricted process before the shell execs anything. An rpath fromflodl-sys's build script does not reach the consuming binary either, sincecargo:rustc-link-argdoes not propagate to dependents. So macOS native commands now passRUSTFLAGS=-C link-arg=-Wl,-rpath,<variant>/lib, which SIP does not touch and which applies to the final link: the binary locates libtorch itself, and keeps working when launched directly rather than throughfdl run. Filled only when absent, so a caller's ownRUSTFLAGSstill wins, and Linux is untouched.Behind that sat a second layer, which the first fix exposed by getting far enough to reach it: upstream's
libtorch-macos-arm64archive BUNDLESlib/libomp.dyliband then haslibtorch_cpu.dylibdepend on it by absolute Homebrew path, so the load failed on any box without that formula while the library it wanted sat in the same directory as the dylib asking for it.brew install libompis the wrong answer there: it installs a second, possibly ABI-divergent copy of something already present.fdl libtorch download(andfdl setupthrough it) now repoints those dependencies at@loader_path/libomp.dylibafter extracting, and re-signs, since editing a Mach-O invalidates its signature and arm64 will not load an unsigned one.@loader_pathrather than@rpathbecause the bundled copy is a sibling of every dylib referencing it, so it resolves without depending on the referrer carrying a correctLC_RPATH. The three tools this needs are checked BEFORE anything is modified and the step is advisory, so a box without the command line tools is told to runxcode-select --installrather than left holding a patched library it cannot re-sign. -
The bootstrap's source fallback could not run where it was most needed.
./fdlverifies a downloaded release binary by running it, and replaces it from source when it cannot run — which is exactly what happens on a distribution whose glibc is older than the build base. But the fallback didcargo build -p flodl-cliin the script's own directory, so it only worked inside a checkout: in thecurl-and-run shape, in a scaffolded project, or on a cloud instance that fetched the script and nothing else, it failed withcould not find Cargo.toml. It now installs from crates.io when there is no local workspace, which needs nothing but cargo. Verified end to end on a RHEL 9 userland, where the released binary cannot run at all: bare directory, one script, workingfdlseven seconds later. -
The released
fdlbinaries were built on a floating base.release-cli.ymlusedubuntu-latestfor the Linux targets, and a dynamically linked binary cannot run where glibc is older than the one it was built against — so the base is a floor on who can use the release, and that floor moved whenever GitHub rotated the image, with no code change and no release note. The Linux runners are pinned, and a gate now reads the artifact's own maximumGLIBC_requirement and fails the build if it rises above the declared floor: raising it drops users, so it should be a decision rather than a side effect. -
Every
fdl initscaffold since crates.io tightened its API policy carried a floating git dependency. The scaffold prefers the latest published flodl version, but its crates.io query sent no identifyingUser-Agent, which crates.io's data-access policy rejects, so the "registry unavailable" fallback (flodl = { git = … }, default branch, floating) had silently become the norm on every box, andfdl add flodl-hfthen refused the very scaffoldfdl initproduced ("needs a pinnable crates.io version"). Found by finally running the stranger's path end to end: a throwaway crate on a box that never built flodl. The query now goes through the update check's probe (correctUser-Agent,--fail, 2s cap), and the offline fallback pins fdl's own version (fdl and flodl are workspace-versioned twins), so the scaffold is always a pinnable registry dep, never a git one. -
The native scaffold's printed next steps did not work.
./fdl libtorch download --cputhen./fdl buildfailed atflodl-sys's build script: fdl downloaded libtorch but never exportedLIBTORCH_PATHfor native commands (docker services get it from their compose env; native got a comment telling the user to export it by hand), and a successful build then failed at./fdl runwithlibtorch_cpu.so: cannot open shared object filefor the same reason on theLD_LIBRARY_PATHside. Native commands now fill both from the active variant (inherited values kept as the override), so download, build and run work with zero hand exports, on the registry crate, verified end to end on two boxes (one with a cold rustup). -
fdl libtorch downloadbuilt an invalid URL on Windows. Windows archives carry a-win-infix that the Linux ones do not, and nothing had ever evaluated that arm on a Windows host, so the 404 shipped unnoticed. -
fdl setupfetched an x86_64 libtorch on Apple Silicon. A Docker-mounted project needs the container's libtorch, which upstream publishes for Linux x86_64 but not for Linux aarch64. Intel Macs still force the Linux build; Apple Silicon gets the host build and a pointer to the wheel-extraction steps, instead of one that cannot load in a linux/arm64 container. -
A GPU could be reported as covered by a libtorch variant that ships no kernel for it. The compute-capability check substring-searched the raw
archs=list, so a bare major matched the minor of an unrelated entry: a Maxwellsm_50/sm_52card tested its5against cu128's7.0 7.5 …, hit the5inside7.5, andfdl diagnoseprinted OK for a build whose first kernel launch fails withno kernel image is available for execution on the device. The list is tokenized before matching now. A+PTXsuffix on a list entry is understood. -
fdl setupcould finish with nolibtorch/.active. The CPU download deliberately does not claim the pointer so a GPU variant fetched after it can, but on a host where no GPU variant follows, nothing ever wrote it, leavingfdl diagnosereporting "no active variant" immediately after a successful setup. CPU now claims the pointer if it is still unclaimed. -
A record-path collision no longer drops a rank out of the monitor tree:
build_treedescends by path segment, and a node claimed as a leaf by one producer and as an interior by another (two producers disagreeing on tiering, whichcohort_tieringexists to prevent) had its terminal entry filtered out of the descent while the interior branch was taken anyway. In debug adebug_assert!named it; in release that rank's numbers simply vanished from the tree, with no warning and with the parent's roll-up averaging over the survivors as if the missing rank had never reported: a wrong number presented with full confidence, the same failure class as the absent-is-not-zero defects above. The collision now warns once and the stray's mass folds into its parent's aggregate, so the work-weighted numbers stay whole. It deliberately gets no record of its own: a record'spathis its file path in the record log, so emitting it as a child of its own parent would append its lines into the parent's file, so the anomaly is surfaced as a warning rather than papered over with a synthetic path segment that would materialize a real file. Promoting the assert to a release panic was rejected for the reason the record log already swallows its I/O errors: a monitoring artifact must never be able to kill a training run. The sibling case that carried no assert at all is covered too: two leaves claiming one path (the "metrics feed splits in half" hazard named oncohort_tiering) now warn instead of silently first-winning. -
--record-logsays where it actually writes: a relative path resolves against the launcher's cwd rather than the directory it was typed in, which is howddp-bench/runs/xsilently becomesddp-bench/ddp-bench/runs/x; nothing echoed the destination, so the doubling was found by hunting for the files. The launcher now prints the resolved directory at startup. It joins the cwd rather than canonicalizing, because the directory is created lazily at the first record and canonicalizing a path that does not exist yet fails exactly when the message is most wanted.
- wire: full zero-elision, structural zeros ship as schema, fold-through by @fab2s in #44
- ddp: RAM-neutral consensus decode. the reply lands in the snapshot st… by @fab2s in #45
- distributed: discovery window - roster-free formation by @fab2s in #46
- distributed: staging hold + fdl start switch by @fab2s in #47
- distributed: fdl join user-facing command by @fab2s in #48
- distributed: sshd guardrail recipe + rig validation by @fab2s in #49
- doc: Correctness and refactor by @fab2s in #50
- monitor: a colliding record path folds into its parent instead of van… by @fab2s in #51
- document join yaml by @fab2s in #52
- Multi vendor support by @fab2s in #53
- ci: semver-checks + llvm-cov by @fab2s in #54
- release-prep: changelog the last PR, and fix two defects the docs rev… by @fab2s in #55
- ci: the rocky cache never once saved, and target had no business in it by @fab2s in #56
Full Changelog: 0.7.0...0.8.0