History / Testbed

Wiki review: remove the dependency framing; flag the plan pages as stale Removed every non-historical 'sister project' reference (Home, Glossary, two Architecture rows, the Roadmap objective, Testbed's dependency section) and the two tracker rows that presupposed adopting another project's code - DC-085 and DC-123. ttd is built from scratch and is MIT, so there is no upstream to track and no code to adopt. Flagged rather than rewritten: Roadmap.md, the milestone pages and most tracker rows still describe the retired DatacenterEngine plan, and Architecture.md's sharding section says expert parallelism while what shipped divides layers. That is a planning decision, not a cleanup.

Pummelchen committed Sep 19, 2026

Home: drop the sister-project row, so this wiki no longer points at the runtime repository

Pummelchen committed Sep 19, 2026

Testbed: cluster runs use the internal LAN addresses, not the node names The names resolve over the mesh VPN at ~2.5x the round trip. The addresses are recorded with the operator's permission so the next run does not rediscover them.

Pummelchen committed Sep 16, 2026

News + tracker + Roadmap + Testbed: M2's gate passed on the real 35B model Same digest as the single-node baseline (b0d382dbabf36df0...), 83 tensors and 40 discrete decisions checked by trace_diff on both machines. Phase table Done, new M2 milestone row. DC-045 closed; DC-109 opened for a sharded generation CLI, which the tok/s gates need.

Pummelchen committed Sep 16, 2026

News + tracker + Testbed: M2's gate passes on two machines (D25) Functional test on the fixture across node4 and node1, trace_diff IDENTICAL on both. Nodes are shared with other work so cluster runs stay functional rather than benchmarked; no peer carries a model install (DC-018), and the 35B two-node run is the timing phase's gate (DC-045).

Pummelchen committed Sep 16, 2026

Baselines measured, target models verified, and two arithmetic hazards DC-019: the numbers the plan depends on, measured rather than assumed, plus the three target models read out of their own checkpoints. Measured (Testbed, "Measured baselines"): - 4 KB round trip on the 1 Gbit segment: 645 us over TCP, 687 us over UDP, against a 515 us empty ICMP round trip. The transport is base-RTT-bound, not bandwidth-bound, so the 4 KB payload is nearly free and UDP buys nothing. - Internal SSD: 1161 MB/s sequential read, but 108 MB/s for random 16 KB reads at queue depth 1 — 9-11x slower. That gap is the entire case for a deep read queue and expert caching, and it is the number the bytes-per-token maths has to use. - ~2.5 GB reclaimable memory on an idle 8 GB node; no external NVMe attached on any node, so the assumed 3 GB/s has no hardware behind it yet. Two probes were thrown away before this: the first reported 13 us round trips (a dead echo server returning EOF instantly) and the second reported 13 GB/s reads (a 2 GB file that fit in the 8 GB machine's cache). Both are recorded as discarded, not published — a plausible wrong number is worse than none. Target models (new page): Qwen3.6-35B-A3B, DeepSeek-V4-Flash and Qwen3.8-Flash-Next read from their own config.json at a named revision, with what the brief got right and the three things it does not say — Qwen3.6 is 30 linear + 10 full attention layers, not conventional; DeepSeek has 3 hash-routed layers, a 64-head indexer and 128x128 FP8 blocks; the Qwen n-gram table is a tri-gram table of 20M entries. Two hazards, both demonstrated with numbers rather than argued: - R10: bit-identity across shard counts does not follow from "accumulate in fp32". Pre-summing each node's own experts and combining the partials is a different association order: 11185 of 20000 random top-8 draws (56%) sum differently, by exactly 1 ULP. The M2 gate would fail on arithmetic alone. Nodes must exchange per-expert contributions and accumulate in fixed expert-id order — determinism needs a fixed order, not a ring. - R11: bf16 router logits flip the top-k set. I3 keeps routers at "bf16 or higher"; over random 256-way routers the top-8 index set changed within 22 draws (expert 102 at 5.424064 rounded to expert 85's 5.413122). Routers go through fp32 logits and fp32 top-k, with a documented tie-break. The sync-cost table on the Target models page turns R4's latency into a per token budget: on the measured link a ring is 77 ms/token at four nodes over 40 MoE layers, against 26 ms for recursive doubling.

Pummelchen committed Sep 15, 2026

Testbed: the farm, measured — and Python moves to 3.14 (DC-017) All four nodes answered over SSH and are identical: Mac mini M2, 8 GB, macOS 27.0 (26A428), Xcode 27, Swift 6.4, Python 3.14.7, Docker 29.8.0 running, QEMU present, 81-107 GB free SSD, two Thunderbolt ports, Homebrew and uv. Node 4 is this checkout's machine. None of them carries coremltools — that lives only in this repository's venv, because whether a node's engine needs Core ML at all is still Q8. Two findings that change how M2 and M3 have to be run: - The nodes have two network paths and the node *names* take the slow one. Measured round trip: 0.49-0.64 ms over the switched 1 Gbit segment versus 1.4-1.8 ms over the mesh VPN. The all-reduce spends exactly that number once per MoE layer, so a run that binds whatever a host name resolves to is 2.5x worse for nothing. The readiness checklist now pings both paths. - No Thunderbolt bridge is configured on any node (bridge0 down) despite two ports each. Thunderbolt in the README is the design target, not the link in use. R4 and Q5 carry the numbers, and R9 is refined to say what it always meant: node names are labels, while addresses, credentials and access paths are never written down. DC-016 is corrected to its final state — coremltools 9.1.dev1 on CPython 3.14.7, an exact pre-release because 3.14 has no stable wheel — with the 3.13-first history kept rather than rewritten. DC-018 is added for provisioning the nodes with a matching build and install before a cluster run.

Pummelchen committed Sep 15, 2026

Tracker: Core ML toolchain installed and verified (DC-016), and Q8 opened DC-016 closes with the evidence rather than the intention: coremltools 9.0 on CPython 3.13.15 in the project venv, pinned in tools/requirements-coreml.txt, verified by a full round trip — MIL program, 95 conversion passes, .mlpackage, load, predict [3.0, 4.0, 5.0, 6.0] — on CPU_ONLY and on CPU_AND_NE. CPython 3.14 was rejected on purpose: its only coremltools wheel is the 9.1.dev1 pre-release, and a conversion toolchain that changes under the project would make a converted model unattributable to the version that produced it. Testbed records the toolchain and that reasoning. Q8 asks what installing a converter does not answer: is Core ML / Neural Engine prefill part of a node's engine, and if so where is the sidecar produced and verified? The toolchain being present is not a decision. Also recorded in the closed row: the venv exposed a defect in the link gate, which had been reading coremltools' own README files inside site-packages.

Pummelchen committed Sep 15, 2026

Tracker: Swift 6.4 on Xcode 27 is the language standard (DC-014) DC-014 closes: the standard is recorded in AGENTS.md, CONTRIBUTING.md, the Testbed page and the Architecture stack row, and it supersedes the earlier 6.3.3 target — the superseded line is written down rather than erased, so a later reader can tell a decision change from an inconsistency. DC-015 is added for the language-feature register. It has to be measured from the real manifest, not copied from a release note, which is the same rule this project applies to every other number. Testbed states the toolchain once, so a node check has something to compare against: Apple Swift 6.4 (swiftlang-6.4.0.34.1), Xcode 27.0 with the Metal toolchain, and swift-tools-version:6.4 in the manifest once it exists.

Pummelchen committed Sep 15, 2026

Tracker: the scaffolding landed, and the development link is 1 Gbit Ethernet DC-003 closes — .gitignore, AGENTS.md, CONTRIBUTING.md, SECURITY.md, the issue and pull-request templates, twelve repository topics and the seven phase milestones on GitHub. DC-004 is in progress: the Markdown link gate is live and tested, the Swift gates land with the code. DC-002 is in progress with its remainder named rather than dropped — the `Shard` codename, `Macs Minis/Studio's`, and the Qwen target-version mismatch. Recorded because it changes how M2 and M3 numbers must be read: the farm's development link is 1 Gbit Ethernet for now. That is a developer limitation, not the design target, and it appears on the Testbed page, in the Architecture transport section and in R4/Q5 instead of the README, whose transports remain what the design assumes. The all-reduce payload is ~4 KB per MoE layer, so the link shows up as latency before bandwidth. Also parked: the sister project's SECURITY.md points private vulnerability reporting at github.com/drumih/turbo-fieldfare rather than at TinyTitan — found while writing this repository's policy, belongs upstream (DC-085). The closed record gains DC-003 and the three README fixes, with the remainder of DC-002 still open above.

Pummelchen committed Sep 15, 2026

Bootstrap the wiki: project pages and the task tracker The wiki held a one-line Home stub. It now carries the project's plan, its status and its design, so that no later commit has to invent them. Pages: - Home — what the project is, the expert-parallel thesis, the milestones at a glance, non-goals, the sister project and the licence. - Roadmap — P0..P6 (M0..M5) with the objective, the deliverables and the gate of each phase; the bit-identity gate at M2 is named as the real one, and the rule that a gate is renegotiated openly, never quietly. - Project Tracker — the tracker the repository did not have: 51 stable DC-nnn tasks across seven phases, a phase/status board, nine risks and traps, seven open questions, the closed record and a parked list. DC-001 (this bootstrap) and DC-082 (the README/wiki/tracker sync rule) are the only two closed items; no source code exists yet. - Architecture — the design as written: expert versus pipeline parallelism, the four invariants (bit-reproducibility, discrete decisions, faithful ports, transcode-don't-requantize), the role-based IR and thin importers, the shard policy, transport, and the stack. - Testbed — the target configurations, what a node must provide, a readiness checklist and the sister project as a dependency. - Glossary — model, distributed, runtime and project terms. Also the sidebar and footer navigation, in the style of the sister project's wiki. Deliberately absent: host names, addresses, credentials and any measured number. Nothing in the repository runs yet, so every page states that the claim is design intent, and the risks say which ones are unproven.

Pummelchen committed Sep 15, 2026