Project Tracker

Current status

Phase	P2 — M1 (P0 in progress; P1's M0 gate passed)
M0	Gate passed — Qwen/Qwen3.5-2B revision 15852e8c…, three frozen prompts, 40,683,520 bytes of trace data identical to the contract and every discrete decision matching the reference (docs/m0-gate.md)
M1	Gate passed, re-established 2026-09-16. On the real 35 B model the engine's trace is byte-identical to a freshly re-run contract (trace_diff: 83 tensors, 0 differing elements, 40 discrete decisions; both digests b8c976c5e7ba8816…), generation is 0.108 tok/s cached and 0.0374 uncached with identical generated tokens, the hit rate is 0.0000 over 2,240 requests (structural: banks are rebuilt per forward), and peak memory is 348.6 MB. Reaching it took three measured fixes: the read path was measured and cleared (997 MB/s), then D15 and D16 removed 24 s of per-read hashing and duplicate reads (docs/m1-decisions.md). D12 was decided from a measurement after this row was written (D31): the slot bank is sized from a budget and the measured hit rate is 0 at every size. What M1 leaves open is the throughput baseline itself, which needs a quiet farm
M2	Gate passed on the real 35 B model, across two machines. A 256-expert plan over two contiguous halves, one node on node3 and one on the development host, every trace checked by trace_diff: 83 tensors, 0 differing elements, 40 discrete decisions, matching digests b0d382dbabf36df0… — the same digest as the single-node M1 baseline. It took the reduction contract (D17), the wire protocol (D18), the failure semantics (D19), the shard plan as data (D20), bring-up (D21), a socket that binds (D22) and a forward that shards (D23–D26). and a sharded generation CLI (D28) now produces the same tokens and digest across two machines
Open tasks	35 across the phases below; closed work and the dated record are on News
Language standard	Swift 6.4 on Xcode 27 with swift-tools-version:6.4 — the only supported toolchain, enforced by tools/check_toolchain.py; Python 3.14
Tests	186 Swift, 276 Python, both run on every push — and nothing is skipped: the Metal kernel tests run on the node's GPU since D34
Farm	Four identical Mac mini M2, 8 GB each, macOS 27.0, Xcode 27, Python 3.14.7
Development link	1 Gbit Ethernet: 0.49–0.64 ms round trip measured; the mesh VPN path measures 1.4–1.8 ms

Phases at a glance

Phase	Milestone	Objective	Gate	Status
P0	Foundation	Decisions, harnesses and conventions	ADRs, harness spec and conventions written	In progress
P1	M0	Staged: harness (M0a), Qwen3.5-2B bf16 (M0b), 4-bit (M0c)	Bit-matches reference golden traces at each stage	Done
P2	M1	Qwen3.6-35B-A3B, single node, 4-bit, SSD-streamed	Correct output + recorded tok/s baseline	Done
P3	M2	2 nodes, expert-parallel	Bit-identical to M1	Done
P4	M3	4 nodes	≥3x the M1 tok/s	In progress
P5	M4	DeepSeek-V4.1-Flash	Matches reference at 128K context	Planned
P6	M5	Qwen3.8-Flash-Next	Same protocol as M4	Planned

How to read this page

• Task IDs (DC-nnn) are stable and are never reused or renumbered. A cancelled task keeps its ID and is marked Dropped with the reason.
• Status is one of Planned · Next · In progress · Blocked · Done · Dropped. Next means it is the immediate queue, not that it has started.
• "Done when" is the evidence, not the activity. A task is done when the named artifact or measurement exists, and the row says where it is.
• A gate is closed only by measurement. When a status here disagrees with a gate on the Roadmap, the measurement wins and this page is corrected.
• Documentation sync rule (DC-082). A commit that changes behaviour, the plan, or a decision updates the README, the affected wiki page and this tracker in the same push. A tracker row that has drifted from the repository is a defect.

P0 — Foundation

Decisions that are expensive to reverse. Each ADR records the alternatives considered and why the others were rejected.

ID	Task	Depends on	Done when	Status
DC-008	ADR — transport: Thunderbolt bridge vs LAN vs SFP/QSFP, measured latency and bandwidth per hop. The implementation half is done (D22): a listener that binds, listens and accepts, a connector with an explicit deadline, and a contribution exchange over TCP that is bit-identical to the single-node forward. What remains is the half the ADR is named for — the per-hop measurement — which needs the testbed, plus IPv6 if the farm needs it	DC-009	The tabled latency and bandwidth come from measurements on the real links, not from the paper	In progress

| DC-009 | ADR — all-reduce wire protocol: framing, dtype, timeouts, retries, node-failure semantics. Decided and implemented for the part that carries the numbers (D18): little-endian, length-framed, floats as IEEE-754 bit patterns, no checksum by design, and every length or dimension refused on the number before anything is allocated. Demonstrated by a two-node socket-pair exchange that is bit-identical to the single-node forward on both nodes. Open: timeouts, retries and the node-failure rule (a missing term fails the run) — DC-043 — and the transport choice, which needs measurement (DC-008) | DC-005 | A node that stops answering fails the run by rule, and a retry cannot change the bits | In progress |

| DC-036 | A Swift gate that actually runs on a clean machine: GitHub's hosted macOS images carry Xcode 26.x (newest Swift 6.3.3) while the project standard is 6.4 on Xcode 27, so today the gate runs locally on the farm and CI reports the gap | DC-004 | swift build && swift test passes on a runner the project does not own | Planned |

DC-004 status: CI carries the repository's own gates on every push — the Markdown link gate (tools/check_markdown_links.py: local links and #anchors, offline), the table gate (tools/check_markdown_tables.py, over the repository and this wiki) and the whole tools suite. The Swift workflow (.github/workflows/swift.yml) fails on a runner that cannot meet the project toolchain instead of warning and skipping (DC-103): GitHub's macos-26 image carries Xcode 26.0–26.5, so that job is red by design until an image ships Xcode 27, and the Swift gate runs locally until then. The tools runner has no torch, so the capture tests skip there.

Closed in this phase: DC-001 Bootstrap the wiki: Home, Roadmap, Project Tracker, · DC-002 README correction pass: the Shard codename in the · DC-003 Repository scaffolding: .gitignore, AGENTS.md, · DC-006 ADR · DC-007 Golden-trace harness specification: trace schema, · DC-010 ADR · DC-012 Repository layout and SwiftPM target plan, consistent · DC-014 Language standard recorded and applied: Swift 6.4, · DC-015 The Swift language-feature register: which upcoming · DC-016 Core ML tooling installed and verified: coremltools · DC-017 Farm inventoried from this checkout: every node's · DC-019 Interconnect, storage and memory baselines measured, · DC-037 M1's numeric contract: the mixture of experts · DC-038 Expand the key and query heads up to the value head. Evidence on News.

P1 — M0: staged, dense Qwen3.5-2B and its harness

The harness is proven before the model, and the model is proven in the order that keeps one variable at a time (D1):

Closed in this phase: DC-020 Choose and pin the M0 model: Qwen3.5-2B (Apache-2.0) · DC-021 Reference runner that emits golden traces for the · DC-022 IR importer for the M0 models (name-to-role mapping · DC-023 Minimal single-node forward pass, driven by the IR, · DC-024 Trace-capture harness (per-layer activations, logits, · DC-025 Diff harness with bit-exact comparison and · DC-026 Gate M0: on the pinned model, zero differing bytes · DC-027 Reference contracts for both M0 models: the official · DC-028 M0a: capture and diff harness, proven able to · DC-029 M0b: Qwen3.5-2B bf16 bit-exact, layer by layer,. Evidence on News.

P2 — M1: Qwen3.6-35B-A3B, single node, 4-bit, SSD-streamed

ID	Task	Depends on	Done when	Status
DC-033	Metal kernels for attention, router, MoE and head	DC-030	The mixture kernels and the family assembly are done: the engine runs the tiny qwen3_5_moe checkpoint bit-for-bit against the contract, router decisions included. Metal remains Landed since: the int4 row read decodes one expert instead of the whole stack — one expert of eight reads 1184 of 9472 bytes, measured	In progress

Closed in this phase: DC-030 Importer for the Qwen3.6-35B-A3B · DC-031 Quantization/transcode · DC-032 SSD expert streaming with a bounded per-node expert. Evidence on News.

Sweep result (2026-09-16). capital's five tokens at bank sizes 2, 8 and 16 produced three byte-identical traces — trace_diff reports 83 tensors, 0 differing elements and 40 discrete decisions — with identical expert counters at every size: 6,977,224,704 bytes from SSD (1,395 MB per token), 2,218 requests, 0 hits, 13.95 GB decoded in memory, and 39.3–39.9 s of prefill (7.90 s per token, 0.127 prefill tok/s — a prefill rate, not the gate's generation baseline). The digest moved from the pre-D11 b8c976c5… to b0d382db… exactly as D11 predicted. Full numbers and the one unexplained figure (2,218 requests against 320 selections per token) are in docs/m1-gate.md.

| DC-107 | The next performance targets: attn.core 3.86 s, the expert unpack 3.45 s, the reads 2.99 s. After D15 and D16 removed 24 s of hashing and duplicate reads, these are what a 14.97 s five-token forward is made of — none of them I/O, and the reads and unpack are at their measured limits (~1 GB/s and ~1,012 M values/s). The attention core is the largest single compute phase and the first target | DC-105 | Each is either faster with the same digest, or documented as at its limit with the measurement that says so | Planned |

P3 — M2: 2 nodes, expert-parallel

ID	Task	Depends on	Done when	Status

P4 — M3: 4 nodes

ID	Task	Depends on	Done when	Status

| DC-051 | Per-token synchronisation budget: all-reduce latency per MoE layer against expert-read time | DC-045 | The measured budget explains the achieved scaling | Planned | | DC-052 | Per-node cache and prefetch tuning at four nodes | DC-050 | Cache budget and prefetch depth are recorded per node | Planned | | DC-053 | Gate M3: ≥3x the frozen M1 tok/s on the same prompt set. The functional half is demonstrated (D27: four machines in a full mesh, one bit-identical trace) and the instrument now exists (D28: datacenter-generate shards, and a two-machine cached generation produced the reference's tokens and digest). What the gate needs is the measurement, on a quiet farm, since the nodes are shared with other work | DC-045 | On a quiet farm, the same prompt set runs at ≥3x the M1 rate with bit-identity intact | In progress |

P5 — M4: DeepSeek-V4.1-Flash

ID	Task	Depends on	Done when	Status
DC-060	DeepSeek importer with transcoding of native FP4 experts and FP8 dense weights	DC-045	No requantization step exists in the path	Planned
DC-061	Compressed / sparse hybrid attention and constant-size recurrent state	DC-060	The model runs at the reference context length	Planned
DC-062	MTP speculative decoding parity	DC-060	Draft and verify match the reference protocol	Planned
DC-063	Sparse-attention block-selection exactness check	DC-061	Selections match the reference exactly, not approximately	Planned
DC-064	Gate M4: matches the reference at 128K context	DC-063	The exactness protocol passes at 128K	Planned

P6 — M5: Qwen3.8-Flash-Next

ID	Task	Depends on	Done when	Status
DC-070	Importer and reference match for Qwen3.8-Flash-Next	DC-064	Same exactness protocol as M4	Planned
DC-071	Sparse n-gram lookup table handling and per-node sharing	DC-070	Table lookups are correct and bounded in memory per node	Planned
DC-072	Expert-parallel scaling on the 125B-A6B shape	DC-070	Gate M3's ratio is reproduced or explained on this shape	Planned
DC-073	Gate M5: matches the reference, same protocol as M4	DC-072	Recorded, with the measurement	Planned

Cross-cutting

ID	Task	Depends on	Done when	Status
DC-080	Shared quantization and shard policy data files, common across model families	DC-011	One policy file drives more than one family	Planned
DC-085	Audited 2026-09-16: the sister project's int4 is unsigned-with-bias and its container family is GTurbo*V1, so there is no defect to report and nothing of ours was copied (D39). Synchronisation with the sister project: track its releases, report cluster-relevant defects upstream	—	A defect found here has a linked upstream issue where it belongs there	Planned

Closed in this phase: DC-082 Documentation sync rule: README, wiki and this · DC-086 Uncached reads for model payloads: open the payload · DC-088 Per-slab digests, so a reader can verify the expert · DC-089 D11 option 1: flush denormal scales to zero in both · DC-093 WITHDRAWN · DC-094 The spec/config correspondence is a tool with tests · DC-095 The wiki's tables have never been gated in CI, and · DC-096 Audited the two largest unprobed engine surfaces and · DC-097 The brief's 16 KB alignment is unimplemented, and · DC-098 I6's two gaps are closed, all three items verified. · DC-099 Audited I4 and L2 and found both · DC-100 The M1 sweep is a script that refuses to start, not a · DC-101 The brief's own · DC-102 The public status claim is corrected in all five · DC-103 The toolchain is enforced, not documented: Xcode 27 · DC-104 Corrected the M0 status claim in six documents: M0 is. Evidence on News.

Risks and traps

ID	Risk	Handling
R1	Bit-identity may not be attainable on Metal. Nondeterministic reduction order or driver scheduling can make two runs differ.	The deterministic reduction contract (DC-005) is written before M0, and M0's gate is where the claim is first tested. If it cannot be met, the gate is renegotiated in this tracker, with the measurement that forced it — never quietly.
R2	The all-reduce may cost more than the reads it hides. One reduction per MoE layer against tens of expert reads: if synchronisation is not small relative to the read time, scaling flattens and the 4–10x thesis is wrong.	DC-051 measures the budget at four nodes before any scaling claim is made.
R3	8 GB per node bounds what is resident. Dense backbone, shared expert, KV state and expert cache all have to fit beside macOS.	The shard policy and the model choice are constrained by measured resident cost, not by intent (DC-011, DC-032).
R4	Transport reality versus documentation. LAN, SFP/QSFP and a Thunderbolt bridge differ in latency and in what macOS permits; a bridge may need elevated entitlements. Measured 2026-09-15: the farm has two paths — a 1 Gbit Ethernet segment at 0.49–0.64 ms round trip and a mesh VPN at 1.4–1.8 ms — and the node names resolve over the VPN, so a run that binds what a host name resolves to silently takes the slow path. No Thunderbolt bridge is configured on any node, despite two ports each.	DC-008 records measurements and the chosen transport; the README's topology claims are corrected if they do not hold. The all-reduce payload is ~4 KB per MoE layer, so this is a latency question before it is a bandwidth one — which makes the 2.5x between the two paths a first-order number, not a footnote.
R5	Licence mismatch. The sister project is Apache-2.0; this repository is MIT. Reusing kernels, the repacker or format code carries Apache-2.0 obligations and a NOTICE.	DC-013 decides what is reused and records the obligation before any reuse.
R6	Top-k routing versus 1/N slicing. With k experts selected per token and N nodes, an unlucky slice or routing imbalance leaves nodes idle — and with N greater than experts/k the sharding stops being meaningful.	The shard policy is validated on real routing statistics (DC-011, DC-040), and M3's ratio is the honest test.
R7	No reference to be exact against. Bit-matching has no target until a reference implementation is chosen for the M0 model.	DC-020/DC-021 pin the model and the reference before the gate is attempted.
R8	Scattered reads on one SSD per node. Sustained throughput on four Mac minis reading expert shards concurrently is not the burst number on one machine.	The M1 baseline is recorded on the same hardware class as the cluster (DC-034).
R9	Public-repository hygiene. This wiki and the repository are public: credentials, addresses, access paths and model-access keys must never be written down.	Node names are used as labels (node1…node4) and nothing else identifies a machine; the Testbed page records hardware class, measurements and roles only (DC-083 keeps it that way).
R10	Bit-identity across shard counts does not follow from "accumulate in fp32". If each node pre-sums the experts it owns and the partials are then combined, the association order differs from a single-node pass. Measured: 11185 of 20000 random top-8 draws (56%) sum differently — by exactly 1 ULP — under the two orders. The M2 gate would fail on arithmetic alone, with every per-tensor check still green.	The reduction must be canonical by construction: nodes exchange their per-expert contributions and every node accumulates in fixed expert-id order, so a single-node pass and an N-node pass perform the same additions in the same sequence. The payload grows to top-k × 4 KB (≤ 40 KB) — affordable, since the transport is latency-bound (Testbed). This is a P0/P3 decision, not an implementation detail.
R11	bf16 router logits flip the top-k set. I3 says discrete decisions must match exactly and keeps routers at "bf16 or higher". Measured over 20,000 random 256-way routers with round-to-nearest-even bf16: 922 (4.61%) changed the top-8 index set. A concrete pair: fp32 logits 5.363673687 and 5.385571480 both round to bf16 5.375, so the ordering that separates them is gone. One flipped index diverges the output completely while per-tensor MSE still looks healthy. (Correction, 2026-09-15: an earlier version of this row quoted a pair at 5.424064/5.413122 — those came from mantissa truncation, which is not what a bf16 cast does. The rate is unchanged in substance; the example was wrong and is replaced.)	Resolved as D5: router logits, normalization, comparison and top-k run in fp32, with an explicit ascending-expert-id tie-break applied identically on both sides; bf16 or quantized router weights are fine because the accumulation is fp32. The I3 assertion is a separate test from any numeric tolerance and needs a fixture whose logits sit within 1 ULP of each other, or it cannot fail — test_bf16_rounding_reproduces_the_router_hazard is that fixture.
R12	M0's model is not the conventional one the milestone was designed around. D1 chose Qwen3.5-2B, whose first layer is a chunked Gated DeltaNet: the bit-exactness harness would be first exercised against a chunked delta rule with cumulative decay, a depthwise causal conv and a gated norm — the kernel family the brief assigns to M4/M5 — and, if 4-bit lands in the same step, against a new quantization path at the same time. Two independent error sources in the first proof of the harness is how a harness gets quietly distrusted.	The milestone is staged (M0a harness + conventional model, M0b bf16 on the real architecture, M0c 4-bit) so exactly one variable moves at a time; the work is not wasted, because M1's Qwen3.6-35B-A3B is 30-of-40 layers of the same Gated DeltaNet family, and the sister project already runs that family single-node as a comparison point.
R13	PyTorch's matmul order is not reproducible by any other implementation. Measured on a real layer's shapes (8×1024 by 512×1024, fp32): an explicitly ordered accumulation differs from torch's in 3544 of 4096 outputs (mean 17 ULP, max 22587 ULP), while both sit exactly 3.148e-07 from an fp64 computation of the same product. The difference is summation order, not accuracy, and the order belongs to torch's BLAS kernels rather than to the model. A gate defined as "bit-match torch" would fail forever, for a reason that has nothing to do with the engine.	Resolved in D3 by splitting the references: tools/ordered_reference.py states the order of every sum and is the bit-exactness target, while the torch trace is the semantic oracle — exact discrete decisions (I3) and per-tensor closeness. The engine's kernels must accumulate in the contract's order (no split-K, no FMA, no reassociation), and I2 compares our implementation to itself. Confirmed on the real model: the ordered forward's logits argmax agrees with torch at every position while the last bits differ from the first matmul onward.

Open questions

Decisions that are not yet made. Each one is answered by an ADR or a task above, and the answer is recorded here even when it is "not yet decided".

ID	Question	Answered by
Q1	Which dense model is M0's target, and which implementation produces its golden traces?	DC-020, DC-021
Q2	Is the IR a shared component with the sister runtime, or a new one built beside it?	DC-010
Q3	How does the 1-node M1 baseline run the sharded model for a bit-identical comparison — one shard holding every expert, or a separate unsharded path?	DC-006, DC-035
Q4	What exactly stays resident on each 8 GB node, and what is streamed?	DC-011, DC-032
Q5	Which transport is mandatory for M2/M3, and does the Thunderbolt bridge need elevated privileges? (Measured: 1 Gbit Ethernet 0.49–0.64 ms vs mesh VPN 1.4–1.8 ms; no bridge is configured — see R4 and DC-017.)	DC-008
Q6	Does "unlimited nodes" mean N is bounded by experts-per-layer divided by k, or is expert duplication allowed?	DC-011, DC-050
Q7	Where do golden traces live — in the repository, or in external storage? Trace size decides it.	DC-007
Q8	Is Core ML / Neural Engine prefill part of a node's engine, and if so where is the sidecar produced and verified? (The conversion toolchain is installed — DC-016 — but installing it decides nothing.)	DC-033, DC-061
Q9	Why 2,218 expert requests for a five-token prefill? That is 1,109 fetches, or 221.8 per token, against 8 experts × 40 layers = 320 selections per token. De-duplicating selections within a layer would land below 320, but that is an inference: settle it by counting distinct experts per layer in the mixture and comparing	DC-034

Later / parked

Ideas that are not scheduled and not promised. They are written down so they are not lost, and they are not tasks until they get a DC-nnn id.

• A docs/ tree in the repository for runbooks and audit registers, mirroring the sister project's split between wiki (user/engineering documentation) and docs/ (runbooks).
• Package the cluster launcher as a single script that reports per-node readiness before a run, in the spirit of the sister project's launcher.
• A public benchmark page once there is a number worth publishing — with the prompt set and the hardware named, or not published at all.
• Report the sister project's SECURITY.md defect upstream: its private vulnerability reporting link points at github.com/drumih/turbo-fieldfare, not at TinyTitan. Found on 2026-09-15 while writing this repository's own policy; it belongs in the sibling tracker or an upstream issue (DC-085).