Skip to content

History / Project Tracker

Revisions

  • News + tracker + Roadmap + Testbed: M2's gate passed on the real 35B model Same digest as the single-node baseline (b0d382dbabf36df0...), 83 tensors and 40 discrete decisions checked by trace_diff on both machines. Phase table Done, new M2 milestone row. DC-045 closed; DC-109 opened for a sharded generation CLI, which the tok/s gates need.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker + Testbed: M2's gate passes on two machines (D25) Functional test on the fixture across node4 and node1, trace_diff IDENTICAL on both. Nodes are shared with other work so cluster runs stay functional rather than benchmarked; no peer carries a model install (DC-018), and the 35B two-node run is the timing phase's gate (DC-045).

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D24, M2's gate as two processes; DC-041 closed The documentation for the previous main commit, which landed without it: two datacenter-node processes over TCP, judged by trace_diff as IDENTICAL on both nodes. DC-045 keeps only the cluster.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D23, the engine runs sharded and the trace is byte-identical Two nodes over real TCP sockets, both matching the single-node trace by bit pattern and by router top-k. One branch, no second forward. Two processes and two machines remain.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D22, the transport over a real socket Bind/listen/accept/connect on loopback, a contribution exchange over TCP bit-identical to the single-node forward, and three failure modes the socket pair could not produce. DC-008 keeps only the per-hop measurement.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D21 bring-up, DC-042 closed, DC-008 carries the real socket Six fields compared before a token, named on refusal, and an install digest deliberately not among them. Twelve tests over a socket pair, both directions.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: the shard plan as data (D20); DC-040/DC-011/DC-050 closed Flat owners array indexed by expert id makes double ownership and quiet non-ownership unrepresentable. Contiguous distribution, no bits moved. A run driven by a loaded plan file is bit-identical at 2 and 3 nodes.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D19 failure semantics, DC-043 closed A clean timeout retries (safe because D17's order is canonical), a mid-frame stop does not, a missing term fails the run. The policy's timeout was decorative twice over; fixed, and the test now asserts duration and applied deadlines rather than behaviour alone.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: the wire protocol (D18) and a bit-identical two-node exchange Little-endian, framed, floats as bit patterns, length fields refused on the number. Demonstrated over a socketpair on the fixture's real weights. Timeouts, retries and failure semantics remain (DC-043); the transport choice needs measurement (DC-008).

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: the sharded forward is bit-identical on real weights The production expert path is the contract itself, demonstrated across 2 and 4 simulated nodes on the fixture's real quantized weights, with the real-model digest unchanged. serves() and isComplete() landed because the first version failed loudly. The transport is what remains.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: M2 opens with the reduction contract decided (D17) Reduce contributions, never per-node partials: fp32 addition is not associative and the measured 20000008/20000006 example is a test. Ordering by (token, expert) is stronger than a fixed ring order, so the ownership map is what must be pinned. DC-005 closed.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • M1's gate passes: re-established after D15/D16, three stale claims corrected Correctness re-verified against a freshly re-run contract (identical digests, same as the original gate run). Generation 0.108 tok/s cached / 0.0374 uncached, identical tokens. Peak memory 348.6 MB. DC-108 opened for the install-path contract gap; the checkpoint contract step's swap incident recorded.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: verification reads were uncounted and cost ~3 GB a forward digestMatches verified regardless of the verify flag and its reads were invisible to the counter, so the measured 3.06 GB was use-traffic reported as a total. D16 fixes both and shares the buffer. 19.8 s -> 15.0 s; two rounds together 39.9 -> 15.0 s (2.67x), same digest.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: profiled the forward, found a 21 s bug, 2x speedup with the same digest SHA-256 slab verification on every read was 53% of a five-token forward. D15 makes it opt-in with integrity established out of band. Verified 39.9 s -> 19.9 s, digest unchanged. DC-105 closed, DC-106 corrected and smaller, DC-107 opened for the phases that remain.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: the read path measured, two claims withdrawn, DC-105 and DC-106 opened 997 MB/s cold on a 3.10 GB sample with no cache effect, against a 1161 MB/s ceiling: the reads are not the bottleneck, so my earlier advice was wrong and is withdrawn in the record. The components bound 9-11 s of a 39.5 s wall clock; the rest needs a profile, not more arithmetic.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Correction: the SSD-bytes figure was an estimate, ~3.5x too large The install counts what it reads; the trace now reports that. The derived 177 MB/s reading is withdrawn with the estimate it came from, and the real traffic is bounded to 404-807 MB per prefill token by the per-expert geometry.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Order each phase's closed index by task id rather than by table order The indexes were emitted in the order the rows happened to sit in the old table, so P1 read DC-020, DC-027, DC-028, DC-029, DC-021... Sorted numerically now, which is how anyone looks a task up.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Restore the risks and open-questions tables my restructure silently dropped The rebuild kept a table only when it still had an open DC- row, so the two tables whose rows are not DC- rows -- Risks and traps (13) and Open questions (8) -- were deleted. Nothing failed: an emptied section is still valid Markdown and the table gate checks alignment, not presence. I found it because the script adding a new open question hit an IndexError on an empty table. Restored verbatim from the pre-restructure commit, verified section by section against it: risks 13/13, open questions 8 plus the new Q9, and the only intended zeroes are P1 (whose work is closed) and Recently closed (moved to News). Q9 records the one figure from the sweep that is unexplained rather than guessed: 2,218 expert requests, which is 221.8 fetches per token against 320 selections per token.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News, tracker and gate doc: the sweep on the real model Byte-identical traces at bank sizes 2, 8 and 16; 1,395 MB from SSD per prefill token; 0 hits at every size; 0.127 prefill tok/s. The tracker's M1 row now carries those numbers, Q9 records the one figure that is unexplained, and the gate doc has the full table.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker holds open items only; the dated record moves to News The tracker had grown to 82 task rows of which 43 were Done, plus a 200-line dated log inside a page whose stated job is what is NOT finished. It is a tracker again: 39 open rows across the phases, each phase keeping a one-line index of what closed, and the record -- 112 dated entries, newest first -- is on the new News page, linked from Home and the sidebar. Corrected in the same pass, because these were wrong rather than merely stale: the tracker's own header still claimed "no source code exists at all"; its status row still quoted the RETIRED 0.0191 tok/s and a 0.0000 cache-hit rate as if current; its counts disagreed with its own rows; P1 was listed In progress although M0 is complete; and DC-004's status paragraph still described the warn-and-skip CI that DC-103 replaced with a hard failure.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker: DC-104 -- M0 is complete, and six documents said otherwise docs/m0-gate.md records status: passing on Qwen/Qwen3.5-2B with 40,683,520 bytes of trace data identical to the contract and every discrete decision matching the reference. Every M0 task is Done, so the answer to "what is next for M0" is nothing -- and the work became correcting my own status rewrite, which called it "fixture scale" without reading the gate doc. The correction also names the 2B-vs-~1B deviation and the renegotiated oracle half.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker: DC-103 -- the toolchain is enforced, and the old check was a no-op check_toolchain.py refuses anything but Xcode 27 / Swift 6.4, the CI job has no skip branch (expected red until an image carries Xcode 27), and Home says so. The bug: the replaced step read `swift --version | tail -1` -- the target line -- so its pattern could never match and it skipped on every runner, correct toolchain included.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Wiki: the status claim corrected on Home and Roadmap, and it now names the toolchain Home said "no source code, no build system and no release" -- the extreme AGENTS.md had already recorded as wrong. Roadmap said "nothing on this page is built yet ... design phase". Both now state what is measured: Swift 6.4 / Xcode 27, clean build, 105 Swift tests, 163 stdlib Python tests in CI; M0 met at fixture scale, M1 run on the real model with its gate open; no releases, no tags. DC-102 records it, and the trap entry now requires any status claim to name the toolchain.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker: DC-101 -- the brief's own deliverable is written down (a) the invariants as the work taught them, (b) four measured disagreements plus a gap the brief leaves, (c) M0's plan and state with its honest caveat. Linked from the gate doc and recorded so it is not rediscovered.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker: DC-100 -- the M1 sweep is a script that refuses to start It checks the disk floor and swap already in use before doing anything, reports the swap numbers always, and executes nothing under --dry-run. The metrics schema is read in document order after its own test caught a LIFO reversal.

    @Pummelchen Pummelchen committed Sep 15, 2026
  • Tracker: the slot bank's contract is pinned, and the first hit-rate curve exists Bounded (peakResidentExperts <= capacity), LRU as the brief requires, and non-decreasing in capacity -- measured at 1->2/48, 2->18/48, 4->26/48, 8->32/48. My first assertion said 48/48; the real answer is 32, because eight experts times two projections is sixteen compulsory first misses. The engine was right and my arithmetic was wrong.

    @Pummelchen Pummelchen committed Sep 15, 2026
  • Tracker: DC-092 gets an instrument; DC-090 is not a blocker on M1 SHARD_EXPERT_SLOTS=<n> lets a run sweep the bank size, so D12 is decided by measurement rather than another opinion. Read once as a static let -- Swift 6 refused the static var as mutable global state, which the code's own comment had already predicted. And a correction: I had been listing DC-090 (the sign of zero) as a door to M1's gate. It is not. Metal is not on the engine's expert path -- MetalUnpack is unused -- so the divergence only affects whether that optimisation could ever be adopted.

    @Pummelchen Pummelchen committed Sep 15, 2026
  • Tracker: DC-098 closed -- and the item I called open was never open Both I6 gaps are fixed and pinned: source digests verified against an independently computed sha256 and against the committed artifacts, and the "local" placeholder made impossible by a property test, with --repo/--revision for what a build cannot know. The third item was my error. I wrote twice that the fixture builders would drop the digests on regeneration; they call quantize.build_install(FIXTURE, ...), the function that now digests, and the check that settles it is one line -- committed == what a regeneration would write, True and True. The false claim came from a regex of mine matching no source block, which is the third time an instrument of mine produced a false reading. Each time the repository was right.

    @Pummelchen Pummelchen committed Sep 15, 2026
  • Tracker: DC-099 -- I4 and L2 audited, and my first reading was wrong Every one of the 27 roles in the real install has a policy entry, the policy justifies each choice against an invariant, and the missing-role stop is tested four times. The reverse check flagged mlp.down/mlp.gate/mlp.up as unused; they are the DENSE fixture's roles, because the policy is shared across families by I4's design. The check is now a union test that fails on rot in either direction -- after I looked, rather than after I wrote "dead entries" into the tracker.

    @Pummelchen Pummelchen committed Sep 15, 2026
  • Tracker: DC-098's two I6 gaps are closed, each with a test that pins it files: {} -> real sha256 of every source weight file, verified against an independently computed hash and against the committed artifacts. revision "local" -> null with --repo and --revision inputs, plus a property test that fails if the placeholder returns. Still open: the fixture builders need the digests wired in too, which the drift test demands rather than letting slide.

    @Pummelchen Pummelchen committed Sep 15, 2026