Skip to content

History / Project Tracker

Revisions

  • News + tracker: M1's divergence is in layer 0's attention half, via an opt-in internals capture Both sides now record attn_out and ff_out per layer behind a flag, off by default so no digest moves. One comparison eliminated the entire mixture path: hidden_in identical, attn_out differs in all 10240 values.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: M1's divergence is systematic in layer 0; the convolution is eliminated All 10240 values of layer.00.hidden_out differ, not one element -- the differ's '1 float' is a tensor. A probe that disagreed with the reference's own record was a broken instrument. Candidates now: the delta rule's first step, the gates' exponentiation, the gated RMSNorm's ordering.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: M1 fails on today's artifacts, localised to layer 0 token 0 The reference now streams experts by index (identical to the stacked path on the real model), which makes the contract run cost two minutes here. The fresh contract diverges from today's engine by 40 discrete decisions and 1 float; layer.00.hidden_in matches and layer.00.hidden_out element 0 differs by 0.0014 absolute.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: an uncached, byte-identical checkpoint reader; the M1 blocker measured as memory safe_open mmaps a 67 GB shard; tools/uncached_safetensors.py preads instead and is byte-identical on a real 4 GB shard. The attempt then showed swap 2048->5120 MB and disk 11.8->8.0 GB in 60 s: the binding constraint is the reference's own 2 GB-per-expert-stack working set, not the page cache.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: milestones re-check themselves (D46); the M1 document's own inconsistency found tools/check_milestones.py separates 'the engine is stable' from 'the engine matches the reference', since conflating them is how the M1 confusion hid. Declared divergences stay visible without failing; undeclared ones fail.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: the M1 gate is not current; DC-111 records the re-establishment and its blocker A fresh engine trace differs from the stored contract by 40 discrete decisions and 1 float, where the two stored traces were byte-identical. The reference is unchanged and reads the checkpoint; the engine reads an install rebuilt after the gate ran. Settling it needs a GB-scale read of the 67 GB checkpoint, so the discriminating step is written down instead of risking the node.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: memory floor and heavy-job lock enforced (D44); DC-110 closed on ten clean runs tools/heavy_job.py refuses what cannot fit, warns on pressure without refusing, and holds one heavy-job slot with stale detection. The first wiring put the claim in a library function, so tests took the production lock and refused each other -- fixed by moving it to the entry point.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: a real killed-node failure test; DC-110 for the one unexplained transient failure The survivor failed fast and by name. Found and fixed: RST reported as a generic socket error, an empty failure message, and a crash on using a closed transport.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-009 and DC-080 closed, wire protocol written as a contract DC-009's acceptance is met by D17/D19 and the tests behind every failure rule, now readable in docs/wire-protocol.md. DC-080 was already verified by the policy test across two fixture families. 16 open tasks.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D19's mid-frame rule now actually holds on real sockets (D42) readDataOfLength: blocked until it had every byte, so a peer that stopped mid-frame hung the run instead of failing it. Fixed with read(2); re-verified with 189 Swift tests and bit-identity on four real machines.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-036 met (Swift gate on a clean machine; one command for the gates), 18 open tasks tools/run_all_gates.py runs every gate and reads the test counts from the suites' own output, so the stale-count mistake is structurally impossible -- and it caught one immediately (288 vs 276). Run on a farm node that has never built this checkout: 186 Swift tests, 0 skipped, 0 failures, all gates OK.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-084 closed (baselines as data), 19 open tasks Counts asserted exactly and re-checkable on a busy farm; observations reported, not asserted. The demonstration matched every recorded count to the byte and re-produced the M2 digest unasked, and found two bugs in my own tool: applicability written as prose (a false failure), and a derivation that crashed on the real metrics shape my tests had invented.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: the invariants audited; I6's artifact gap named (D39 round) I1-I4 verified with current evidence, I5 not applicable yet (bf16 checkpoint), I6 partly -- the code is fixed but the install predates it, so verify_install.py now reports the placeholder revision, the empty file digests and the swapped repo field. check_status_claims.py keeps the audit from decaying.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-018 closed (farm provisioned and verified on the nodes), DC-085 audited (D39) Every node holds the same verified install (693 tensors tiling 21,700,655,616 bytes; 5 digests each). The audit found a portability defect in our own verifier and settled that the sister project's int4 is unsigned-with-bias in GTurbo*V1 containers, so no upstream defect exists.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker + roadmap: M3's gate built, refusing on a busy farm (D38) It asks every node for its load average first and stops if any is above the threshold, naming the numbers. Built is not passed, and the record says so: --allow-busy-farm records an observation that cannot pass. Staging the install on node1 and node2 in the background.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker: the claims gate now checks command examples too (47 claims)

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-081 closed -- per-node cluster accounting (D37) ExchangeLedger counts every all-reduce attempt; each node writes metrics.json and the harness prints a self-checking table (7/5 and 5/7 terms from the two nodes). Sharded generation unchanged: identical tokens and digests.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-013 closed -- nothing is copied, so no NOTICE transfers (D36) A file-by-file comparison against a read-only clone: no shared module names, four same-named files and none above 5% line overlap, and only format/model vocabulary in common. THIRD_PARTY_NOTICES.md records the position and what would change it; check_provenance.py guards it offline.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: a gate for the documentation's claims (D35), and four stale rows closed check_status_claims.py checks test counts, decision citations and the release state, and refuses to pass on zero. Its first run found a News entry that was true when written, which is why claims and records are now separate ideas. DC-004, DC-034, DC-035 and DC-044 closed -- all four were done.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker + AGENTS: D34 settles the zero and NaN contract; no test is skipped now Three defects, not one: the GPU never implemented D11's denormal flush, the sign of zero is emergent on both sides, and a NaN was undefined. The additive fix was folded away by the Metal compiler, so the rule is a comparison with a bit-pattern result. Real-model digest unchanged (b0d382dbabf36df0...). DC-087 and DC-090 closed.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D33 corrects D32's repeat-read guess with an instrument 611 tensors asked for more than once, layer 0's x8 -- and 8 is the number of forwards, not a duplicate inside one. The arithmetic closes to the byte. DC-081's row now says what observability exists.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-106 closed -- the dense backbone is resident (D32) 8.35 GB of dense payload reads became 1.04 GB (8x) across three cached-decode steps, with identical tokens and an identical trace digest. Packed bytes, 1.04 GB resident, and stacked experts cannot reach the cache by construction.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Roadmap and tracker: D12 is decided (D31), so the open list is the tok/s measurements

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D12 decided by measurement -- one slot, from a budget 2218 requests, 0 hits and byte-identical reads at 1, 2, 4, 8 and 16 slots on the real install. The bank is now sized from a budget and the layer's geometry; the expected failure in SlotBudgetTests is deleted deliberately, as its own note asked. DC-091 and DC-092 closed.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + Architecture + tracker: DC-083 closed, security posture written down The wire audit found two gaps -- a peer's declared geometry reached a precondition, and a count bound that was per-number rather than per-product. Both fixed and fuzzed; the Architecture page now states what is exposed, what is refused, and what is deliberately absent.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • Tracker and Roadmap: DC-108 and DC-109 are closed, so stop calling them open Both were named as what M1 and M2 leave open. The install verifies from Python now and the sharded generation CLI exists; the Roadmap says what actually remains, which is D12 and the tok/s measurements.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: DC-108 closed -- Python reads and verifies the install 693 tensors tiling 21,700,655,616 bytes exactly, every role matching the policy, all 693 payload digests verified in 26.3 s uncached with disk and swap flat. The source-checkpoint comparison is reported as not checked. Python tests 179 -> 200.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker: D28 sharded generation; DC-109 closed The cached path would have ignored the shard and produced plausible tokens; one entry point fixes it. Two-machine cached generation produced the reference's tokens and digest on the real 35B install. M3's instrument exists; its 3x gate waits for a quiet farm.

    @Pummelchen Pummelchen committed Sep 16, 2026
  • News + tracker + Roadmap: all four machines run one forward (D27), M3 under way Full mesh, every node bit-identical to the single-node trace (58518422914cfe2b...). That is M3's functional half; its 3x throughput gate needs a quiet farm and a sharded generation CLI (DC-109).

    @Pummelchen Pummelchen committed Sep 16, 2026
  • M1's phase row said In progress while M2's said Done M1's gate passed and was re-established in an earlier round; the phase table never caught up.

    @Pummelchen Pummelchen committed Sep 16, 2026