From da07d0ac4ac5e6202de650be373ac0c659a1d729 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 18:44:26 +0000 Subject: [PATCH 1/3] =?UTF-8?q?W0=20(D-DCR-0):=20the=20replay=20budget,=20?= =?UTF-8?q?measured=20=E2=80=94=20mask=20half=20dominates=203.8x?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Harness: lance-graph-contract/examples/dcr_w0_replay_budget (release, deterministic LCG — a probe for a replay plan is itself replayable). Corpus magnitudes are READ from dismech-causality-v3-v1 §11, not re-derived: 2,449 oracle edges / 534 diseases, 4,076 restraint, 361 unknown. Measured: - step (NarsTruth::revision + EvidenceMask::intersection): 14,285 steps/ms at a 4096-wide candidate set (70 ns/step). - branching shrink 1.66x per evidence item, flat 10^3..10^5 — labelled FIXTURE-SET: it measures the mechanism, never the corpus's real discriminative power (that needs D-CV3-0..2's frozen TSVs). - KILL check did NOT fire: scan 0.906 ms vs decision 0.008 ms, crossover ~25 chains, corpus ~98x above it => W5 stays live on cost (still HELD on the operator rung 5-9 ruling, a different gate). - kernel split, the ALU wave's actual question: revision 24.0 ns vs 4096-bit intersect 90.6 ns => MASK dominates 3.8x. The 64x64 tile is aimed at the half that costs; the whole oracle arm replays in 2.74 ms, so BUY is stated at >10x that in one budget. Correctly aimed, correctly deferred. Also ⊘ CORRECTION: the plan's §0 audit was under-cited — it swept the sibling repos (F-RLR-11) but not this repo's own contract module list or plans index, missing contract::dismech_evidence (which already carries the hidden-mediator oracle + epistemic-restraint control populations) and dismech-causality-v3-v1 §11 (the held-out benchmark, measured three ways). Nothing is contradicted; the cost was scoping, and W1-W3 now consume those arms instead of inventing a corpus. Plan §3a/§3b, EPIPHANIES prepend, STATUS_BOARD D-DCR-0 Shipped + D-DCR-0a. Gates: contract 1278/1278, clippy -D warnings clean, fmt clean, supersession index regenerated last. --- .claude/board/EPIPHANIES.md | 61 +++ .claude/board/STATUS_BOARD.md | 3 +- .claude/plans/dismech-causal-replay-v1.md | 61 +++ .../examples/dcr_w0_replay_budget.rs | 370 ++++++++++++++++++ 4 files changed, 494 insertions(+), 1 deletion(-) create mode 100644 crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 043a55caa..84f9de343 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,64 @@ +## 2026-08-31 — E-W0-MEASURED-THE-MASK-HALF-DOMINATES-AND-THE-PLAN-WAS-UNDER-CITED-1 — two findings from one W0 run + +**Status:** FINDING (measurements) + CORRECTION (prior-art citation gap). +**Confidence:** measured — `lance-graph-contract/examples/dcr_w0_replay_budget`, +release, committed and re-runnable. D-DCR-0. + +### 1. The kernel split is the ALU wave's real answer — and it points the right way + +The deferred p64 64×64 wave was justified by shape ("4096 bits = one node's +budget"). W0 asked which half of a replay step that tile would even touch: + +| kernel | rate | per op | +|---|---|---| +| `NarsTruth::revision` alone | 41,596 ops/ms | 24.0 ns | +| 4096-bit `intersection` + count | 11,038 ops/ms | 90.6 ns | + +**MASK dominates by 3.8×.** The tile is aimed at the half that actually costs +— the shape argument survives contact with a measurement, which is not the +usual outcome. What does NOT survive is urgency: the whole 2,449-edge oracle +arm replays in **2.74 ms** at chain length 16, so the BUY threshold is stated +as *>10× this corpus in one budget*, and until a workload asks for that the +scalar path is not the bottleneck. + +### 2. The pre-registered KILL check did NOT fire — W5 stays live + +Full scan of the oracle arm (2,449 chains, len 4): **0.906 ms**. One frontier +decision over 64 candidate observations: **0.008 ms** — ~100× cheaper, with +the crossover at **~25 chains**. The corpus is ~98× above crossover, so +scheduling is not decoration even at DisMech's small scale. Recorded because +the plan pre-registered the opposite outcome as a real possibility; it was +checked, not assumed. + +### 3. ⊘ CORRECTION — `dismech-causal-replay-v1`'s §0 audit was UNDER-CITED + +The plan (#1117, merged) audited OGAR's `ogar-dismech`, the consumer +transcode, and `lance-graph-ontology`. It did **not** cite two pieces of prior +art in the repo it was written for: + +- **`contract::dismech_evidence`** (686 LOC, shipped) — the compact evidence + vocabulary measured on the real corpus, including the two populations that + are *already* a two-sided replay falsifier: `IndirectKnownIntermediates` + (the hidden-mediator oracle) and `IndirectUnknownIntermediates` (the + epistemic-restraint control, where "recovering" a mediator IS the failure). +- **`dismech-causality-v3-v1.md`** (2026-08-21) + its `D-CV3-0..6` board rows + — whose §11 specifies the held-out benchmark with measured arms: **2,449** + oracle edges over **534** diseases, **4,076** restraint rows, **361** + unknown rows, split A/B/C/D/E, "two-sided by construction". + +**Nothing in the plan is contradicted** — "no baked slab exists" holds (the +corpus pin is D-CV3-0, still Queued). The cost was scoping: W1–W3 would have +invented a falsifier corpus that §11 already specifies with numbers measured +three independent ways. W0 therefore consumes those magnitudes instead of +inventing scales, and the reconciliation is now the plan's §3a. + +**The rule this instance re-proves** (rubicon §F's own ⊘ made the identical +mistake five days earlier, and `CLAUDE.md` states it): *grep the existing ~100 +files before writing a new one* — and "prior art" includes the plans index and +the contract's own module list, not only the sibling repos an audit happens to +be thinking about. A sibling-repo sweep (F-RLR-11) does not discharge the +same-repo sweep. + ## 2026-08-31 — E-A-THRESHOLD-IS-BOUND-TO-ITS-STATISTIC-AND-ITS-SAMPLE-1 — a constant carried across either one silently becomes a different gate **Status:** FINDING — two independent instances in one wave, both caught by diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index be2adc0af..5914c46ce 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -2,7 +2,8 @@ | D-id | deliverable | status | |---|---|---| -| D-DCR-0 | W0 measurements: chain-step evals/ms, EvidenceMask branching shrink, palette round-trip; ALU BUY threshold stated | Queued | +| D-DCR-0 | W0 measurements: chain-step evals/ms, EvidenceMask branching shrink, palette round-trip; ALU BUY threshold stated | **Shipped** — 14,285 steps/ms @4096-wide; shrink 1.66x (fixture-set); KILL did NOT fire (scan 0.906 ms vs decision 0.008 ms, crossover ~25 chains); MASK dominates revision 3.8x; BUY at >10x the 2.74 ms oracle arm. Probe `dcr_w0_replay_budget` | +| D-DCR-0a | prior-art reconciliation: `contract::dismech_evidence` + `dismech-causality-v3-v1` §11 arms (2,449 / 4,076 / 361) are W1-W3's falsifier; plan §3a | **Shipped** (E-W0-MEASURED-THE-MASK-HALF-DOMINATES-...-1) | | D-DCR-1 | replay core: loco calls under the dismech vocabulary -> CausalEdge64/NarsTruth steps -> temporal.rs trace; determinism + perturbation falsifiers | Queued (after D-DCR-0) | | D-DCR-2 | Mengenlehre candidate evaluation via `contract::revision::EvidenceMask` (support ∩ / refute ∖, skip-words as ordinals) | Queued | | D-DCR-3 | counterfactual replay (edge cut through `contract::counterfactual`, Pearl rung 3), two-sided load-bearing/redundant gates | Queued | diff --git a/.claude/plans/dismech-causal-replay-v1.md b/.claude/plans/dismech-causal-replay-v1.md index 03c3e9bd0..e8abba697 100644 --- a/.claude/plans/dismech-causal-replay-v1.md +++ b/.claude/plans/dismech-causal-replay-v1.md @@ -204,3 +204,64 @@ source in / one shape out. Haiku: only the guarded-executor card. | D-DCR-4 | Σ transport + Shannon readout | Queued (entropy decision first) | | D-DCR-5 | frontier scheduling | **HELD** (operator rung table + W0 KILL) | | D-DCR-6 | consumer-leg pointer honoured (no corpus data lands here) | standing gate | + +--- + +## §3a — PRIOR-ART RECONCILIATION (added 2026-08-31, W0) + +⊘ **§0's audit was under-cited.** It swept the sibling repos (F-RLR-11) but +not this repo's own contract module list or plans index. Two pieces of prior +art belong in the state table, and both STRENGTHEN the plan: + +| prior art | what it gives this plan | +|---|---| +| `contract::dismech_evidence` (686 LOC, shipped) | the measured evidence vocabulary AND a ready-made two-sided falsifier population: `IndirectKnownIntermediates` = hidden-mediator oracle; `IndirectUnknownIntermediates` = epistemic-restraint control (recovering a mediator there IS the failure) | +| `dismech-causality-v3-v1.md` §11 + `D-CV3-0..6` | the held-out benchmark, already specified with measured arms: **2,449** oracle edges / **534** diseases, **4,076** restraint rows, **361** unknown rows; splits A–E reported separately; "two-sided by construction" | + +**The join:** D-CV3's benchmark **is** this plan's W1–W3 falsifier; D-DCR is +the engine it grades. W1–W3 therefore consume those arms (once D-CV3-0..2 +land the frozen TSVs consumer-side) and MUST NOT invent a parallel corpus. +Nothing in §0 is contradicted — "no baked slab exists" still holds, and the +corpus pin is D-CV3-0, Queued. + +## §3b — W0 RESULT (D-DCR-0, measured 2026-08-31) + +Harness: `crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs` +(release; deterministic LCG, no clock seeding — a probe for a replay plan is +itself replayable). Corpus magnitudes are READ from §11, never re-derived. + +**1. Step throughput** — one step = `NarsTruth::revision` + `EvidenceMask::intersection`: + +| candidate set | chain len | steps/ms | ns/step | +|---|---|---|---| +| 64 | 16 | 29,184 | 34.3 | +| 1,024 | 16 | 23,628 | 42.3 | +| 4,096 | 16 | 14,285 | 70.0 | + +**2. Branching shrink** — 1.53×–1.66× per evidence item, flat from 10³ to 10⁵ +candidates. **Fixture-set densities** (2/3 support, 1/10 refute): this measures +the MECHANISM's cost and scaling, never the corpus's real discriminative power. +That number needs the frozen oracle/restraint TSVs (D-CV3-0..2). + +**3. KILL check — did NOT fire.** Full scan of the oracle arm (2,449 chains, +len 4) = **0.906 ms**; one frontier decision over 64 observations = **0.008 ms** +(~100× cheaper), crossover at **~25 chains**. The corpus sits ~98× above +crossover ⇒ **W5 stays live on cost grounds** (it remains HELD on the operator +rung 5–9 table, which is a different gate). + +**4. Kernel split — the ALU wave's actual question.** `NarsTruth::revision` +alone 41,596 ops/ms (24.0 ns); 4096-bit intersect+count 11,038 ops/ms +(90.6 ns) ⇒ **MASK dominates by 3.8×**. A 64×64 tile accelerates the half that +costs — the "4096 bits = one node" shape argument survives measurement. + +**5. ALU BUY threshold (stated, as W0 owed).** The whole oracle arm replays in +**2.74 ms** at chain length 16. **BUY only when a workload sustains +>10× that in one budget** (≈143,000 steps/ms); below it the scalar path is not +the bottleneck. The tile is correctly aimed and correctly deferred. + +**What W0 did NOT measure**, so nobody cites it as if it had: the real +per-evidence discriminative power (needs D-CV3-0..2), `CausalEdge64`'s packed +step (planner-side, one dependency layer out of this zero-dep crate), and any +loco dispatch cost (OGAR-side palette; the round-trip is covered by +`ogar-dismech`'s own tests, and duplicating them here would be a second +truth). diff --git a/crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs b/crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs new file mode 100644 index 000000000..c74c52446 --- /dev/null +++ b/crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs @@ -0,0 +1,370 @@ +//! **D-DCR-0 (W0) — the replay budget, measured.** +//! +//! `.claude/plans/dismech-causal-replay-v1.md` §3 W0. Three numbers the +//! deferred p64 64x64 ALU wave states its BUY threshold in, plus the +//! pre-registered KILL check for W5 (frontier scheduling). +//! +//! ```text +//! cargo run -p lance-graph-contract --example dcr_w0_replay_budget --release +//! ``` +//! +//! # What is measured, and what it is measured ON +//! +//! **The corpus scale is NOT invented here.** `dismech-causality-v3-v1.md` +//! §3a/§11 already measured the real supervision corpus three independent +//! ways: **2,449** oracle edges (`INDIRECT_KNOWN_INTERMEDIATES` that actually +//! name a mediator) over **534** diseases, a **4,076**-row restraint arm +//! (4,150 minus the 74 that contradict their own label) and **361** `UNKNOWN` +//! rows kept as a third arm. This probe therefore reports its per-scale +//! numbers against those magnitudes rather than against a round number +//! somebody liked. +//! +//! 1. **Step throughput** — one replay step is a `NarsTruth::revision` +//! (evidence fusion) plus the `EvidenceMask` intersect that keeps the +//! candidate set live. Both are contract-level and already shipped; this +//! times the composition, never a new kernel. +//! 2. **Branching shrink** — how much a single evidence item reduces a +//! candidate set, at 10^3 / 10^4 / 10^5 chains. This is the Mengenlehre +//! half: `intersection` for support, `difference` for refute. +//! 3. **KILL check (pre-registered, two-sided)** — if a FULL SCAN of the real +//! corpus costs less than one frontier decision, the scheduler is +//! decoration at DisMech scale and W5 is descoped *for this corpus*, with +//! the crossover size reported so a bigger corpus can re-open it. +//! +//! # Non-carriers +//! +//! `Bits` below is an EXAMPLE-LOCAL fixture implementing the existing +//! [`EvidenceMask`] trait for sets wider than the shipped `u64` impl. The +//! trait exists to be implemented; a probe fixture is not a new carrier and +//! nothing outside this file may use it (`F-RLR-2` is about the production +//! path). The step kernel is `NarsTruth` + `EvidenceMask` — `CausalEdge64`'s +//! packed form lives planner-side (`causal-edge`), one dependency layer out +//! from this zero-dep crate, and is deliberately NOT pulled in to time it. + +use std::time::Instant; + +use lance_graph_contract::exploration::NarsTruth; +use lance_graph_contract::revision::EvidenceMask; + +/// Example-local wide bitset (see the module doc's non-carrier note). +#[derive(Clone, PartialEq, Eq, Debug)] +struct Bits(Vec); + +impl Bits { + fn with_capacity(n: usize) -> Self { + Bits(vec![0u64; n.div_ceil(64)]) + } + fn set(&mut self, i: usize) { + self.0[i / 64] |= 1u64 << (i % 64); + } + fn count(&self) -> usize { + self.0.iter().map(|w| w.count_ones() as usize).sum() + } +} + +impl EvidenceMask for Bits { + fn empty() -> Self { + Bits(Vec::new()) + } + fn is_empty(&self) -> bool { + self.0.iter().all(|w| *w == 0) + } + fn union(&self, other: &Self) -> Self { + Bits(zip_words(&self.0, &other.0, |a, b| a | b)) + } + fn intersection(&self, other: &Self) -> Self { + Bits(zip_words(&self.0, &other.0, |a, b| a & b)) + } + fn difference(&self, other: &Self) -> Self { + Bits(zip_words(&self.0, &other.0, |a, b| a & !b)) + } + fn is_subset_of(&self, other: &Self) -> bool { + self.0 + .iter() + .enumerate() + .all(|(i, w)| w & !other.0.get(i).copied().unwrap_or(0) == 0) + } +} + +fn zip_words(a: &[u64], b: &[u64], f: impl Fn(u64, u64) -> u64) -> Vec { + let n = a.len().max(b.len()); + (0..n) + .map(|i| { + f( + a.get(i).copied().unwrap_or(0), + b.get(i).copied().unwrap_or(0), + ) + }) + .collect() +} + +/// Deterministic PRNG — a probe must be replayable (that is this plan's whole +/// keystone), so no `rand`, no clock seeding. +struct Lcg(u64); +impl Lcg { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1); + self.0 + } + fn below(&mut self, n: usize) -> usize { + (self.next() >> 33) as usize % n + } +} + +/// The real supervision-corpus magnitudes, measured in +/// `dismech-causality-v3-v1.md` §3a/§11 — never re-derived here. +const ORACLE_EDGES: usize = 2_449; +const RESTRAINT_ROWS: usize = 4_076; +const UNKNOWN_ROWS: usize = 361; +const ORACLE_DISEASES: usize = 534; + +/// One replay step: fuse this step's evidence into the running truth, and +/// keep the candidate set live. Both halves are shipped contract surface. +#[inline] +fn replay_step( + running: NarsTruth, + step_evidence: NarsTruth, + live: &Bits, + support: &Bits, +) -> (NarsTruth, Bits) { + (running.revision(&step_evidence), live.intersection(support)) +} + +fn measure_step_throughput(chain_len: usize, chains: usize, set_bits: usize) -> (f64, f64) { + let mut live = Bits::with_capacity(set_bits); + for i in 0..set_bits { + live.set(i); + } + let mut rng = Lcg(0x051E_D270_B5A1_11E5); + // Pre-build per-step support masks so the timed loop measures the STEP, + // not fixture construction. + // 2/3 density: evidence that halves the candidate set at EVERY step is + // unrealistically strong and collapses the set within ~12 steps, which + // would put a clone-reset inside the timed loop and measure the fixture. + let supports: Vec = (0..chain_len) + .map(|_| { + let mut m = Bits::with_capacity(set_bits); + for i in 0..set_bits { + if rng.below(3) != 0 { + m.set(i); + } + } + m + }) + .collect(); + let ev = NarsTruth::new(0.8, 0.6); + + let t0 = Instant::now(); + let mut sink = 0usize; + for _ in 0..chains { + let mut running = NarsTruth::prior(); + let mut cur = live.clone(); + for s in &supports { + let (r, c) = replay_step(running, ev, &cur, s); + running = r; + cur = c; + } + sink += cur.count() + (running.expectation() > 0.0) as usize; + } + let dt = t0.elapsed().as_secs_f64(); + std::hint::black_box(sink); + let steps = (chain_len * chains) as f64; + (steps / dt / 1000.0, dt * 1000.0) // (steps per ms, total ms) +} + +/// The two halves of a step, timed apart. This is the question the deferred +/// 64x64 ALU wave actually asks: a tile is 4096 bits = one node's budget, so +/// it accelerates the MASK half. If revision dominates, the tile is aimed at +/// the wrong half and the BUY is not merely early, it is misdirected. +fn measure_kernel_split(set_bits: usize, iters: usize) -> (f64, f64) { + let mut rng = Lcg(0x0000_DDBA_11C0_FFEE); + let mut a = Bits::with_capacity(set_bits); + let mut b = Bits::with_capacity(set_bits); + for i in 0..set_bits { + if rng.below(3) != 0 { + a.set(i); + } + if rng.below(3) != 0 { + b.set(i); + } + } + let (x, y) = (NarsTruth::new(0.7, 0.5), NarsTruth::new(0.4, 0.3)); + + let t0 = Instant::now(); + let mut acc = NarsTruth::prior(); + for _ in 0..iters { + acc = std::hint::black_box(&acc).revision(std::hint::black_box(&x)); + acc = std::hint::black_box(&acc).revision(std::hint::black_box(&y)); + } + std::hint::black_box(acc.expectation()); + let revision_per_ms = (iters * 2) as f64 / (t0.elapsed().as_secs_f64() * 1000.0); + + let t1 = Instant::now(); + let mut sink = 0usize; + for _ in 0..iters { + sink += std::hint::black_box(&a) + .intersection(std::hint::black_box(&b)) + .count(); + } + std::hint::black_box(sink); + let mask_per_ms = iters as f64 / (t1.elapsed().as_secs_f64() * 1000.0); + + (revision_per_ms, mask_per_ms) +} + +/// One frontier decision, priced honestly: score EVERY candidate observation +/// by the candidate-set split it would produce. That is the W4/W5 shape +/// (expected information gain), so the KILL check compares like with like. +fn measure_frontier_decision(candidates: usize, observations: usize) -> (f64, usize) { + let mut live = Bits::with_capacity(candidates); + for i in 0..candidates { + live.set(i); + } + let mut rng = Lcg(0x0D15_EC40_11EA_DBEE); + let obs: Vec = (0..observations) + .map(|_| { + let mut m = Bits::with_capacity(candidates); + for i in 0..candidates { + if rng.below(4) != 0 { + m.set(i); + } + } + m + }) + .collect(); + + let t0 = Instant::now(); + let mut best = (usize::MAX, 0usize); + let total = live.count(); + for (i, o) in obs.iter().enumerate() { + let kept = live.intersection(o).count(); + let dropped = live.difference(o).count(); + // Balanced split = most informative; |kept - dropped| is its proxy. + let imbalance = kept.abs_diff(dropped); + if imbalance < best.0 { + best = (imbalance, i); + } + std::hint::black_box((kept, dropped, total)); + } + (t0.elapsed().as_secs_f64() * 1000.0, best.1) +} + +/// A full scan: replay every chain once, no frontier at all. +fn measure_full_scan(chains: usize, chain_len: usize, set_bits: usize) -> f64 { + let (_, ms) = measure_step_throughput(chain_len, chains, set_bits); + ms +} + +fn main() { + println!("D-DCR-0 (W0) — replay budget"); + println!("corpus (measured in dismech-causality-v3-v1 §3a/§11, not re-derived here):"); + println!(" oracle arm {ORACLE_EDGES} edges over {ORACLE_DISEASES} diseases"); + println!(" restraint arm {RESTRAINT_ROWS} rows"); + println!(" unknown arm {UNKNOWN_ROWS} rows"); + + // ── 1. Step throughput ─────────────────────────────────────────────── + println!("\n1. STEP THROUGHPUT (NarsTruth::revision + EvidenceMask::intersection)"); + println!( + " {:>10} {:>10} {:>14} {:>12}", + "cand.set", "chain len", "steps/ms", "ns/step" + ); + let mut steps_per_ms_at_2k = 0.0f64; + for &set_bits in &[64usize, 1_024, 4_096] { + for &chain_len in &[4usize, 16] { + let chains = 2_000; + let (spms, _) = measure_step_throughput(chain_len, chains, set_bits); + if set_bits == 4_096 && chain_len == 16 { + steps_per_ms_at_2k = spms; + } + println!( + " {set_bits:>10} {chain_len:>10} {spms:>14.1} {:>12.1}", + 1_000_000.0 / spms + ); + } + } + + // ── 2. Branching shrink ────────────────────────────────────────────── + println!("\n2. BRANCHING SHRINK (one evidence item, support ∩ / refute ∖)"); + println!(" NOTE: densities are FIXTURE-SET (2/3 support, 1/10 refute), so this"); + println!(" measures the MECHANISM's cost and scaling — never the corpus's real"); + println!(" discriminative power. That needs the frozen oracle/restraint TSVs"); + println!(" (D-CV3-0..2, consumer-side)."); + println!( + " {:>10} {:>12} {:>12} {:>10}", + "chains", "after ∩", "after ∖", "factor" + ); + for &n in &[1_000usize, 10_000, 100_000] { + let mut live = Bits::with_capacity(n); + for i in 0..n { + live.set(i); + } + let mut rng = Lcg(0xA5A5_1234_DEAD_C0DE); + let mut support = Bits::with_capacity(n); + let mut refute = Bits::with_capacity(n); + for i in 0..n { + if rng.below(3) != 0 { + support.set(i); + } + if rng.below(10) == 0 { + refute.set(i); + } + } + let after_and = live.intersection(&support); + let after_diff = after_and.difference(&refute); + println!( + " {n:>10} {:>12} {:>12} {:>10.2}x", + after_and.count(), + after_diff.count(), + n as f64 / after_diff.count().max(1) as f64 + ); + } + + // ── 3. KILL check ──────────────────────────────────────────────────── + println!("\n3. KILL CHECK (pre-registered): is a full scan cheaper than one decision?"); + let scan_ms = measure_full_scan(ORACLE_EDGES, 4, 4_096); + let (dec_ms, _pick) = measure_frontier_decision(ORACLE_EDGES, 64); + println!(" full scan of {ORACLE_EDGES} chains (len 4): {scan_ms:.3} ms"); + println!(" one frontier decision over 64 observations: {dec_ms:.3} ms"); + let verdict = if scan_ms <= dec_ms { + "KILL FIRES — scheduling is decoration at DisMech scale; W5 descoped for THIS corpus" + } else { + "KILL does not fire — a decision is cheaper than a scan; W5 stays live" + }; + println!(" => {verdict}"); + // Crossover: how many chains before a scan costs a decision? + let per_chain_ms = scan_ms / ORACLE_EDGES as f64; + println!( + " crossover: a scan costs one decision at ~{:.0} chains ({:.1}x the oracle arm)", + dec_ms / per_chain_ms, + (dec_ms / per_chain_ms) / ORACLE_EDGES as f64 + ); + + // ── 4. Kernel split — which half a 64x64 tile would even touch ─────── + println!("\n4. KERNEL SPLIT (the ALU wave's actual question)"); + let (rev_pm, mask_pm) = measure_kernel_split(4_096, 200_000); + println!( + " NarsTruth::revision alone : {rev_pm:>10.0} ops/ms ({:>6.1} ns)", + 1_000_000.0 / rev_pm + ); + println!( + " 4096-bit intersect+count : {mask_pm:>10.0} ops/ms ({:>6.1} ns)", + 1_000_000.0 / mask_pm + ); + let (dominant, factor) = if mask_pm < rev_pm { + ("MASK", rev_pm / mask_pm) + } else { + ("REVISION", mask_pm / rev_pm) + }; + println!(" => {dominant} dominates by {factor:.1}x — a 64x64 tile accelerates the MASK half"); + + // ── 5. ALU BUY threshold, stated in the measured numbers ───────────── + println!("\n5. ALU BUY THRESHOLD (deferred p64 64x64 wave)"); + println!(" measured step rate (4096-wide set, len 16): {steps_per_ms_at_2k:.0} steps/ms"); + let corpus_ms = ORACLE_EDGES as f64 * 16.0 / steps_per_ms_at_2k; + println!(" => whole oracle arm at chain len 16: {corpus_ms:.2} ms"); + println!( + " BUY only when a workload needs > {:.0} steps/ms sustained, i.e. > {:.0}x this corpus\n in one budget — until then the scalar path is not the bottleneck.", + steps_per_ms_at_2k * 10.0, + 10.0 + ); +} From 5d9147d7d29aac469d4c56082afcb6ddecbdbf83 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 18:59:13 +0000 Subject: [PATCH 2/3] =?UTF-8?q?review=20#1118=20(codex):=20W0=20corrected?= =?UTF-8?q?=20=E2=80=94=20the=20promised=20kernel,=20alloc-free=20masks,?= =?UTF-8?q?=20both=20KILL=20scales?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three codex findings were valid, and a fourth surfaced while fixing them. None was a bug in the code under test; every one was a defect in the probe's FIXTURE: P1 the eval timed NarsTruth::revision (f32, contract-side) where the plan defines it as NarsTables lookup + CausalEdge64 revision. Disclosing the substitution did not make it the promised measurement. Probe MOVED to lance-graph-planner, where causal-edge is reachable, and now times NarsTables::revise + CausalEdge64:: forward: 34.7 ns/step (28,818 steps/ms). P1 the mask fixture was Bits(Vec) — a heap alloc inside every timed intersection — while impl EvidenceMask for [u64; N] already ships (revision.rs:70) and IS the p64 shape ([u64;64] = 4096 bits). Both are now timed and the delta reported: 61.5 ns alloc-free vs 73.4 ns allocating (~16%). P2 the KILL gate ran at 2,449 chains where the plan pre-registers 10^5. Both scales now run at one candidate width; it fires at NEITHER (10^5: 13.88 ms scan vs 0.007 ms decision; 2,449: 0.340 ms). Crossover ~53 chains. + self-caught: dense_mask(rng, 1) sets NO bits (x % 1 == 0), so the frontier decision was scored against an empty live set. all_ones() is now its own constructor; dense_mask asserts one_in >= 2. Net: MASK dominates by 1.77x, NOT the 3.8x v1 reported. The direction survives (a 64x64 tile is aimed at the half that costs) but the margin is under half the first claim, so the ALU case is materially weaker than v1 said; BUY stands at >10x the 1.36 ms oracle arm. Records are append-only: plan §3b kept and superseded by §3c with the reason; STATUS_BOARD D-DCR-0 superseded in place; EPIPHANIES prepend E-EVERY-DEFECT-IN-A-MEASUREMENT-WAS-IN-ITS-FIXTURE-NOT-ITS-CODE-1 (23888 -> 23935 lines, verified non-truncating). Gates: planner 370/370, my example clippy -D warnings clean, fmt clean. NOTE: workspace --all-targets clippy is red on probe_r2il_* examples from #1051 — pre-existing, untouched here, and rust-test.yml documents that a workspace -D warnings step would be red. --- .claude/board/EPIPHANIES.md | 47 +++ .claude/board/STATUS_BOARD.md | 2 +- .claude/plans/dismech-causal-replay-v1.md | 41 ++ .../examples/dcr_w0_replay_budget.rs | 370 ------------------ .../examples/dcr_w0_replay_budget.rs | 312 +++++++++++++++ 5 files changed, 401 insertions(+), 371 deletions(-) delete mode 100644 crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs create mode 100644 crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 84f9de343..7dbdb4509 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,50 @@ +## 2026-08-31 — E-EVERY-DEFECT-IN-A-MEASUREMENT-WAS-IN-ITS-FIXTURE-NOT-ITS-CODE-1 — four in one probe, three found by review + +**Status:** FINDING — corrective, measured both ways. +**Confidence:** measured — `lance-graph-planner/examples/dcr_w0_replay_budget` +(corrected) vs the superseded contract-side v1. D-DCR-0, codex review on #1118. + +W0's first probe produced a headline ("MASK dominates by 3.8×") that set the +direction of a deferred hardware wave. Codex filed three findings; **all three +were valid**, and a fourth surfaced while fixing them. Not one was a bug in +the code under test. Every one was a defect in the **fixture**: + +| # | fixture defect | what it faked | +|---|---|---| +| 1 | timed `NarsTruth::revision` (f32) where the plan defines the eval as `NarsTables` lookup + `CausalEdge64` revision | a substitute kernel, disclosed in a doc-comment but still the source of the headline number | +| 2 | mask fixture was `Vec` → `collect()` allocation inside the timed loop, though `impl EvidenceMask for [u64; N]` already ships and is the real p64 shape | charged `malloc` to the arithmetic the proposed tile would accelerate | +| 3 | ran the KILL gate at 2,449 chains where the plan pre-registered **10^5** | a pre-registered gate recorded as evaluated without being evaluated | +| 4 | `dense_mask(rng, 1)` sets **no** bits (`x % 1 == 0` always) | scored a frontier decision against an EMPTY live set (caught in the fix pass, no review help) | + +**Corrected:** promised step **34.7 ns**, alloc-free 4096-bit mask **61.5 ns** +⇒ **MASK dominates 1.77×**, not 3.8×. KILL fires at neither scale +(10^5: 13.88 ms scan vs 0.007 ms decision; 2,449: 0.340 ms). Oracle arm +**1.36 ms** at chain length 16. + +**The direction survived; the margin did not.** A 64×64 tile is still aimed at +the half that costs — but at 1.77× the case is materially weaker than the +first pass claimed, and a BUY argued on 3.8× would have been argued on a +number that measured `malloc` and an f32 stand-in. + +**The rule.** *A measurement's fixture is part of its claim.* The existing +falsifiability rule covers tests that cannot fail; this is its measurement +twin — a probe whose fixture is wrong produces a number that CANNOT be +falsified by re-running it, because the fixture reproduces faithfully every +time. Concretely, before a probe's number enters a plan or board: +1. **Does it time the kernel the plan NAMES?** A documented substitution is + still a substitution; move the probe to where the real kernel lives. +2. **Does the timed loop allocate?** If the shipped type is fixed-width, time + the fixed-width type — and report both when correcting, so the delta is + visible rather than swapped in silently. +3. **Is the pre-registered workload the one that ran?** Changing the scale + un-runs the gate. +4. **Does the fixture construct what its name says?** (`all_ones()` is a + constructor, not `dense(1)`.) + +Cross-ref: `E-W0-MEASURED-THE-MASK-HALF-DOMINATES-AND-THE-PLAN-WAS-UNDER-CITED-1` +(the entry this corrects — its §1 table is superseded, its §3 prior-art +correction stands), plan `dismech-causal-replay-v1.md` §3c. + ## 2026-08-31 — E-W0-MEASURED-THE-MASK-HALF-DOMINATES-AND-THE-PLAN-WAS-UNDER-CITED-1 — two findings from one W0 run **Status:** FINDING (measurements) + CORRECTION (prior-art citation gap). diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 5914c46ce..05ff879cf 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -2,7 +2,7 @@ | D-id | deliverable | status | |---|---|---| -| D-DCR-0 | W0 measurements: chain-step evals/ms, EvidenceMask branching shrink, palette round-trip; ALU BUY threshold stated | **Shipped** — 14,285 steps/ms @4096-wide; shrink 1.66x (fixture-set); KILL did NOT fire (scan 0.906 ms vs decision 0.008 ms, crossover ~25 chains); MASK dominates revision 3.8x; BUY at >10x the 2.74 ms oracle arm. Probe `dcr_w0_replay_budget` | +| D-DCR-0 | W0 measurements: chain-step evals/ms, EvidenceMask branching shrink, palette round-trip; ALU BUY threshold stated | **Shipped (CORRECTED #1118)** — promised kernel (NarsTables::revise + CausalEdge64::forward) **34.7 ns/step**; alloc-free `[u64;64]` mask 61.5 ns ⇒ **MASK dominates 1.77x** (v1's 3.8x superseded: substitute kernel + allocating fixture); KILL does not fire at EITHER the pre-registered 10^5 (13.88 ms) or the real 2,449 arm (0.340 ms), crossover ~53 chains; BUY at >10x the **1.36 ms** oracle arm. Probe `lance-graph-planner/examples/dcr_w0_replay_budget` | | D-DCR-0a | prior-art reconciliation: `contract::dismech_evidence` + `dismech-causality-v3-v1` §11 arms (2,449 / 4,076 / 361) are W1-W3's falsifier; plan §3a | **Shipped** (E-W0-MEASURED-THE-MASK-HALF-DOMINATES-...-1) | | D-DCR-1 | replay core: loco calls under the dismech vocabulary -> CausalEdge64/NarsTruth steps -> temporal.rs trace; determinism + perturbation falsifiers | Queued (after D-DCR-0) | | D-DCR-2 | Mengenlehre candidate evaluation via `contract::revision::EvidenceMask` (support ∩ / refute ∖, skip-words as ordinals) | Queued | diff --git a/.claude/plans/dismech-causal-replay-v1.md b/.claude/plans/dismech-causal-replay-v1.md index e8abba697..408b20b11 100644 --- a/.claude/plans/dismech-causal-replay-v1.md +++ b/.claude/plans/dismech-causal-replay-v1.md @@ -265,3 +265,44 @@ step (planner-side, one dependency layer out of this zero-dep crate), and any loco dispatch cost (OGAR-side palette; the round-trip is covered by `ogar-dismech`'s own tests, and duplicating them here would be a second truth). + +## §3c — W0 CORRECTED (codex review on #1118; three findings, all valid) + +⊘ **§3b's numbers are SUPERSEDED.** They are kept above (append-only) because +the correction is the finding. Three defects, all pushing the same direction — +they inflated the mask half and measured a kernel this plan never named: + +| # | defect | consequence | +|---|---|---| +| P1 | v1 timed `NarsTruth::revision` (f32, contract-side); §3 W0 defines the eval as **`NarsTables` lookup + `CausalEdge64` revision** | the headline throughput and the 2.74 ms corpus figure came from a substitute kernel. Disclosing the substitution did not make it the promised measurement | +| P1 | v1's mask fixture was `Bits(Vec)` → a heap alloc inside every timed `intersection`, while `impl EvidenceMask for [u64; N]` **already ships** (`revision.rs:70`) and is the real p64 shape (`[u64; 64]` = 4096 bits) | `malloc` was charged to the arithmetic a tile would accelerate — the exact claim it supported | +| P2 | v1 ran the KILL gate on the 2,449-edge corpus; §3 names **10^5 chains** verbatim | a pre-registered gate was never evaluated as pre-registered | + +**Corrected measurements** (probe moved to `lance-graph-planner/examples/`, +where `causal-edge` is reachable; `[u64; 64]` via the shipped impl; both KILL +scales at one candidate width): + +| quantity | v1 (superseded) | corrected | +|---|---|---| +| chain step | 70.0 ns (f32 substitute + allocating mask) | **34.7 ns** (`NarsTables::revise` + `CausalEdge64::forward`) — 28,818 steps/ms | +| 4096-bit mask | 90.6 ns (allocating) | **61.5 ns** alloc-free `[u64; 64]`; the Vec shape costs 73.4 ns, so allocation was ~16% of it | +| kernel split | "MASK dominates 3.8×" | **MASK dominates 1.77×** | +| KILL @ 10^5 (pre-registered) | never run | scan **13.88 ms** vs decision **0.007 ms** ⇒ does not fire | +| KILL @ 2,449 (real corpus) | 0.906 ms vs 0.008 ms | scan **0.340 ms** vs 0.007 ms ⇒ does not fire | +| crossover | ~25 chains | **~53 chains** | +| oracle arm @ len 16 | 2.74 ms | **1.36 ms** | + +**What survives and what does not.** The *direction* survives — the mask half +still dominates on the corrected, allocation-free numbers, so a 64×64 tile is +aimed at the half that costs. The *margin* does not: 1.77× is under half of +what v1 claimed, so the ALU case is materially weaker than the first pass +said, and the BUY threshold stands at >10× this corpus in one budget +(≈288,000 steps/ms). W5 stays live on cost at BOTH scales. + +**The lesson (recorded because it repeated inside one probe):** every one of +the three defects was a *fixture* defect, not a code defect — a substitute +kernel, an allocating container, a moved goalpost. A measurement's fixture is +part of its claim. A fourth instance was caught in the same pass without +review help: `dense_mask(rng, 1)` sets **no** bits (`x % 1 == 0` always), so +the frontier decision was scored against an EMPTY live set; `all_ones()` is +now its own constructor and `dense_mask` asserts `one_in >= 2`. diff --git a/crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs b/crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs deleted file mode 100644 index c74c52446..000000000 --- a/crates/lance-graph-contract/examples/dcr_w0_replay_budget.rs +++ /dev/null @@ -1,370 +0,0 @@ -//! **D-DCR-0 (W0) — the replay budget, measured.** -//! -//! `.claude/plans/dismech-causal-replay-v1.md` §3 W0. Three numbers the -//! deferred p64 64x64 ALU wave states its BUY threshold in, plus the -//! pre-registered KILL check for W5 (frontier scheduling). -//! -//! ```text -//! cargo run -p lance-graph-contract --example dcr_w0_replay_budget --release -//! ``` -//! -//! # What is measured, and what it is measured ON -//! -//! **The corpus scale is NOT invented here.** `dismech-causality-v3-v1.md` -//! §3a/§11 already measured the real supervision corpus three independent -//! ways: **2,449** oracle edges (`INDIRECT_KNOWN_INTERMEDIATES` that actually -//! name a mediator) over **534** diseases, a **4,076**-row restraint arm -//! (4,150 minus the 74 that contradict their own label) and **361** `UNKNOWN` -//! rows kept as a third arm. This probe therefore reports its per-scale -//! numbers against those magnitudes rather than against a round number -//! somebody liked. -//! -//! 1. **Step throughput** — one replay step is a `NarsTruth::revision` -//! (evidence fusion) plus the `EvidenceMask` intersect that keeps the -//! candidate set live. Both are contract-level and already shipped; this -//! times the composition, never a new kernel. -//! 2. **Branching shrink** — how much a single evidence item reduces a -//! candidate set, at 10^3 / 10^4 / 10^5 chains. This is the Mengenlehre -//! half: `intersection` for support, `difference` for refute. -//! 3. **KILL check (pre-registered, two-sided)** — if a FULL SCAN of the real -//! corpus costs less than one frontier decision, the scheduler is -//! decoration at DisMech scale and W5 is descoped *for this corpus*, with -//! the crossover size reported so a bigger corpus can re-open it. -//! -//! # Non-carriers -//! -//! `Bits` below is an EXAMPLE-LOCAL fixture implementing the existing -//! [`EvidenceMask`] trait for sets wider than the shipped `u64` impl. The -//! trait exists to be implemented; a probe fixture is not a new carrier and -//! nothing outside this file may use it (`F-RLR-2` is about the production -//! path). The step kernel is `NarsTruth` + `EvidenceMask` — `CausalEdge64`'s -//! packed form lives planner-side (`causal-edge`), one dependency layer out -//! from this zero-dep crate, and is deliberately NOT pulled in to time it. - -use std::time::Instant; - -use lance_graph_contract::exploration::NarsTruth; -use lance_graph_contract::revision::EvidenceMask; - -/// Example-local wide bitset (see the module doc's non-carrier note). -#[derive(Clone, PartialEq, Eq, Debug)] -struct Bits(Vec); - -impl Bits { - fn with_capacity(n: usize) -> Self { - Bits(vec![0u64; n.div_ceil(64)]) - } - fn set(&mut self, i: usize) { - self.0[i / 64] |= 1u64 << (i % 64); - } - fn count(&self) -> usize { - self.0.iter().map(|w| w.count_ones() as usize).sum() - } -} - -impl EvidenceMask for Bits { - fn empty() -> Self { - Bits(Vec::new()) - } - fn is_empty(&self) -> bool { - self.0.iter().all(|w| *w == 0) - } - fn union(&self, other: &Self) -> Self { - Bits(zip_words(&self.0, &other.0, |a, b| a | b)) - } - fn intersection(&self, other: &Self) -> Self { - Bits(zip_words(&self.0, &other.0, |a, b| a & b)) - } - fn difference(&self, other: &Self) -> Self { - Bits(zip_words(&self.0, &other.0, |a, b| a & !b)) - } - fn is_subset_of(&self, other: &Self) -> bool { - self.0 - .iter() - .enumerate() - .all(|(i, w)| w & !other.0.get(i).copied().unwrap_or(0) == 0) - } -} - -fn zip_words(a: &[u64], b: &[u64], f: impl Fn(u64, u64) -> u64) -> Vec { - let n = a.len().max(b.len()); - (0..n) - .map(|i| { - f( - a.get(i).copied().unwrap_or(0), - b.get(i).copied().unwrap_or(0), - ) - }) - .collect() -} - -/// Deterministic PRNG — a probe must be replayable (that is this plan's whole -/// keystone), so no `rand`, no clock seeding. -struct Lcg(u64); -impl Lcg { - fn next(&mut self) -> u64 { - self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1); - self.0 - } - fn below(&mut self, n: usize) -> usize { - (self.next() >> 33) as usize % n - } -} - -/// The real supervision-corpus magnitudes, measured in -/// `dismech-causality-v3-v1.md` §3a/§11 — never re-derived here. -const ORACLE_EDGES: usize = 2_449; -const RESTRAINT_ROWS: usize = 4_076; -const UNKNOWN_ROWS: usize = 361; -const ORACLE_DISEASES: usize = 534; - -/// One replay step: fuse this step's evidence into the running truth, and -/// keep the candidate set live. Both halves are shipped contract surface. -#[inline] -fn replay_step( - running: NarsTruth, - step_evidence: NarsTruth, - live: &Bits, - support: &Bits, -) -> (NarsTruth, Bits) { - (running.revision(&step_evidence), live.intersection(support)) -} - -fn measure_step_throughput(chain_len: usize, chains: usize, set_bits: usize) -> (f64, f64) { - let mut live = Bits::with_capacity(set_bits); - for i in 0..set_bits { - live.set(i); - } - let mut rng = Lcg(0x051E_D270_B5A1_11E5); - // Pre-build per-step support masks so the timed loop measures the STEP, - // not fixture construction. - // 2/3 density: evidence that halves the candidate set at EVERY step is - // unrealistically strong and collapses the set within ~12 steps, which - // would put a clone-reset inside the timed loop and measure the fixture. - let supports: Vec = (0..chain_len) - .map(|_| { - let mut m = Bits::with_capacity(set_bits); - for i in 0..set_bits { - if rng.below(3) != 0 { - m.set(i); - } - } - m - }) - .collect(); - let ev = NarsTruth::new(0.8, 0.6); - - let t0 = Instant::now(); - let mut sink = 0usize; - for _ in 0..chains { - let mut running = NarsTruth::prior(); - let mut cur = live.clone(); - for s in &supports { - let (r, c) = replay_step(running, ev, &cur, s); - running = r; - cur = c; - } - sink += cur.count() + (running.expectation() > 0.0) as usize; - } - let dt = t0.elapsed().as_secs_f64(); - std::hint::black_box(sink); - let steps = (chain_len * chains) as f64; - (steps / dt / 1000.0, dt * 1000.0) // (steps per ms, total ms) -} - -/// The two halves of a step, timed apart. This is the question the deferred -/// 64x64 ALU wave actually asks: a tile is 4096 bits = one node's budget, so -/// it accelerates the MASK half. If revision dominates, the tile is aimed at -/// the wrong half and the BUY is not merely early, it is misdirected. -fn measure_kernel_split(set_bits: usize, iters: usize) -> (f64, f64) { - let mut rng = Lcg(0x0000_DDBA_11C0_FFEE); - let mut a = Bits::with_capacity(set_bits); - let mut b = Bits::with_capacity(set_bits); - for i in 0..set_bits { - if rng.below(3) != 0 { - a.set(i); - } - if rng.below(3) != 0 { - b.set(i); - } - } - let (x, y) = (NarsTruth::new(0.7, 0.5), NarsTruth::new(0.4, 0.3)); - - let t0 = Instant::now(); - let mut acc = NarsTruth::prior(); - for _ in 0..iters { - acc = std::hint::black_box(&acc).revision(std::hint::black_box(&x)); - acc = std::hint::black_box(&acc).revision(std::hint::black_box(&y)); - } - std::hint::black_box(acc.expectation()); - let revision_per_ms = (iters * 2) as f64 / (t0.elapsed().as_secs_f64() * 1000.0); - - let t1 = Instant::now(); - let mut sink = 0usize; - for _ in 0..iters { - sink += std::hint::black_box(&a) - .intersection(std::hint::black_box(&b)) - .count(); - } - std::hint::black_box(sink); - let mask_per_ms = iters as f64 / (t1.elapsed().as_secs_f64() * 1000.0); - - (revision_per_ms, mask_per_ms) -} - -/// One frontier decision, priced honestly: score EVERY candidate observation -/// by the candidate-set split it would produce. That is the W4/W5 shape -/// (expected information gain), so the KILL check compares like with like. -fn measure_frontier_decision(candidates: usize, observations: usize) -> (f64, usize) { - let mut live = Bits::with_capacity(candidates); - for i in 0..candidates { - live.set(i); - } - let mut rng = Lcg(0x0D15_EC40_11EA_DBEE); - let obs: Vec = (0..observations) - .map(|_| { - let mut m = Bits::with_capacity(candidates); - for i in 0..candidates { - if rng.below(4) != 0 { - m.set(i); - } - } - m - }) - .collect(); - - let t0 = Instant::now(); - let mut best = (usize::MAX, 0usize); - let total = live.count(); - for (i, o) in obs.iter().enumerate() { - let kept = live.intersection(o).count(); - let dropped = live.difference(o).count(); - // Balanced split = most informative; |kept - dropped| is its proxy. - let imbalance = kept.abs_diff(dropped); - if imbalance < best.0 { - best = (imbalance, i); - } - std::hint::black_box((kept, dropped, total)); - } - (t0.elapsed().as_secs_f64() * 1000.0, best.1) -} - -/// A full scan: replay every chain once, no frontier at all. -fn measure_full_scan(chains: usize, chain_len: usize, set_bits: usize) -> f64 { - let (_, ms) = measure_step_throughput(chain_len, chains, set_bits); - ms -} - -fn main() { - println!("D-DCR-0 (W0) — replay budget"); - println!("corpus (measured in dismech-causality-v3-v1 §3a/§11, not re-derived here):"); - println!(" oracle arm {ORACLE_EDGES} edges over {ORACLE_DISEASES} diseases"); - println!(" restraint arm {RESTRAINT_ROWS} rows"); - println!(" unknown arm {UNKNOWN_ROWS} rows"); - - // ── 1. Step throughput ─────────────────────────────────────────────── - println!("\n1. STEP THROUGHPUT (NarsTruth::revision + EvidenceMask::intersection)"); - println!( - " {:>10} {:>10} {:>14} {:>12}", - "cand.set", "chain len", "steps/ms", "ns/step" - ); - let mut steps_per_ms_at_2k = 0.0f64; - for &set_bits in &[64usize, 1_024, 4_096] { - for &chain_len in &[4usize, 16] { - let chains = 2_000; - let (spms, _) = measure_step_throughput(chain_len, chains, set_bits); - if set_bits == 4_096 && chain_len == 16 { - steps_per_ms_at_2k = spms; - } - println!( - " {set_bits:>10} {chain_len:>10} {spms:>14.1} {:>12.1}", - 1_000_000.0 / spms - ); - } - } - - // ── 2. Branching shrink ────────────────────────────────────────────── - println!("\n2. BRANCHING SHRINK (one evidence item, support ∩ / refute ∖)"); - println!(" NOTE: densities are FIXTURE-SET (2/3 support, 1/10 refute), so this"); - println!(" measures the MECHANISM's cost and scaling — never the corpus's real"); - println!(" discriminative power. That needs the frozen oracle/restraint TSVs"); - println!(" (D-CV3-0..2, consumer-side)."); - println!( - " {:>10} {:>12} {:>12} {:>10}", - "chains", "after ∩", "after ∖", "factor" - ); - for &n in &[1_000usize, 10_000, 100_000] { - let mut live = Bits::with_capacity(n); - for i in 0..n { - live.set(i); - } - let mut rng = Lcg(0xA5A5_1234_DEAD_C0DE); - let mut support = Bits::with_capacity(n); - let mut refute = Bits::with_capacity(n); - for i in 0..n { - if rng.below(3) != 0 { - support.set(i); - } - if rng.below(10) == 0 { - refute.set(i); - } - } - let after_and = live.intersection(&support); - let after_diff = after_and.difference(&refute); - println!( - " {n:>10} {:>12} {:>12} {:>10.2}x", - after_and.count(), - after_diff.count(), - n as f64 / after_diff.count().max(1) as f64 - ); - } - - // ── 3. KILL check ──────────────────────────────────────────────────── - println!("\n3. KILL CHECK (pre-registered): is a full scan cheaper than one decision?"); - let scan_ms = measure_full_scan(ORACLE_EDGES, 4, 4_096); - let (dec_ms, _pick) = measure_frontier_decision(ORACLE_EDGES, 64); - println!(" full scan of {ORACLE_EDGES} chains (len 4): {scan_ms:.3} ms"); - println!(" one frontier decision over 64 observations: {dec_ms:.3} ms"); - let verdict = if scan_ms <= dec_ms { - "KILL FIRES — scheduling is decoration at DisMech scale; W5 descoped for THIS corpus" - } else { - "KILL does not fire — a decision is cheaper than a scan; W5 stays live" - }; - println!(" => {verdict}"); - // Crossover: how many chains before a scan costs a decision? - let per_chain_ms = scan_ms / ORACLE_EDGES as f64; - println!( - " crossover: a scan costs one decision at ~{:.0} chains ({:.1}x the oracle arm)", - dec_ms / per_chain_ms, - (dec_ms / per_chain_ms) / ORACLE_EDGES as f64 - ); - - // ── 4. Kernel split — which half a 64x64 tile would even touch ─────── - println!("\n4. KERNEL SPLIT (the ALU wave's actual question)"); - let (rev_pm, mask_pm) = measure_kernel_split(4_096, 200_000); - println!( - " NarsTruth::revision alone : {rev_pm:>10.0} ops/ms ({:>6.1} ns)", - 1_000_000.0 / rev_pm - ); - println!( - " 4096-bit intersect+count : {mask_pm:>10.0} ops/ms ({:>6.1} ns)", - 1_000_000.0 / mask_pm - ); - let (dominant, factor) = if mask_pm < rev_pm { - ("MASK", rev_pm / mask_pm) - } else { - ("REVISION", mask_pm / rev_pm) - }; - println!(" => {dominant} dominates by {factor:.1}x — a 64x64 tile accelerates the MASK half"); - - // ── 5. ALU BUY threshold, stated in the measured numbers ───────────── - println!("\n5. ALU BUY THRESHOLD (deferred p64 64x64 wave)"); - println!(" measured step rate (4096-wide set, len 16): {steps_per_ms_at_2k:.0} steps/ms"); - let corpus_ms = ORACLE_EDGES as f64 * 16.0 / steps_per_ms_at_2k; - println!(" => whole oracle arm at chain len 16: {corpus_ms:.2} ms"); - println!( - " BUY only when a workload needs > {:.0} steps/ms sustained, i.e. > {:.0}x this corpus\n in one budget — until then the scalar path is not the bottleneck.", - steps_per_ms_at_2k * 10.0, - 10.0 - ); -} diff --git a/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs b/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs new file mode 100644 index 000000000..d355faabb --- /dev/null +++ b/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs @@ -0,0 +1,312 @@ +//! **D-DCR-0 (W0) — the replay budget, measured.** `dismech-causal-replay-v1` §3. +//! +//! ```text +//! cargo run -p lance-graph-planner --example dcr_w0_replay_budget --release +//! ``` +//! +//! # ⊘ CORRECTED after codex review on #1118 — three findings, all valid +//! +//! The first version of this probe lived in `lance-graph-contract` and was +//! wrong in three ways that all pushed the same direction (they inflated the +//! mask half and measured a kernel the plan never named): +//! +//! 1. **It benchmarked a substitute kernel.** §3 W0 defines an eval as +//! *"a `NarsTables` lookup + `CausalEdge64` revision"*. The v1 probe timed +//! `NarsTruth::revision` (f32, contract-side) instead, because the real +//! kernel lives in `causal-edge`, one dependency layer outside the zero-dep +//! contract crate. Disclosing the substitution in a doc-comment did not make +//! the headline number the promised measurement. **Fix: the probe moved +//! HERE** (planner path-deps `causal-edge`) and now times +//! [`NarsTables::revise`] + [`CausalEdge64::forward`]. +//! 2. **It put a heap allocation inside the timed mask loop.** The v1 fixture +//! was `Bits(Vec)`, so every `intersection` ran `…collect()`. The +//! workspace already ships `impl EvidenceMask for [u64; N]` +//! (`revision.rs:70`), which is allocation-free and is the shape the p64 +//! surface actually has (`[u64; 64]` = 4096 bits = one node's budget). The +//! v1 "MASK dominates by 3.8x" therefore charged `malloc` to the arithmetic +//! a tile would accelerate — the exact claim it was used to support. **Fix: +//! both are timed below and the allocation share is REPORTED**, so the +//! correction is visible rather than quietly swapped. +//! 3. **It moved the pre-registered goalpost.** §3's KILL gate names 10^5 +//! chains verbatim; v1 substituted the 2,449-edge oracle arm and recorded +//! the verdict from that. A pre-registered gate must be evaluated as +//! pre-registered. **Fix: BOTH scales are run**, at one candidate width. +//! +//! Recorded rather than silently rewritten (append-only convention): the v1 +//! numbers are not deleted from the board, they are superseded there with this +//! reason. +//! +//! # Corpus scale is READ, never re-derived +//! +//! `dismech-causality-v3-v1.md` §3a/§11 measured the supervision corpus three +//! independent ways: **2,449** oracle edges over **534** diseases, **4,076** +//! restraint rows, **361** `UNKNOWN` rows. + +use std::time::Instant; + +use causal_edge::edge::InferenceType; +use causal_edge::tables::{unpack_c, unpack_f, NarsTables}; +use causal_edge::{CausalEdge64, CausalMask, PlasticityState}; +use lance_graph_contract::revision::EvidenceMask; + +/// Measured in `dismech-causality-v3-v1.md` §3a/§11 — never re-derived here. +const ORACLE_EDGES: usize = 2_449; +const ORACLE_DISEASES: usize = 534; +const RESTRAINT_ROWS: usize = 4_076; +const UNKNOWN_ROWS: usize = 361; + +/// The pre-registered KILL scale, verbatim from the plan's §3 W0 gate. +const PREREGISTERED_SCAN_CHAINS: usize = 100_000; + +/// 4096 bits = one node's value budget = the p64 tile shape (`[u64; 64]`). +const MASK_WORDS: usize = 64; +type Mask = [u64; MASK_WORDS]; + +/// Deterministic PRNG — a probe for a *replay* plan must itself be replayable, +/// so no `rand`, no clock seeding. +struct Lcg(u64); +impl Lcg { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_mul(6364136223846793005).wrapping_add(1); + self.0 + } + fn below(&mut self, n: usize) -> usize { + (self.next() >> 33) as usize % n + } +} + +/// A mask with roughly `(one_in - 1) / one_in` of its bits set. +/// `one_in == 1` would set NOTHING (`x % 1 == 0` always), so an all-ones set +/// has its own constructor rather than a degenerate parameter value — the +/// first draft of this probe hit exactly that trap and measured a frontier +/// decision against an EMPTY live set. +fn dense_mask(rng: &mut Lcg, one_in: usize) -> Mask { + assert!(one_in >= 2, "use all_ones() for a full set"); + let mut m = [0u64; MASK_WORDS]; + for w in m.iter_mut() { + let mut word = 0u64; + for b in 0..64 { + if rng.below(one_in) != 0 { + word |= 1u64 << b; + } + } + *w = word; + } + m +} + +fn all_ones() -> Mask { + [u64::MAX; MASK_WORDS] +} + +fn popcount(m: &Mask) -> u32 { + m.iter().map(|w| w.count_ones()).sum() +} + +/// **The promised chain step** (`dismech-causal-replay-v1` §3 W0, verbatim): +/// a `NarsTables` lookup plus a `CausalEdge64` revision. `forward` IS that +/// revision — palette compose + NARS truth propagation in one packed edge. +#[inline] +fn promised_step( + running: CausalEdge64, + weight: CausalEdge64, + tables: &NarsTables, + cs: &[u8; 256 * 256], + cp: &[u8; 256 * 256], + co: &[u8; 256 * 256], +) -> CausalEdge64 { + // 1. the table lookup half + let revised = tables.revise( + running.frequency_u8(), + running.confidence_u8(), + weight.frequency_u8(), + weight.confidence_u8(), + ); + // 2. the packed-edge half + let out = running.forward(weight, cs, cp, co); + // fold the lookup back in so neither half can be optimised away + CausalEdge64(out.0 ^ ((unpack_f(revised) as u64) << 32) ^ (unpack_c(revised) as u64)) +} + +fn build_compose() -> Box<[[u8; 256 * 256]; 3]> { + let mut t = Box::new([[0u8; 256 * 256]; 3]); + for (k, tab) in t.iter_mut().enumerate() { + for i in 0..256usize { + for j in 0..256usize { + tab[i * 256 + j] = ((i + j + k) % 256) as u8; + } + } + } + t +} + +fn edge(rng: &mut Lcg, infer: InferenceType) -> CausalEdge64 { + CausalEdge64::pack( + rng.below(256) as u8, + rng.below(256) as u8, + rng.below(256) as u8, + (128 + rng.below(128)) as u8, + (128 + rng.below(100)) as u8, + CausalMask::PO, + 0b101, + infer, + PlasticityState::S_HOT, + 0, + ) +} + +/// Time the promised kernel alone. Returns (steps/ms, ns/step). +fn measure_promised_step(iters: usize) -> (f64, f64) { + let tables = NarsTables::build(1); + let c = build_compose(); + let mut rng = Lcg(0x051E_D270_B5A1_11E5); + let weights: Vec = (0..64) + .map(|_| edge(&mut rng, InferenceType::Deduction)) + .collect(); + let mut running = edge(&mut rng, InferenceType::Revision); + + let t0 = Instant::now(); + for i in 0..iters { + let w = weights[i & 63]; + running = promised_step(running, w, &tables, &c[0], &c[1], &c[2]); + } + let dt = t0.elapsed().as_secs_f64(); + std::hint::black_box(running.0); + (iters as f64 / dt / 1000.0, dt * 1e9 / iters as f64) +} + +/// Time the mask half BOTH ways: the shipped allocation-free `[u64; 64]`, and +/// the v1 `Vec` shape, so the allocation share the codex review named is +/// reported rather than assumed. +fn measure_mask_both(iters: usize) -> (f64, f64, f64, f64) { + let mut rng = Lcg(0x0000_DDBA_11C0_FFEE); + let a = dense_mask(&mut rng, 3); + let b = dense_mask(&mut rng, 3); + + let t0 = Instant::now(); + let mut sink = 0u32; + for _ in 0..iters { + let m = std::hint::black_box(&a).intersection(std::hint::black_box(&b)); + sink += popcount(&m); + } + let fixed_ns = t0.elapsed().as_secs_f64() * 1e9 / iters as f64; + std::hint::black_box(sink); + + let (va, vb) = (a.to_vec(), b.to_vec()); + let t1 = Instant::now(); + let mut sink2 = 0u32; + for _ in 0..iters { + // the v1 shape: a fresh Vec per intersection + let m: Vec = std::hint::black_box(&va) + .iter() + .zip(std::hint::black_box(&vb).iter()) + .map(|(x, y)| x & y) + .collect(); + sink2 += m.iter().map(|w| w.count_ones()).sum::(); + } + let vec_ns = t1.elapsed().as_secs_f64() * 1e9 / iters as f64; + std::hint::black_box(sink2); + + (1e6 / fixed_ns, fixed_ns, 1e6 / vec_ns, vec_ns) +} + +/// A full scan: replay every chain once, promised kernel, no frontier. +fn measure_full_scan(chains: usize, chain_len: usize, step_ns: f64) -> f64 { + (chains * chain_len) as f64 * step_ns / 1e6 // ms +} + +/// One frontier decision at a FIXED candidate width: score every candidate +/// observation by the split it would produce (the W4/W5 shape). +fn measure_frontier_decision(observations: usize) -> f64 { + let mut rng = Lcg(0x0D15_EC40_11EA_DBEE); + let live = all_ones(); + let obs: Vec = (0..observations).map(|_| dense_mask(&mut rng, 4)).collect(); + + let t0 = Instant::now(); + let mut best = (u32::MAX, 0usize); + for (i, o) in obs.iter().enumerate() { + let kept = popcount(&live.intersection(o)); + let dropped = popcount(&live.difference(o)); + let imbalance = kept.abs_diff(dropped); + if imbalance < best.0 { + best = (imbalance, i); + } + } + std::hint::black_box(best); + t0.elapsed().as_secs_f64() * 1000.0 +} + +fn main() { + println!("D-DCR-0 (W0) — replay budget [CORRECTED after codex review on #1118]"); + println!("corpus (read from dismech-causality-v3-v1 §3a/§11, not re-derived):"); + println!(" oracle {ORACLE_EDGES} edges / {ORACLE_DISEASES} diseases · restraint {RESTRAINT_ROWS} · unknown {UNKNOWN_ROWS}"); + + // ── 1. the PROMISED kernel ─────────────────────────────────────────── + println!("\n1. PROMISED CHAIN STEP — NarsTables::revise + CausalEdge64::forward"); + let (steps_per_ms, step_ns) = measure_promised_step(2_000_000); + println!(" {steps_per_ms:>10.0} steps/ms ({step_ns:.1} ns/step)"); + println!(" (v1 timed NarsTruth::revision f32 instead — a substitute the plan never named)"); + + // ── 2. the mask half, both shapes ──────────────────────────────────── + println!("\n2. MASK HALF at 4096 bits — shipped [u64; 64] vs the v1 Vec shape"); + let (fixed_pm, fixed_ns, vec_pm, vec_ns) = measure_mask_both(1_000_000); + println!( + " [u64; 64] (EvidenceMask, alloc-free): {fixed_pm:>9.0} ops/ms ({fixed_ns:>6.1} ns)" + ); + println!(" Vec (v1 shape, allocates) : {vec_pm:>9.0} ops/ms ({vec_ns:>6.1} ns)"); + println!( + " => allocation was {:.1}x of the v1 mask cost — charged to arithmetic a tile would accelerate", + vec_ns / fixed_ns + ); + + // ── 3. kernel split, re-derived from the corrected numbers ─────────── + println!("\n3. KERNEL SPLIT (the ALU wave's actual question), corrected"); + let (dominant, factor) = if fixed_ns > step_ns { + ("MASK", fixed_ns / step_ns) + } else { + ("STEP", step_ns / fixed_ns) + }; + println!(" promised step {step_ns:.1} ns vs alloc-free 4096-bit mask {fixed_ns:.1} ns"); + println!(" => {dominant} dominates by {factor:.2}x"); + println!(" v1 reported MASK by 3.8x on the allocating shape — SUPERSEDED by this line."); + + // ── 4. KILL check at BOTH scales, one candidate width ──────────────── + println!( + "\n4. KILL CHECK — pre-registered 10^5 AND the real corpus, at {} candidate bits", + MASK_WORDS * 64 + ); + let dec_ms = measure_frontier_decision(64); + for (label, chains) in [ + ("pre-registered", PREREGISTERED_SCAN_CHAINS), + ("real oracle arm", ORACLE_EDGES), + ] { + let scan_ms = measure_full_scan(chains, 4, step_ns); + let fires = scan_ms <= dec_ms; + println!( + " {label:>16}: scan {chains:>6} chains = {scan_ms:>9.3} ms vs decision {dec_ms:.3} ms => KILL {}", + if fires { "FIRES" } else { "does not fire" } + ); + } + let per_chain_ms = measure_full_scan(1, 4, step_ns); + println!( + " crossover: a scan costs one decision at ~{:.0} chains", + dec_ms / per_chain_ms + ); + + // ── 5. ALU BUY threshold from the corrected numbers ────────────────── + println!("\n5. ALU BUY THRESHOLD (deferred p64 64x64 wave)"); + let corpus_ms = measure_full_scan(ORACLE_EDGES, 16, step_ns); + println!( + " whole oracle arm at chain len 16: {corpus_ms:.2} ms ({steps_per_ms:.0} steps/ms)" + ); + println!( + " BUY only when a workload sustains > {:.0} steps/ms (10x this corpus in one budget).", + steps_per_ms * 10.0 + ); + if dominant == "STEP" { + println!(" ⚠ DIRECTION: the STEP half now dominates, so a mask-only tile is NOT the"); + println!(" first lever — the packed-edge/table path is. Re-aim before any BUY."); + } else { + println!(" DIRECTION: the mask half dominates even allocation-free — a 64x64 tile is aimed right."); + } +} From 9e3104bca4bfe8e5d3429a78fef9abe1f9dfb2d3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 19:05:54 +0000 Subject: [PATCH 3/3] =?UTF-8?q?W0=20=C2=A73d:=20does=20the=20replay=20engi?= =?UTF-8?q?ne=20need=20ndarray's=20masking=3F=20measured=20=E2=80=94=20dea?= =?UTF-8?q?d=20heat,=20and=20the=20reduction=20is=20why?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator question, answered with the W0 harness instead of architecture. Three objects are called 'masking' here and they are not the same kind: - ogar_r2il::CallMask is [u64; 3] (192 bits, lazy word-tests, no alloc) => needs nothing; a slice-API call would cost more than the work. - the replay candidate set is [u64; 64] (4096 bits) => the real candidate, and W0 measures it dominating the promised step ~1.8x. - ndarray::hpc::jitson is a JSON-config -> Cranelift native SCAN kernel compiler, not a masking library. Relevant to W5's frontier decision, never to the mask half. Measured at 4096 bits: scalar [u64;64] 65.2 ns vs ndarray simd_int_ops::mask_and (U64x8) 60.4 ns = 1.08x, a DEAD HEAT. The decomposition says why: SIMD and 11.1 ns + scalar popcount 56.7 ns, so the POPCOUNT is 5.1x the AND. Borrowing the AND changes nothing because the AND was never the cost. The primitive that would pay does not exist: a fused mask_and_popcount(&[u64], &[u64]) -> u32 keeping the AND in registers and reducing with VPOPCNTDQ. ndarray has popcnt on its AVX-512 typed wrapper but no fused slice API, and the workspace SIMD invariant means it must be added IN ndarray, never hand-rolled here. Surfaced, not filed — upstream asks are the operator's call. Consequence: the deferred p64 64x64 wave's TARGET moves (fuse the reduction, don't accelerate the op); its BUY threshold is unchanged. Probe extended with the ndarray arm + decomposition; a >1.15x/<0.87x band replaces the earlier faster/slower label, which called 1.00x 'FASTER'. Gates: planner 370/370, example clippy -D warnings clean, fmt clean. --- .claude/board/STATUS_BOARD.md | 1 + .claude/plans/dismech-causal-replay-v1.md | 43 ++++++++++ .../examples/dcr_w0_replay_budget.rs | 86 +++++++++++++++++++ 3 files changed, 130 insertions(+) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 05ff879cf..74e1ea9a8 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -3,6 +3,7 @@ | D-id | deliverable | status | |---|---|---| | D-DCR-0 | W0 measurements: chain-step evals/ms, EvidenceMask branching shrink, palette round-trip; ALU BUY threshold stated | **Shipped (CORRECTED #1118)** — promised kernel (NarsTables::revise + CausalEdge64::forward) **34.7 ns/step**; alloc-free `[u64;64]` mask 61.5 ns ⇒ **MASK dominates 1.77x** (v1's 3.8x superseded: substitute kernel + allocating fixture); KILL does not fire at EITHER the pre-registered 10^5 (13.88 ms) or the real 2,449 arm (0.340 ms), crossover ~53 chains; BUY at >10x the **1.36 ms** oracle arm. Probe `lance-graph-planner/examples/dcr_w0_replay_budget` | +| D-DCR-0b | "borrow masking from ndarray?" measured (§3d): `mask_and` is a DEAD HEAT (1.08x) because the AND is 11.1 ns of a 65.2 ns half — the scalar POPCOUNT is 5.1x it. R2IL's `CallMask` (`[u64;3]`) needs nothing; jitson is a kernel compiler, not masking. The primitive that would pay (fused `mask_and_popcount`) does not exist and belongs IN ndarray | **Shipped** (measured direction; no wave scheduled) | | D-DCR-0a | prior-art reconciliation: `contract::dismech_evidence` + `dismech-causality-v3-v1` §11 arms (2,449 / 4,076 / 361) are W1-W3's falsifier; plan §3a | **Shipped** (E-W0-MEASURED-THE-MASK-HALF-DOMINATES-...-1) | | D-DCR-1 | replay core: loco calls under the dismech vocabulary -> CausalEdge64/NarsTruth steps -> temporal.rs trace; determinism + perturbation falsifiers | Queued (after D-DCR-0) | | D-DCR-2 | Mengenlehre candidate evaluation via `contract::revision::EvidenceMask` (support ∩ / refute ∖, skip-words as ordinals) | Queued | diff --git a/.claude/plans/dismech-causal-replay-v1.md b/.claude/plans/dismech-causal-replay-v1.md index 408b20b11..8b4d552ef 100644 --- a/.claude/plans/dismech-causal-replay-v1.md +++ b/.claude/plans/dismech-causal-replay-v1.md @@ -306,3 +306,46 @@ part of its claim. A fourth instance was caught in the same pass without review help: `dense_mask(rng, 1)` sets **no** bits (`x % 1 == 0` always), so the frontier decision was scored against an EMPTY live set; `all_ones()` is now its own constructor and `dense_mask` asserts `one_in >= 2`. + +## §3d — "Does the replay engine need to borrow masking from ndarray?" — MEASURED + +Operator question, 2026-08-31. Answered with the W0 harness rather than from +architecture. Three different objects are called "masking" in this stack and +**they are not the same kind of thing**: + +| object | width | shape | verdict | +|---|---|---|---| +| `ogar_r2il::CallMask` | `[u64; 3]` = 192 bits (≤180 calls/node) | lazy word-tests, no alloc, no `Vec` built | **needs nothing.** At three words, a slice-API call costs more than the work. It is already allocation-free and lazy — the properties SIMD would be bought for | +| replay candidate set | `[u64; 64]` = 4096 bits | the shipped `impl EvidenceMask for [u64; N]` | the real candidate — W0 measures it dominating the promised step by ~1.8× | +| `ndarray::hpc::jitson` | n/a | JSON config → Cranelift **native scan kernels** (`ScanParams`/`RecipeIR`/`ScanKernel`) | **not a masking library at all.** A kernel *compiler*. Its relevance is to W5's frontier decision (a scan-shaped workload), never to the mask half | + +**Measured, at 4096 bits (probe §2):** + +``` +[u64; 64] scalar (EvidenceMask) 65.2 ns +ndarray simd_int_ops::mask_and (U64x8) 60.4 ns => 1.08x — DEAD HEAT +decomposition: SIMD and 11.1 ns + scalar popcount 56.7 ns + => the POPCOUNT half is 5.1x the AND +``` + +**So the answer is neither yes nor no.** ndarray's SIMD mask kernel is fast — +11.1 ns for 64 words is the `U64x8` path working exactly as advertised. It +changes nothing end-to-end because **the AND was never the cost**: the +reduction is, at 84% of the mask half. `mask_and` writes 512 bytes to `dst` +and the popcount then re-reads them scalar. + +**The primitive that would pay does not exist yet:** a fused +`mask_and_popcount(&[u64], &[u64]) -> u32` that keeps the AND result in +registers and reduces with `VPOPCNTDQ`. ndarray HAS `popcnt` on its AVX-512 +typed wrapper (`simd_avx512.rs:2987`) but exposes no fused slice-level API, and +the workspace's SIMD invariant ("all SIMD from `ndarray::simd`") means that +primitive must be **added in ndarray**, never hand-rolled here. + +**Consequence for the deferred p64 64×64 wave:** its target moves. A tile that +accelerates the *bitwise op* is aimed at 11.1 ns of a 65.2 ns half; a tile that +fuses op+reduction is aimed at all of it. The BUY threshold is unchanged +(>10× this corpus in one budget) — what changed is what to buy. + +**Not scheduled, deliberately.** This is a measured direction, not a wave. +The cross-repo ask (an ndarray fused mask+popcount primitive) is the +operator's call to make, per commitment on upstream asks. diff --git a/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs b/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs index d355faabb..3eb8cda00 100644 --- a/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs +++ b/crates/lance-graph-planner/examples/dcr_w0_replay_budget.rs @@ -210,6 +210,66 @@ fn measure_mask_both(iters: usize) -> (f64, f64, f64, f64) { (1e6 / fixed_ns, fixed_ns, 1e6 / vec_ns, vec_ns) } +/// The SAME 4096-bit intersection through **ndarray's shipped SIMD mask +/// kernel** (`simd_int_ops::mask_and`, `U64x8` lanes), which is the workspace's +/// only sanctioned home for SIMD. This answers "does the replay engine need to +/// borrow masking from ndarray?" with a number instead of an opinion — and it +/// is the honest first increment of the deferred p64 64x64 wave, because +/// 64 words x 64 bits IS the tile, already written and already dispatched. +fn measure_mask_ndarray(iters: usize) -> f64 { + let mut rng = Lcg(0x0000_DDBA_11C0_FFEE); // same seed as the other two + let a = dense_mask(&mut rng, 3); + let b = dense_mask(&mut rng, 3); + let mut dst = [0u64; MASK_WORDS]; + + let t0 = Instant::now(); + let mut sink = 0u32; + for _ in 0..iters { + ndarray::simd_int_ops::mask_and( + std::hint::black_box(&a), + std::hint::black_box(&b), + &mut dst, + ); + sink += dst.iter().map(|w| w.count_ones()).sum::(); + } + std::hint::black_box(sink); + t0.elapsed().as_secs_f64() * 1e9 / iters as f64 +} + +/// Decompose the mask half: AND alone vs popcount alone. If the SIMD and the +/// scalar AND paths tie, the AND is not what costs — which changes what an +/// accelerator would have to BE. +fn measure_mask_decomposition(iters: usize) -> (f64, f64) { + let mut rng = Lcg(0x0000_DDBA_11C0_FFEE); + let a = dense_mask(&mut rng, 3); + let b = dense_mask(&mut rng, 3); + let mut dst = [0u64; MASK_WORDS]; + + let t0 = Instant::now(); + for _ in 0..iters { + ndarray::simd_int_ops::mask_and( + std::hint::black_box(&a), + std::hint::black_box(&b), + &mut dst, + ); + std::hint::black_box(&dst); + } + let and_ns = t0.elapsed().as_secs_f64() * 1e9 / iters as f64; + + let t1 = Instant::now(); + let mut sink = 0u32; + for _ in 0..iters { + sink += std::hint::black_box(&dst) + .iter() + .map(|w| w.count_ones()) + .sum::(); + } + std::hint::black_box(sink); + let pop_ns = t1.elapsed().as_secs_f64() * 1e9 / iters as f64; + + (and_ns, pop_ns) +} + /// A full scan: replay every chain once, promised kernel, no frontier. fn measure_full_scan(chains: usize, chain_len: usize, step_ns: f64) -> f64 { (chains * chain_len) as f64 * step_ns / 1e6 // ms @@ -259,6 +319,32 @@ fn main() { vec_ns / fixed_ns ); + let nd_ns = measure_mask_ndarray(1_000_000); + println!( + " ndarray simd_int_ops::mask_and (U64x8): {:>9.0} ops/ms ({nd_ns:>6.1} ns)", + 1e6 / nd_ns + ); + let ratio = fixed_ns / nd_ns; + let verdict = if ratio > 1.15 { + "FASTER — borrowing pays" + } else if ratio < 0.87 { + "SLOWER — borrowing costs" + } else { + "DEAD HEAT — borrowing changes nothing at this width" + }; + println!(" => vs scalar [u64; 64]: {ratio:.2}x — {verdict}"); + let (and_ns, pop_ns) = measure_mask_decomposition(1_000_000); + println!(" decomposition: SIMD and {and_ns:.1} ns + scalar popcount {pop_ns:.1} ns"); + println!( + " => the {} half is {:.1}x the other; an accelerator must target THAT", + if pop_ns > and_ns { "POPCOUNT" } else { "AND" }, + if pop_ns > and_ns { + pop_ns / and_ns + } else { + and_ns / pop_ns + } + ); + // ── 3. kernel split, re-derived from the corrected numbers ─────────── println!("\n3. KERNEL SPLIT (the ALU wave's actual question), corrected"); let (dominant, factor) = if fixed_ns > step_ns {