From 08ccc6a1dbb799f7b359932eb17d0c3d0fc51a09 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 20:52:49 +0000 Subject: [PATCH 1/3] =?UTF-8?q?board:=20PR=20#44=20arc=20entry=20(R1=20?= =?UTF-8?q?=E2=86=92=20minor=2010,=20the=20whole=20arc)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hygiene-only; per the termination clause this generates no further arc obligation. The one substantive note the entry carries: the PR body predates its own last three commits (it still says R2 is not landed), so the arc entry is the current record of what actually merged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DCfrD5y19cvFc4AoyydXYv --- .claude/board/PR_ARC_INVENTORY.md | 62 +++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index dda75c0..8c340f1 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -8,6 +8,68 @@ > anti-pattern the imported board rules name. Backfilled below in one > pass rather than left stale; PR #4 onward gets its entry at merge time. +## PR #44 — R1→minor 10: mask algebra restored, the reduction repatriated, the doctrine pinned, the columnar store landed (merged 2026-08-27, merge — 5 commits, head `bd6f666`) + +- **Added, as one arc** (the PR body predates its own last three commits — + it says "R2 measured, not landed", which commit 5 then landed; this entry + is the current record): + 1. **R1** (`0385269`) — `lgj_hop` selects with `src ∧ class_f ∧ struct_f`, + word-parallel; the walk only EMITS. F2 closed: the structured-edge gate + was an `if` in EVERY prior version incl. #22's — it is the same strided + primitive at `+12`, one call site, zero new kernels. + `facet_bits`/`facet_cache` deleted (a stored projection). Byte-identical + (pinned 10/19/29), and honestly a 19× regression on AoS — the layout + named as the defect, per R11's prior 9.2× pricing. + 2. **§13 de-staled** (`ba377b3`) — abi.md described the pre-R1 + composition; the #39-shaped prose-lag caught by adversarial re-read. + 3. **Minor 9** (`c3ecf37`) — `lgj_rowstore_facet_match_count`: the + facet-match reduction computed where the data is, after THREE Java-side + shapes of it (segment popcount loop; 32 composed counts summed in + Java; a proposed buffer-popcount symbol). Two independent oracles; + both compat directions vs a real minor-8 library. Exposed en route: + root-invoked release builds were silently REFUSED (toolchain floor, + error hidden by tail-piping) — earlier R1 Java runs had loaded a + pre-R1 `.so`; harmless (no observable change) and caught by the + minor-9 gate itself. + 4. **Doctrine** (`824996d`) — the simd.rs isomorphism as root CLAUDE.md's + ENFORCEMENT LAYER E1–E6 (Java=facade/37-fns-0-instructions, + Valhalla+Panama=polyfill, Rust=backends/488-intrinsics; scalar is a + backend BELOW the facade; facade intrinsics only as cfg(test) oracles). + J2 closed: `Layouts` derives the geometry, the facade names it. + CODEX_REVIEW_CHECKLIST §8 added (the "saves a crossing" tell). + 5. **Minor 10** (`bd6f666`) — `lgj_rowstore_open_columnar`: facet-major + over the (row × facet) plane, same 512n bytes/draws/content (pinned, + bytes-differ anti-vacuity). Lane table 33→97 (lo64/hi32 lanes join + classid) so EVERY field is descriptor-served; Java proven LAYOUT-BLIND + (accessors read only via descriptors; `rowOffset` + last facade + geometry constants deleted; disable-run red at row 1 facet 0 — + the first address divergence — AoS green). Register-sweep family + refuses facet-major with new `UNSUPPORTED_LAYOUT` (−18), two-sided. +- **Locked:** HOP EXECUTES AS MASK × CLASSVIEW → MASK, at every layer with + a named gate; a layout is a SCHEMA (constructor + descriptors, never a + resource kind); carving groups ≤ 4 B and 64-alignment of 512/regions/ + blocks pinned as tests (`carving_groups_fit_the_flattening_budget…`) — + the substrate half of R4/R10's Valhalla measurements. +- **Measured, banked on the board:** AoS mask algebra 19× WORSE than the + old sweep (`hop-mask-algebra-vs-columnar.txt` — the finding that forced + minor 10); through the REAL ABI, columnar hop **4.7×/5.9×/3.8×** over AoS + at classid/2-hop/full arms, equivalence asserted before timing + (`columnar-store-abi-bench.txt`). +- **Deferred, named:** the fused single-plane pass (~10× further, per the + lab arm); the register-sweep family on facet-major (an honest −18, not a + gap); Vector API permanently lab (E4). +- **Docs:** abi.md §13 rewrite, §18 new, symbol count 25→26, status −18, + minor 9+10 history; root CLAUDE.md E1–E6. +- **Gates:** Rust 138/139 (both feature configs) · clippy `-D warnings` + + fmt · Java **314 core** (ColumnarStoreTest 10 new) **+ 143 consumer** · + runtime-confirmed `abi 0.10` · OldAbiCompatTest both directions vs REAL + minor-8 AND minor-9 libraries built from prior commits in worktrees + (path deps resolve relative to the worktree — it must sit beside the + sibling repos, not in /tmp). +- **Confidence:** high — every claim above is a pinned test, a banked + measurement, or a disable-run observed red-then-green; the one narrative + caveat is that the PR BODY describes only commit 1's state. + ## PR #42 — lgj-abi: the REAL OGAR ClassView provider, bound behind a feature (merged 2026-08-27, `507cc93`) - **Added:** `ogar-classview` feature on `native/lgj-abi` binding From 6376415f5dc06b3e3fad8eaba99b3f2bd6fb026c Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 21:38:52 +0000 Subject: [PATCH 2/3] board: bank the EXP-KIA-A2-64K fresh in-tree measurement Runs measure_wal_curve fresh (lance-graph@1d7bc1b1) to answer "measure the 64k execution end first" before any BatchWriter/kanban seam design. Finding: compute parallelizes as documented (3.27x @ workers=8), but the cast/collect/wal/apply convergence tail is flat across every worker count and dominates the cycle (~90% of the workers=8 total) -- the sequential publish boundary this repo's own maxims already name, not compute, is where the seam design should focus. --- .claude/board/EPIPHANIES.md | 40 ++++++++ .claude/board/exp-kia-a2-64k-fresh-run.txt | 109 +++++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 .claude/board/exp-kia-a2-64k-fresh-run.txt diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index cbdea2a..81b9d19 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -4,6 +4,46 @@ > `**Status:**`/`**Confidence:**` line. A correction gets its own new, > dated entry that references the one it corrects — the storno rule. +## 2026-08-27 — E-EXP-KIA-A2-64K-CONVERGENCE-TAIL-DOMINATES-1 + +**Status:** MEASURED — in-tree, reproducible, this run banked verbatim at +`.claude/board/exp-kia-a2-64k-fresh-run.txt`. +**Confidence:** High for this host/harness; explicitly NOT the operator's +out-of-tree 125ms/233ms weather prior (different workload, per +`mask-native-navigation-correction-v1.md` §8.3 — that gap stays open). + +Fresh `lance-graph-supervisor/examples/measure_wal_curve` release run +(lance-graph@1d7bc1b1), answering "measure the 64k execution end first" +before any `BatchWriter`/kanban seam design: the EXP-KIA-A2-64K arm's +**compute** phase parallelizes as documented (21.8ms seq → 6.68ms @ +workers=8, 3.27x, matching the plan's own "~3.2-3.5x on 4 cores" prior +exactly) — but compute is the SMALL part of a cycle. Per-cycle steady-state +total (build/population excluded, sequential vs best-parallel): + +``` +workers=1: think 21.81 + cast 21.52 + collect 10.06 + wal 14.62 + apply 17.73 = 85.74 ms +workers=8: think 6.68 + cast 20.83 + collect 9.45 + wal 15.62 + apply 15.87 = 68.44 ms +``` + +**The cast/collect/wal/apply tail is FLAT across every worker count +(1/2/4/8/16) — it does not shrink when compute parallelizes** — and it is +~64-65ms of every cycle regardless: ~90% of the workers=8 total, ~10% +compute. Sequential-vs-parallel sealed-cycle digests MATCH at every worker +count (correctness holds; this is a WHERE-does-the-time-go finding, not a +correctness one). + +Consequence for the seam question root `CLAUDE.md` names ("64K COMPUTE WAS +PARALLEL … not yet the production loop"): parallelizing compute alone +recovers at most ~15ms of an ~85ms cycle in this harness. The convergence +boundary this repo's own maxims already flag as sequential-by-design +(ONE COMPUTATION IS NOT ONE LANCE WRITE; the SOLE native writer) is where +the time actually is — any seam design that only speeds up compute is +optimizing the part that was never the bottleneck on this measurement. +Cross-ref `CLAUDE.md`'s compute-model maxims and the GridLake block +(deterministic-landing-identity gate) — this measurement is evidence FOR +prioritizing that gate's resolution over a parallel-compute seam, not +against parallel compute itself. + ## 2026-08-27 — E-JAVA-IS-SIMD-RS-VALHALLA-PANAMA-IS-THE-POLYFILL-1 **Status:** DOCTRINE — [OPERATOR-FRAMED]. Pinned as the ENFORCEMENT LAYER in diff --git a/.claude/board/exp-kia-a2-64k-fresh-run.txt b/.claude/board/exp-kia-a2-64k-fresh-run.txt new file mode 100644 index 0000000..a9eb217 --- /dev/null +++ b/.claude/board/exp-kia-a2-64k-fresh-run.txt @@ -0,0 +1,109 @@ +measure_wal_curve — five-axis 64k measurement (release-mode; plan: .claude/plans/measure-64k-axes-v1.md) +FLEET_OWNERS=65536 CANONICAL_ROW_BYTES=512 CANONICAL_FRAME_BYTES=33554432 +measure.csv: writing rows to /tmp/measure_wal_curve.csv + +== B0 — DummyOwner cast baseline (owner lookup, write-on-behalf rebind, CastId allocation, staging, collect) == +B0 repeat 0: build=2532013ns scan=234930ns cast=17175060ns collect=17866807ns freeze=12147456ns total=49956266ns +B0 repeat 1: build=1972110ns scan=158831ns cast=9721674ns collect=12088807ns freeze=10378788ns total=34320210ns +B0 repeat 2: build=1551380ns scan=144869ns cast=8970797ns collect=10181449ns freeze=10208316ns total=31056811ns + +== B1a — 65,536 x MailboxSoA<4> (the actual hot runtime owner) == +B1a repeat 0: build=2769611970ns scan=991456ns think=8893074ns cast=61410849ns collect=37243161ns freeze=27997868ns apply=45104337ns total=2951252715ns peak_rss=1703395328B +B1a repeat 1: build=294982556ns scan=749351ns think=7612199ns cast=17293633ns collect=12161386ns freeze=10846729ns apply=14839613ns total=358485467ns peak_rss=1703608320B +B1a repeat 2: build=289954027ns scan=791278ns think=7604849ns cast=13131324ns collect=9485796ns freeze=10336337ns apply=14084119ns total=345387730ns peak_rss=1703981056B + +== B1b — 65,536 x NodeRow512 = 32 MiB canonical storage envelope (memory-only) == +B1b repeat 0: build=13910192ns peak_rss=1703981056B +B1b repeat 1: build=13834997ns peak_rss=1703981056B +B1b repeat 2: build=13618500ns peak_rss=1703981056B + +== L1a — 64 x MailboxSoA<1024> chunks, 65,536 LOGICAL owners (owner = chunk*1024+lane) == +L1a repeat 0: build=119493386ns scan=2524ns think=65388ns cast=7236769ns collect=10460086ns freeze=9486638ns total=146744791ns +L1a repeat 1: build=125003297ns scan=2775ns think=65234ns cast=7091750ns collect=9473531ns freeze=9165400ns total=150801987ns +L1a repeat 2: build=124457084ns scan=2370ns think=72025ns cast=9214763ns collect=11507104ns freeze=10489058ns total=155742404ns + +== L1b — 64 chunks AS owners (mislabelling CONTROL, never the 64k-owner model) == +L1b: build=127909536ns cast=7278541ns collect=8277040ns -> 65536 staged, 65472 of 65536 HELD (chunk-as-owner collapse, proves the control) + +== WAL curve — W1-contiguous (storage/cache ceiling) vs W0-current (SweepSlot/BTreeMap) == +WAL w1_contiguous segment_bytes=1048576 segments/cycle=32: median write=119035402ns p95=131000486ns sync=42980879ns (197.5 MiB/s) +WAL w0_current segment_bytes=1048576 segments/cycle=32: median write=11403469ns p95=127678714ns sync=44287259ns (574.6 MiB/s) +WAL w1_contiguous segment_bytes=2097152 segments/cycle=16: median write=8258261ns p95=22816912ns sync=41621083ns (641.5 MiB/s) +WAL w0_current segment_bytes=2097152 segments/cycle=16: median write=137842004ns p95=193008254ns sync=50270292ns (170.1 MiB/s) +WAL w1_contiguous segment_bytes=4194304 segments/cycle=8: median write=8749981ns p95=21850140ns sync=48620512ns (557.8 MiB/s) +WAL w0_current segment_bytes=4194304 segments/cycle=8: median write=144589584ns p95=203694904ns sync=56600040ns (159.1 MiB/s) +WAL w1_contiguous segment_bytes=8388608 segments/cycle=4: median write=8566550ns p95=29651854ns sync=46816680ns (577.8 MiB/s) +WAL w0_current segment_bytes=8388608 segments/cycle=4: median write=117406893ns p95=126960430ns sync=51230528ns (189.8 MiB/s) +WAL w1_contiguous segment_bytes=33554432 segments/cycle=1: median write=8070001ns p95=9303069ns sync=43727144ns (617.8 MiB/s) +WAL w0_current segment_bytes=33554432 segments/cycle=1: median write=121043589ns p95=164016760ns sync=50722889ns (186.3 MiB/s) +WAL curve gains (W1-contiguous, by segment_bytes): + segment_bytes= 1048576 throughput=197.5 MiB/s gain=n/a + segment_bytes= 2097152 throughput=641.5 MiB/s gain=+224.8% + segment_bytes= 4194304 throughput=557.8 MiB/s gain=-13.1% + segment_bytes= 8388608 throughput=577.8 MiB/s gain=+3.6% + segment_bytes= 33554432 throughput=617.8 MiB/s gain=+6.9% +WAL curve plateau: NOT MEASURABLE ON THIS HOST — 1 of 5 configs exceed the p95/median spread ceiling (3x); worst 3.5x. The write phase is being driven by page-cache / dirty-writeback state rather than by segment size, so NO knee is reported. Re-run on a quiet host with headroom (this one was ~90% full) and O_DIRECT or a drop_caches barrier per config. + +== Temporal — T0 scan_sealed / T1 local_trajectories / T2 deinterlace (post-WAL only) == +Temporal: T0 scan_sealed=21614316ns (1048576 rows) | T1 local_trajectories=81525493ns (65536 owners) | T2 deinterlace=8285517ns (524288 of 1048576 visible) + +== EXP-KIA-A2-64K — exploratory concurrency (non-claiming; D-KIA-A2 untouched) == +EXP-KIA: worker counts under test: [1, 2, 4, 8, 16] (available_parallelism=4) +EXP-KIA workers=1: build=334938430ns think(compute)=21811267ns max_active=1 cast=21518783ns collect=10059046ns wal=14618779ns apply=17727983ns total=420674288ns digest=248c6e7b991d3b25 +EXP-KIA workers=2: build=294678244ns think(compute)=9034649ns max_active=2 cast=20434826ns collect=10175085ns wal=14312366ns apply=17207461ns total=365842631ns digest=248c6e7b991d3b25 +EXP-KIA workers=4: build=289380526ns think(compute)=8656061ns max_active=4 cast=21645750ns collect=11782735ns wal=14877500ns apply=16317295ns total=362659867ns digest=248c6e7b991d3b25 +EXP-KIA workers=8: build=345653168ns think(compute)=6675257ns max_active=4 cast=20827314ns collect=9452535ns wal=15619567ns apply=15865932ns total=414093773ns digest=248c6e7b991d3b25 +EXP-KIA workers=16: build=288735149ns think(compute)=7041295ns max_active=5 cast=19674829ns collect=10330886ns wal=14213679ns apply=14658036ns total=354653874ns digest=248c6e7b991d3b25 +EXP-KIA: sequential-vs-parallel digests MATCH across all worker counts + +== M-arm — Morton reorder inserted before the seal (plan v3) == +M-arm digest identity: natural=68128e3662df105c morton=68128e3662df105c MATCH +M-arm natural: cast=27741249ns collect=10995879ns seal=55879501ns write=131989145ns sync=32276255ns T1=77641737ns +M-arm morton: cast=27991560ns collect=10912983ns reorder=8008125ns seal=58284224ns write=131767424ns sync=32715922ns T1=92222181ns fastpath=86846372ns (fastpath-vs-generic digest match=true) +M-arm SUM verdict: reorder_cost=8008125ns, downstream (seal+write+sync+T1) natural=297786638ns morton=314989751ns savings=-17203113ns -> delta_total=+25211238ns (Morton does NOT win under this workload/host (reorder cost exceeds downstream savings)) +M-arm reference: A0 measured T1 at 78-86ms over 1,048,576 rows — the number the fast path must beat; this run's fast path=86846372ns is the direct comparison (implementation-scoped: this implementation, this workload, this host). + +== O-arm — ordering source: O-A (cast->seal->WAL->temporal replay) vs O-B (cast->temporal replay->seal->WAL) == +O-arm firewall: derive_order_from_temporal_replay (1351 bytes) contains no scan_sealed / sealed-store read; local_trajectories present (scan mechanism proven live) +O-arm PRIMARY OBSERVABLE (decided before timing): O-A digest=64565f362db2e4a5 O-B digest=3e71c2aa7be8e325 -> DIVERGED +O-arm VERDICT: DIVERGED — ordering sourced from temporal replay does NOT reproduce the seal's ordering. Under this construction the seal's ordering is LOAD-BEARING and cannot be re-scoped away. Honest scope: this falsifies the hypothesis FOR THIS O-B CONSTRUCTION; it does not prove that no construction could match. The divergence itself is the finding. +O-arm kill-condition check: CONSTRUCTIBLE. O-B's ordering derivation (`local_trajectories` grouping, ~O(n log n) via BTreeMap insertion) is a DIFFERENT code path from O-A's seal-side sort (`order_cycle_stably`'s Vec sort_by_key, also O(n log n)) — not literally shared code, so this is not a disguised O-A. Under THIS harness's one-row-per-owner-per-cycle shape the two algorithms are doing comparable asymptotic work; the redundancy the plan asks about is SEMANTIC (does temporal's grouping make the seal's own sort unnecessary for correctness), not literal code-sharing — reported honestly, not glossed over. +O-A (today's pipeline): cast=36869556ns collect=12512526ns seal=60766385ns commit=7365397ns T1=236612430ns +O-B (ordering sourced first): cast=53955419ns collect=15452292ns order_derive=43480687ns seal=71368994ns commit=4062285ns T1=208751643ns + +measure.csv: 183 rows written +measure.csv: file at /tmp/measure_wal_curve.csv +hot representation overhead: B1a MEASURED VmRSS delta +54661120B (52.1 MiB) minus canonical envelope 33554432B (32.0 MiB, exact by construction) = +21106688B (+20.1 MiB, +63%) + (B1b in-process VmRSS delta +0B — reads ~0 by allocator reuse, which is why the line above uses the exact canonical size) + +================ THE FOUR ANSWERS ================ +1. Ownership cost (B1a MailboxSoA<4> minus B0 DummyOwner, median of 3): scan +632447ns, cast/rebind +7571959ns, freeze +467941ns (B1a also pays a real per-owner think phase B0 has none of: 7612199ns, and a real apply phase: 14839613ns — B0 has neither). +2. Physical layout cost (L1a 64x MailboxSoA<1024> minus B1a 65,536x MailboxSoA<4>, equal 65,536 logical owners, build..freeze phases only — apply is not comparable, see the §12 doc comment): build -170525472ns, cast/rebind -10056864ns, freeze -1360091ns. +3. WAL amortisation plateau: NOT MEASURABLE ON THIS HOST — 1 of 5 configs exceeded the p95/median spread ceiling (worst 3.5x). The write phase is driven by page-cache / dirty-writeback state, not segment size, so NO knee is claimed. Four runs of this binary disagreed by up to 6x at identical configs. Needs a quiet host with disk headroom plus O_DIRECT or a per-config cache barrier. W0-current numbers are in the CSV, never substituted here. +4. EXP-KIA-A2-64K (exploratory, NON-CLAIMING — D-KIA-A2's own median-of-5 >=2x gate is untouched by this number): sequential (workers=1) compute-phase median 21811267ns vs best observed parallel compute-phase median 6675257ns at workers=8; sequential-vs-parallel sealed-cycle digests MATCH across every worker count tested. +==================================================== + +--- +Banked 2026-08-27, in-tree run (lance-graph@1d7bc1b1, release build, this host, +available_parallelism=4). Full stderr of +`cargo run --release -p lance-graph-supervisor --features cycle-driver +--example measure_wal_curve` (all five axes; EXP-KIA-A2-64K is the section +this bank exists for). Answers the mask-native-navigation-correction-v1.md +§8.3 receipt-provenance gap: the 125ms/233ms pair stays operator-measured/ +out-of-tree (still not this workload — a different weather probe); THIS run +is the in-tree, reproducible number for the EXP-KIA-A2-64K shape itself. + +Per-cycle steady-state total (build excluded — build is one-time population, +not part of the repeating cycle), summing think+cast+collect+wal+apply: + workers=1: 21811267+21518783+10059046+14618779+17727983 = 85735858 ns (~85.7 ms) + workers=8: 6675257+20827314+9452535+15619567+15865932 = 68440605 ns (~68.4 ms) + +Compute (think) alone: 21.8ms seq -> 6.68ms @ workers=8 (3.27x, matches the +plan's "~3.2-3.5x compute overlap on 4 cores" prior exactly). +Cast/collect/wal/apply (the convergence/publish tail): ~64-65ms, FLAT across +every worker count (20.8-21.6ms cast, 9.5-11.8ms collect, 14.2-15.6ms wal, +14.7-17.7ms apply) -- unparallelised in this harness, and it dominates the +cycle: at workers=8 the sequential tail is ~90% of the steady-state total +while compute is ~10%. Digests match sequential-vs-parallel at every worker +count -- correctness holds; the finding is about WHERE the time goes, not +about correctness. From 22f3293c1148ff541602e76127153f73f4caad41 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 27 Aug 2026 22:04:22 +0000 Subject: [PATCH 3/3] doctrine: pin zero-copy + memory safety as normative, merge-gating Operator issued a 32-point zero-copy/memory-safety addendum. Audited it mechanically against the tree (generation-checked handles, checked- overflow bounds arithmetic, explicit alignment/endianness contract fields, manifest-first handshake ordering, FFM quarantine, named/bounded materialization only, independently-derived layout parity, SIMD-backend- diagnostic-only, worker-topology-substrate-private) rather than redesigning from taste. No gap found -- every item is already structurally enforced. Pins the doctrine as a compact, cited section in CLAUDE.md so future membrane PRs are reviewed against it directly. --- .claude/board/EPIPHANIES.md | 45 ++++++++++++++++++++++ CLAUDE.md | 76 +++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 81b9d19..99fda8a 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -4,6 +4,51 @@ > `**Status:**`/`**Confidence:**` line. A correction gets its own new, > dated entry that references the one it corrects — the storno rule. +## 2026-08-27 — E-ZERO-COPY-MEMORY-SAFETY-AUDITED-CLEAN-1 + +**Status:** AUDITED CLEAN — pinned as normative doctrine in root +`CLAUDE.md` (new "Zero-copy + memory safety" section), same commit. +**Confidence:** High — every item checked against source, not assumed. + +Operator issued a 32-point normative zero-copy/memory-safety addendum +(merge-gating). Ran it mechanically against the tree rather than +redesigning from taste, per the session's standing method: + +- Provenance-bound generation-checked handles: `registry.rs` — + `fabricated_handles_are_rejected_not_dereferenced`, + `a_reused_slot_invalidates_the_old_handle` (already present). +- Checked-overflow bounds arithmetic: `checked_mul`/`checked_add` + throughout `rowstore.rs`/`kernels.rs` (already present). +- Alignment/endianness as explicit contract fields (never inferred): + `LgjAbiManifest.align_of_*` filled from `core::mem::align_of` on the + real types; `Abi.java` rejects non-little endianness before any + projection (already present). +- Manifest-first handshake ordering (magic → major → minor → sizes → + endianness, `requireMinor(N)` gating): `Abi.java` (already present). +- FFM quarantine: zero `java.lang.foreign`/`java.lang.invoke` in any + public signature; internal use in `RowStore.java`/`FacetMatchView.java` + is private-field-only (verified by grep for `public .*MemorySegment/ + Arena/ValueLayout/MethodHandle` — zero hits). +- Named/bounded materialization only: every production `long[]`/ + `copyOf`/`toArray` traces to `Mask.materializeRows()`, the manifest + handshake's own name-string read, or `rowLayoutProbe`'s ≤32-byte + diagnostic — no hidden proportional-to-n_rows copy. +- Independently-derived layout parity (not self-compared): + `AbiContractTest`, with a deliberately-impossible-expectation arm + proving the check can fail. +- SIMD backend diagnostic-only: `NativeRuntime.simdBackend()` is a + manifest string; zero `if` branches on it in `src/main`. +- Worker topology substrate-private: zero `workers(`/`workerCount`/ + `parallelism(`/`threads(` in production Java/ABI/exports (reconfirmed + from the prior turn's audit, unchanged). + +**Verdict: no gap found, nothing redesigned.** The addendum's own +mandatory checklist (§32) is answered YES on every line by structure +already in the tree — this entry and the CLAUDE.md section exist to +PIN the doctrine as merge-gating going forward, not to fix a defect +found today. Any future PR touching the membrane is reviewed against +this section directly. + ## 2026-08-27 — E-EXP-KIA-A2-64K-CONVERGENCE-TAIL-DOMINATES-1 **Status:** MEASURED — in-tree, reproducible, this run banked verbatim at diff --git a/CLAUDE.md b/CLAUDE.md index f10d322..9f175df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -110,6 +110,82 @@ per-owner `advance(owner)` RPCs are exactly the deleted shape. `BatchWriter::cast()` = staging into a batch image, NOT command/ack messaging. +## Zero-copy + memory safety — NORMATIVE, MERGE-GATING (operator-ruled, 2026-08-27) + +**Zero-copy means the canonical bytes remain owned by `lance-graph`; Java/ +Panama project operations onto those bytes without duplicating canonical +state.** What crosses the membrane is identity, descriptor, layout +contract, projection hint, operation, and result identity — never a +second graph. **One-copy law**: multiple views/lanes/masks over the same +substrate are fine; multiple *authorities* are not. + +**Shape may cross. Meaning may cross. Operations may cross. Ownership +does not cross.** The 512×32×(4+12) row/facet contract, offsets, stride, +alignment, endianness are legitimate contract facts Java may know — +zero-copy is about storage *ownership*, never about layout *opacity*. + +Non-negotiables, each with its enforcement site (audited clean +2026-08-27 — this is confirmation of existing structure, not a new +build): + +- **Pointer value is not provenance.** Every native address is bound to + owner identity + generation + length + kind via the generation-checked + handle registry (`registry.rs::{encode_handle, resolve, resolve_kind}`); + a stale generation fails closed before dereference. Falsifiers: + `fabricated_handles_are_rejected_not_dereferenced`, + `a_reused_slot_invalidates_the_old_handle`. +- **Bounds/overflow are checked, never wrapped.** `rowstore.rs`/ + `kernels.rs` use `checked_mul`/`checked_add` throughout; overflow + fails closed, never truncates. +- **Alignment and endianness are contract fields, never inferred.** + `LgjAbiManifest.align_of_*` are filled from `core::mem::align_of` on + the real types (never a literal); `endianness` is verified explicit + (`Abi.java` rejects non-zero before any projection). +- **Manifest-first handshake.** `Abi.java` reads magic → abi_major + (exact) → minor (>=) → struct sizes/alignment → endianness, in that + order, before any dependent layout is resolved — never a speculative + read past the guaranteed prefix. `requireMinor(N)` gates every + minor-N-or-later operation; an older library fails cleanly at the + call, not at load. +- **FFM is quarantined.** `java.lang.foreign.*`/`java.lang.invoke.*` + never appear in a public signature (`ApiSurfaceTest`); internal use in + `RowStore.java`/`FacetMatchView.java` is private-field-only, verified + by reflection, not by convention. +- **Mutation crosses as verbs, not writable memory.** No public API + exposes a writable canonical segment; mutation happens through named + ABI operations (`mask_and`/`apply_projection`/etc.), never + `segment.set(...)`. +- **Materialization is named and bounded.** The only production + `long[]`/`copyOf`/`toArray` call sites are `Mask.materializeRows()` + (the one named terminal), the manifest-name read during handshake, and + `rowLayoutProbe`'s ≤32-byte diagnostic — none is a hidden + proportional-to-n_rows population copy. Temporary kernel scratch + (SIMD scratch masks, decode buffers) is allowed and is NOT the same + claim as a second canonical copy. +- **Layout parity is independently derived, not self-compared.** + `AbiContractTest`: Java's own layout constants vs. the artifact's + runtime self-description, with a deliberately-impossible-expectation + arm proving the check can actually fail. +- **SIMD backend is diagnostic only.** `NativeRuntime.simdBackend()` is + a manifest string for logging; no `if` branches on it anywhere in + `src/main` — backend parity is `ndarray::simd`'s business (see E1-E6 + below), never Java's. +- **Worker topology stays substrate-private** — see §E of the + mask-native-navigation-correction-v1.md enforcement pass: no + `workers(`/`workerCount`/`parallelism(`/`threads(` in any production + Java, ABI struct, or export; `EXP-KIA-A2-64K`'s worker sweep is a + native benchmark independent variable, never a consumer-facing API. + +Wording discipline (use exactly): *"lance-graph owns the only canonical +copy. lance-graph-java projects semantic operations across Panama onto +that state without row/population duplication."* Never *"Java borrows +the native database memory directly"* — that overclaims the +abstraction. Never *"FFI is memory-safe end-to-end"* — the defensible +claim is *"the Java consumer cannot directly express arbitrary native +memory access; native resources are accessed through generation-checked, +version-checked, bounds-checked ABI operations whose ownership remains +in lance-graph."* + ## The simd.rs isomorphism — ENFORCEMENT LAYER (operator-ruled, 2026-08-27) The repo's whole shape is `ndarray`'s own SIMD architecture repeated one