Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 70 additions & 3 deletions .claude/board/EPIPHANIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,78 @@
> `**Status:**`/`**Confidence:**` line. A correction gets its own new,
> dated entry that references the one it corrects — the storno rule.

## 2026-08-28 — E-ZERO-COPY-MEMORY-SAFETY-OVERCLAIM-CORRECTION-1

**Status:** CORRECTS `E-ZERO-COPY-MEMORY-SAFETY-AUDITED-CLEAN-1` (below) —
that entry's "no gap found" verdict itself overclaimed. Storno entry per
board README's append-only rule; the corrected entry's prose is left
untouched below except this note and its own Status/Confidence lines.
**Confidence:** High — every correction below is a five-savant-verified,
three-reviewer-ratified finding (5+3 council, PR #45 as target, run
2026-08-28), not a single-pass re-read.

The prior entry's single-pass audit (a general-purpose lens, not a
council) confirmed all 9 doctrine bullets by citation but did not
independently RE-DERIVE several claims — a 5+3 council convened
specifically because that audit was the only review PR #45 got
(CodeRabbit's own comment was just its auto-summary, "Review failed — the
pull request is closed"; Bugbot hit its usage limit twice and never ran).
The council found 5 real overclaims in the CLAUDE.md doctrine text,
fixed in the same commit as this entry:

1. **"a stale generation fails closed before dereference"** was stated as
universal; it is false for the cached-descriptor path
(`lgj_lane_describe`/`lgj_mask_describe` hand Java a raw `addr` once,
read directly thereafter with no further registry call — guarded only
by a Java `closed` boolean, not the generation registry). Fixed:
scoped to "handle-mediated" operations; the cached-descriptor gap
named explicitly, its `epoch` re-check tracked as `ISS-LGJ-EPOCH-
UNCHECKED`.
2. **"an older library fails cleanly at the call, not at load"** was
stated as blanket; false for ABI minors 2-4 (row store/edges/hop),
which fail at `Downcalls.<clinit>` via eager `MethodHandle` resolution
— a gap `Downcalls.java`'s own comment already tracked, that the new
doctrine text presented as already closed. Fixed: scoped to the
minor-5+ lazy-holder pattern; minors 2-4 named as the tracked
exception.
3. **"checked_mul/checked_add throughout rowstore.rs and kernels.rs"**
overclaimed uniform coverage: `kernels.rs` has zero occurrences (it
bounds against an already-allocated slice's real `.len()` via
`assert_eq!`, a sound but DIFFERENT mechanism); `rowstore.rs` has
exactly two, both at the two allocation sites. Fixed: scoped to "at
the point n_rows is first derived", with `kernels.rs`'s actual
mechanism named rather than left silent.
4. **"never `segment.set(...)`"** was contradicted by the already-named
Import exception (`RowStore.importRows` → `Engine.setU64` →
`segment.set(...)`, on a lane the ABI marks `LGJ_FLAG_WRITABLE` by
design). Fixed: the bullet now cross-references the exception it
already names elsewhere in this file, rather than reading as an
absolute the codebase visibly violates.
5. **The materialization list was a closed enumeration missing two real
call sites** (`Abi.java`'s `readCarvings`, `Engine.facetSumResolved`'s
`long[2]`) — both bounded/non-population-proportional, so the
underlying property held even though the list didn't. The council's
reviewers split on the fix shape (overclaim-auditor favored converting
to a property claim; dilution-collapse-sentinel BLOCKed that as a
collapse — a checkable exhaustive list has real falsifiability value a
property claim loses) — the stricter verdict won: the list is now
five items, kept exhaustive, not converted to prose.

**Process finding, not a doctrine-text issue:** PR #45 itself had no
`PR_ARC_INVENTORY.md` entry — a real board-hygiene gap independent of the
doctrine overclaims, caught by this council's `firewall-warden` pass and
backfilled in this same commit (see that file).

## 2026-08-27 — E-ZERO-COPY-MEMORY-SAFETY-AUDITED-CLEAN-1

**Status:** AUDITED CLEAN — pinned as normative doctrine in root
`CLAUDE.md` (new "Zero-copy + memory safety" section), same commit.
**Confidence:** High — every item checked against source, not assumed.
**Status:** CORRECTED 2026-08-28 by `E-ZERO-COPY-MEMORY-SAFETY-OVERCLAIM-
CORRECTION-1` (above) — a 5+3 council found 5 real overclaims in the
doctrine text this entry certified. Originally: "AUDITED CLEAN — pinned
as normative doctrine in root `CLAUDE.md`, same commit."
**Confidence:** Medium — the single-pass audit's citation checks were
individually accurate (every named function/test does exist), but the
"no gap found" verdict overclaimed; it verified citations, not universal
scope. See the correction entry for what was actually wrong.

Operator issued a 32-point normative zero-copy/memory-safety addendum
(merge-gating). Ran it mechanically against the tree rather than
Expand Down
33 changes: 33 additions & 0 deletions .claude/board/ISSUES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
# Issues Log — Open + Resolved (double-entry, append-only)

## ISS-LGJ-EPOCH-UNCHECKED (2026-08-28) — OPEN

**Found.** By the 5+3 council's `handle-lifecycle-auditor` pass on PR #45
(the zero-copy/memory-safety doctrine review). `LgjLaneDesc`/`LgjMaskDesc`
carry an `epoch` field (`exports.rs`, lane/mask describe exports)
specifically so a Java-side holder of a cached descriptor address could
re-validate it against the live resource. Grepped `epoch` across
`java/src/main` — it appears only in layout/record construction
(`Engine.java`'s `LaneWindow`), never in a comparison. The re-check the
field exists for is unwired; the only guard on the cached-descriptor path
is a Java `closed` boolean (weaker: it does not detect a slot that closed
and was reused for a different resource within the same process, the way
a generation comparison would).

**Not urgent, not silent-corruption-shaped today**: `RowStore`/`Mask`
mark themselves permanently closed on `close()` (no slot reuse observed
from the Java facade's own lifecycle — a `RowStore` handle is never
recycled to address a different resource while a `lanes[]` cache still
points at the old one, per `RowStoreLifetimeTest`). The gap is real but
currently unreachable through the facade's own public API; it becomes
load-bearing only if a future capability lets a Java-held descriptor
outlive a same-slot resource swap. Filed so it is not silently assumed
covered by the generation-registry claim in `CLAUDE.md`'s zero-copy
section (see that section's "Pointer value is not provenance" bullet,
corrected 2026-08-28 to name this gap explicitly rather than imply
`epoch` is checked).

**Next step, not yet scheduled**: wire `epoch` comparison into
`RowStore`/`Mask`'s per-access path (`lane(int)`, `checkedRow`), or, if
measurement shows the facade's own close-discipline makes it provably
unreachable, downgrade this entry to a documented invariant rather than
a live gap — either resolution needs the measurement, not assumption.

## ISS-LGJ-HOP-LAYOUT-BLOCKS-THE-ALGEBRA (2026-08-27) — RESOLVED (same day; ABI minor 10)

**Found.** By landing R1 (selection as mask algebra) and measuring it.
Expand Down
41 changes: 41 additions & 0 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,47 @@
> anti-pattern the imported board rules name. Backfilled below in one
> pass rather than left stale; PR #4 onward gets its entry at merge time.

## PR #45 — board: PR #44 arc entry + EXP-KIA-A2-64K fresh measurement + zero-copy/memory-safety doctrine pin (merged 2026-08-28, `b2956d3` — 2 commits, head `22f3293`)

- **Added:** the PR #44 backfill entry below (this file); a fresh
in-tree `measure_wal_curve` run answering "measure the 64k execution
end first" (`.claude/board/exp-kia-a2-64k-fresh-run.txt`,
`E-EXP-KIA-A2-64K-CONVERGENCE-TAIL-DOMINATES-1`); a 9-bullet "Zero-copy
+ memory safety" NORMATIVE, MERGE-GATING section in root `CLAUDE.md`
transcribing the operator's 32-point addendum, each bullet cited to a
real enforcement site.
- **Locked (at merge, before the correction below):** compute
parallelizes (3.27× @ workers=8) but the cast/collect/wal/apply tail is
flat across worker count and dominates the cycle (~90% at workers=8) —
evidence for prioritizing the GridLake deterministic-landing-identity
gate over a parallel-compute seam.
- **Post-merge correction, same-day (2026-08-28):** PR #45 was the ONLY
review layer this doctrine got — CodeRabbit posted only its
auto-summary (PR merged before its review cycle ran), Bugbot hit its
usage limit twice and never ran. A 5+3 council (5 savants incl. this
repo's own `abi-membrane-warden`/`zero-copy-warden`/`handle-lifecycle-
auditor`/`java-surface-warden` cards, 3 reviewers) found 5 real
overclaims in the doctrine text and fixed them in a follow-up commit
(see `E-ZERO-COPY-MEMORY-SAFETY-OVERCLAIM-CORRECTION-1`): the
generation-registry claim didn't cover the cached-descriptor path;
`requireMinor`'s "fails cleanly at the call" claim didn't hold for
minors 2-4; the bounds-checking claim overclaimed `kernels.rs`
coverage; "never `segment.set(...)`" was contradicted by the
already-named Import exception; the materialization list was missing
two bounded call sites. Also caught: this PR's own missing arc entry
(this row, backfilled in the correction commit).
- **Deferred, named:** `ISS-LGJ-EPOCH-UNCHECKED` (the `epoch`
re-validation field exists in the ABI but is never consulted by the
Java facade — currently unreachable through the public API, not yet a
live gap).
- **Docs:** `CLAUDE.md` doctrine section corrected in place (5 bullets
reworded); `ISSUES.md` gains `ISS-LGJ-EPOCH-UNCHECKED`; this file
backfilled for both #44 and #45.
- **Confidence:** the doctrine text is now council-ratified (5+3, full
sequencing) rather than single-pass-audited; the 64K measurement entry
is unaffected by the correction (a different EPIPHANIES entry, not
reviewed by this council).

## PR #44 — R1→minor 10: mask algebra restored, the reduction repatriated, the doctrine pinned, the columnar store landed (merged 2026-08-27, merge — 5 commits, head `bd6f666`)

- **Added, as one arc** (the PR body predates its own last three commits —
Expand Down
92 changes: 67 additions & 25 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,15 +128,37 @@ Non-negotiables, each with its enforcement site (audited clean
2026-08-27 — this is confirmation of existing structure, not a new
build):

- **Pointer value is not provenance.** Every native address is bound to
owner identity + generation + length + kind via the generation-checked
handle registry (`registry.rs::{encode_handle, resolve, resolve_kind}`);
a stale generation fails closed before dereference. Falsifiers:
`fabricated_handles_are_rejected_not_dereferenced`,
`a_reused_slot_invalidates_the_old_handle`.
- **Bounds/overflow are checked, never wrapped.** `rowstore.rs`/
`kernels.rs` use `checked_mul`/`checked_add` throughout; overflow
fails closed, never truncates.
- **Pointer value is not provenance, on the handle-mediated path.** Every
ABI *handle* (`Mask`/`RowStore`/`NativePattern` opaque `long`s) is bound
to owner identity + generation + kind via the generation-checked handle
registry (`registry.rs::{encode_handle, resolve, resolve_kind}`); a
stale generation fails closed before dereference, on every resolve call.
Falsifiers: `fabricated_handles_are_rejected_not_dereferenced`,
`a_reused_slot_invalidates_the_old_handle` (the latter is conditionally
vacuous under some registry-slot orderings — cite alongside
`use_after_close_is_a_status_not_a_crash`, the falsifier that holds
unconditionally). Generation is `u32` and wraps after 2³² closes of one
slot (documented in-source at the wrap site; not reachable in practice).
**Scope note, not covered by the registry check:** `lgj_lane_describe`/
`lgj_mask_describe` hand Java a raw `addr` once, which Java then caches
(`RowStore.java`'s `lanes[]`, `Mask.java`'s `words`) and reads directly
on every subsequent access with NO further registry call — that repeat
path is guarded only by a Java-side `closed` boolean, a strictly weaker,
non-generation-checked mechanism. `LgjLaneDesc` carries an `epoch` field
designed for exactly this re-check and it is currently unconsulted
anywhere in `src/main` (tracked: `.claude/board/ISSUES.md`
`ISS-LGJ-EPOCH-UNCHECKED`). Do not read "fails closed before dereference"
as covering the cached-descriptor path until that epoch check is wired.
- **Bounds/overflow are checked, never wrapped, at the point n_rows is
first derived.** `rowstore.rs` uses `checked_mul` at its two allocation
sites (`generate_in`/`generate_with_edges_in`); overflow there fails
closed, never truncates. `kernels.rs` does not re-derive `n_rows` — it
operates on an already-allocated, already-length-checked slice and
bounds itself against that slice's real `.len()` via `assert_eq!`
(e.g. `kernels.rs:199`), not a second checked multiplication; that is a
sound bound, not an unchecked one, but it is a different mechanism than
the allocation-time check and should not be described as the same
`checked_mul`/`checked_add` machinery "throughout" both files.
- **Alignment and endianness are contract fields, never inferred.**
`LgjAbiManifest.align_of_*` are filled from `core::mem::align_of` on
the real types (never a literal); `endianness` is verified explicit
Expand All @@ -145,23 +167,43 @@ build):
(exact) → minor (>=) → struct sizes/alignment → endianness, in that
order, before any dependent layout is resolved — never a speculative
read past the guaranteed prefix. `requireMinor(N)` gates every
minor-N-or-later operation; an older library fails cleanly at the
call, not at load.
minor-5-or-later operation cleanly (an older library fails at the call,
via the lazy nested-holder pattern). **Minors 2-4 (row store, edges,
hop) are a tracked exception, not yet covered**: their `Downcalls`
holders resolve `MethodHandle`s eagerly at class-init, so an ABI-0.1
library fails at `Downcalls.<clinit>` before any `requireMinor` guard
can report a clean error (`Downcalls.java`'s own comment documents
this). Do not claim "fails cleanly at the call, not at load" as a
blanket property until minors 2-4 adopt the same lazy-holder shape.
- **FFM is quarantined.** `java.lang.foreign.*`/`java.lang.invoke.*`
never appear in a public signature (`ApiSurfaceTest`); internal use in
`RowStore.java`/`FacetMatchView.java` is private-field-only, verified
by reflection, not by convention.
- **Mutation crosses as verbs, not writable memory.** No public API
exposes a writable canonical segment; mutation happens through named
ABI operations (`mask_and`/`apply_projection`/etc.), never
`segment.set(...)`.
- **Materialization is named and bounded.** The only production
`long[]`/`copyOf`/`toArray` call sites are `Mask.materializeRows()`
(the one named terminal), the manifest-name read during handshake, and
`rowLayoutProbe`'s ≤32-byte diagnostic — none is a hidden
proportional-to-n_rows population copy. Temporary kernel scratch
(SIMD scratch masks, decode buffers) is allowed and is NOT the same
claim as a second canonical copy.
never appear in a public signature (`ApiSurfaceTest`, verified by real
reflection over compiled classes — `Class.getMethods/getFields` plus a
`com.adaworldapi.lancegraph.internal.` package-prefix check — not a
source-text grep); internal use in `RowStore.java`/`FacetMatchView.java`
is private-field-only, and no other file in the public package touches
an FFM type.
- **Mutation crosses as verbs, not writable memory — except through the
named Import exception.** No public API exposes a writable canonical
segment for general use; mutation happens through named ABI operations
(`mask_and`/`apply_projection`/etc.), never ad hoc `segment.set(...)`.
The one sanctioned exception is the **Import** path already named above
(`RowStore.importRows`, `Graph.from(long...)`): it does call
`Engine.setU64` → `segment.set(...)` on a mask lane the ABI marks
`LGJ_FLAG_WRITABLE` by design, and it does run a per-row Java loop over
the imported ids — both are the Import exception's documented cost, not
a second, unnamed violation of this bullet.
- **Materialization is named and bounded.** Production `long[]`/`copyOf`/
`toArray` call sites, exhaustively: `Mask.materializeRows()` (the one
named terminal, O(n)); the manifest-name read during handshake;
`rowLayoutProbe`'s ≤32-byte-per-call diagnostic (bounded by facet count,
not row count — verified fixed-size on both the Rust and Java sides);
`Abi.java`'s `readCarvings` (bounded by `CARVING_SLOTS`, a manifest
constant, not n_rows); `Engine.facetSumResolved`'s fixed `long[2]`
result pair. None of the five is a hidden proportional-to-n_rows
population copy — keep this list exhaustive when a sixth site is added,
rather than letting the enumeration silently go stale again.
Temporary kernel scratch (SIMD scratch masks, decode buffers) is
allowed and is NOT the same claim as a second canonical copy.
- **Layout parity is independently derived, not self-compared.**
`AbiContractTest`: Java's own layout constants vs. the artifact's
runtime self-description, with a deliberately-impossible-expectation
Expand Down