From c7f853cfb309326dd09043a390ed64619fb84d68 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 25 Aug 2026 13:45:15 +0000 Subject: [PATCH] =?UTF-8?q?abi:=20minor=207=20=E2=80=94=20the=20whole-row?= =?UTF-8?q?=20layout=20probe,=20and=20the=20classid=20table=20made=20globa?= =?UTF-8?q?l?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three corrections and one addition, all from the same observation: a classid is a GLOBAL address that captures LAYOUT. CLASSIDS ARE NOT PER-SoA. The classid -> grouping table was on RowStore, one 64 KiB copy per dataset. The same classid means the same class in every SoA, so the resolution is dataset-independent -- provably here, since FixtureClassView is a unit struct with no per-store state. Hoisted to a process-global LazyLock in class_view_provider, where it belongs: it is a property of the ClassView, not of any store. Wrong in shape rather than in output -- the answers were right, but the placement implied two datasets could disagree, which the address space does not permit. THE TABLE CAPTURES LAYOUT ONLY. Meaning, RBAC, ontology category and render template are separate resolutions off the same address; none belong in it and none can be inferred from it. Stated in the doc so the table does not accrete. lgj_row_layout_probe: for EVERY facet, the set of groupings its selected rows carry, in ONE crossing. Asking per facet would be 32 crossings and is how a consumer drifts into the per-element loop §6 forbids. Alignment falls out as arithmetic rather than a scan: per facet, OR-accumulate a 3-bit set (plus one bit for an unanswerable classid), then aligned <=> popcount(byte) == 1 && no unanswerable bit One `or` per (row, facet), no comparison, no early exit, cost independent of the data. An OR-accumulated SET is exact where cheaper accumulators are not: a sum of wire values cannot tell {0,2} from {1,1}, and an XOR cannot tell {1,1} from {}. The set forgets multiplicity, which is exactly what the question does not need. 0 means the EMPTY set and is deliberately distinguishable from disagreement. It paid immediately. A test asserting that a maskOfFacetClass(facet 3, …) selection is fully aligned FAILED -- and the expectation was wrong, not the code: that mask constrains facet 3 only, so the other 31 facets carry whatever classids the generator gave them. Measured 1 of 32 facets aligned. That is the confusion a whole-row probe exists to remove. R10 (valhalla-lab) bolts the same three schemas into Valhalla and Panama and proves all three descriptions agree: raw storage bytes, a Panama MemoryLayout, and a Valhalla value class decode every register identically, each layout describes exactly 12 bytes, and the schemas genuinely read differently so the agreement is not trivial. The schema bolts on at the GROUP, not the register: 12 = 6x2 = 4x3 = 3x4 means the largest group is 4 B (half the budget) while the register is 12 and the facet 16, neither of which Java can flatten or needs to. Gates: 132 Rust, 288 Java, clippy -D warnings clean, fmt clean, abi 0.7. --- docs/abi.md | 67 +++++++- .../com/adaworldapi/lancegraph/FacetId.java | 4 + .../com/adaworldapi/lancegraph/RowLayout.java | 98 +++++++++++ .../com/adaworldapi/lancegraph/RowStore.java | 17 ++ .../lancegraph/internal/ffm/Downcalls.java | 24 +++ .../lancegraph/internal/ffm/Engine.java | 13 ++ .../lancegraph/FacetSumParityTest.java | 58 +++++++ native/lgj-abi/src/abi.rs | 2 +- native/lgj-abi/src/class_view_provider.rs | 115 +++++++++++++ native/lgj-abi/src/exports.rs | 86 ++++++++-- native/lgj-abi/src/kernels.rs | 77 +++++++++ native/lgj-abi/src/rowstore.rs | 44 ----- valhalla-lab/reproducers/R10-observed.txt | 12 ++ .../R10_SchemaAlignsWithStorage.java | 152 ++++++++++++++++++ valhalla-lab/reproducers/README.md | 23 +++ 15 files changed, 734 insertions(+), 58 deletions(-) create mode 100644 java/src/main/java/com/adaworldapi/lancegraph/RowLayout.java create mode 100644 valhalla-lab/reproducers/R10-observed.txt create mode 100644 valhalla-lab/reproducers/R10_SchemaAlignsWithStorage.java diff --git a/docs/abi.md b/docs/abi.md index 49945c4..97446e2 100644 --- a/docs/abi.md +++ b/docs/abi.md @@ -62,7 +62,7 @@ cannot disagree with itself. The ABI is a **machine membrane**. It is not the product. The product is the Java semantic API (see `architecture.md`). Therefore: -- It is **small** — currently 23 symbols (minor 6; the "14" this line carried +- It is **small** — currently 24 symbols (minor 7; the "14" this line carried at minor 1 was arithmetic drift — the §7 list it referred to already enumerated 15). Growth is a design smell to be argued for, not a default; minor 2's three additions are argued in §11, minor 3's one addition in @@ -79,7 +79,7 @@ semantic API (see `architecture.md`). Therefore: ``` LGJ_ABI_MAJOR = 0 // incompatible change ⇒ bump; Java refuses to load -LGJ_ABI_MINOR = 6 // additive change ⇒ bump; older Java may still load +LGJ_ABI_MINOR = 7 // additive change ⇒ bump; older Java may still load LGJ_MAGIC = 0x4C_47_4A_5F_41_42_49_00 // "LGJ_ABI\0" big-endian-read ``` @@ -132,6 +132,8 @@ required — a gate that rejected everything would satisfy a rejection-only test - **Minor 4** (2026-08-18, D-LGJ-W8) — `lgj_mask_andnot` (mask complement) and `lgj_hop` (one-hop graph traversal, gated by the `lance-graph-contract` `ClassView`/`FieldMask` LAW — §13). +- **Minor 7** (2026-08-25) — `lgj_row_layout_probe` (§16): the whole-row + alignment answer, all 32 facets in one crossing. No new status. - **Minor 6** (2026-08-25) — `lgj_reduce_facet_sum_resolved` (§15): the same sweep, but under the grouping the POPULATION resolves to via `ClassView::cascade_shape`, rather than one the caller asserts. One new @@ -326,7 +328,7 @@ predicates or rows are involved. The unfused per-predicate ops are retained only so the fused path can be benchmarked *against* something and so parity can be checked predicate-by-predicate. -## 7. The function surface (23 symbols) +## 7. The function surface (24 symbols) All symbols are prefixed `lgj_`. All return `i32` status except the manifest getter. `out_*` parameters are written only on `OK`. @@ -1006,3 +1008,62 @@ answer rather than reading as an empty memo. A test-only counter (`RESOLUTIONS`) makes the memo's behaviour observable rather than asserted: the first sweep resolves, five repeats over the same population do not, a different facet does, and a rewritten population does. + +## 16. The whole-row layout probe (ABI minor ≥ 7) + +``` +i32 lgj_row_layout_probe(u64 res, u64 mask, u8* out, u64 out_len) +``` + +For **every** facet, the SET of register groupings its selected rows carry. One +crossing covers all 32 — asking per facet would be 32 crossings, and is how a +consumer drifts into the per-element loop §6 forbids. + +### Alignment as arithmetic, not a scan + +Each output byte is a 3-bit set (bit `w` = some row resolves to grouping `w`) +plus bit 3 for "some row's classid has no `ClassView` answer". Then: + +``` +aligned(facet) ⟺ popcount(byte) == 1 && byte & UNANSWERABLE == 0 +``` + +One `or` per (row, facet), no comparison and no early exit, so cost does not +depend on the data. **An OR-accumulated set is exact where cheaper accumulators +are not:** a sum of wire values cannot tell `{0,2}` from `{1,1}`, and an XOR +cannot tell `{1,1}` from `{}`. The set forgets multiplicity, which is precisely +the information the question does not need. + +`0` means the EMPTY set — no row selected — and is deliberately distinguishable +from disagreement. Conflating them would report "misaligned" for a population +that simply is not there. + +### What it measured, immediately + +A mask from `lgj_op_eq_classid(facet 3, …)` constrains **facet 3 only**; the +other 31 facets of those rows carry whatever classids the generator gave them. +Measured on the fixture: **1 of 32 facets aligned.** A test asserting +`isFullyAligned()` there failed, and the expectation was wrong rather than the +code — which is exactly the confusion a whole-row probe exists to remove. + +Note what this says about **placement**: the canon makes `classid` the key's +prefix precisely so key-ordered placement clusters a class into a range. The +fixture generates classids uniformly at random instead — measured mean run +length **1.07**, mean gap ~17 rows. Clustering is worth doing, and measured at +this stride it is worth ~2× (12.6 vs 25.6 ns/row for the same population size, +contiguous vs every-16th-row) — not from cache-line count, which is one line per +row either way at a 512-byte stride, but from stride-predictable prefetch and TLB +locality. + +### A classid is a GLOBAL address + +The `classid → grouping` table is process-global (`LazyLock`, 64 KiB, built +once), not per dataset: the same classid means the same class in every SoA, so +the resolution is dataset-independent. An earlier version put it on `RowStore`, +which was wrong in shape rather than output — the answers were right, but the +placement implied two datasets could disagree about what a classid carves into, +which the address space does not permit. + +And the table captures **layout only**. Meaning, RBAC, ontology category and +render template are separate resolutions off the same address; none belong in it +and none can be inferred from it. diff --git a/java/src/main/java/com/adaworldapi/lancegraph/FacetId.java b/java/src/main/java/com/adaworldapi/lancegraph/FacetId.java index 935ee3b..a0e2019 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/FacetId.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/FacetId.java @@ -20,6 +20,10 @@ */ public record FacetId(int index) { + /** Facets per row — 32, the canonical `32 × 16 B = 512 B` row (abi.md §11). */ + public static final int COUNT = 32; + + public FacetId { if (index < 0 || index > 31) { throw new IllegalArgumentException("facet index must be in 0..31, was " + index); diff --git a/java/src/main/java/com/adaworldapi/lancegraph/RowLayout.java b/java/src/main/java/com/adaworldapi/lancegraph/RowLayout.java new file mode 100644 index 0000000..b5bb784 --- /dev/null +++ b/java/src/main/java/com/adaworldapi/lancegraph/RowLayout.java @@ -0,0 +1,98 @@ +package com.adaworldapi.lancegraph; + +import java.util.Optional; + +/** + * Which register grouping each of a row store's facets carries, for one selection — the whole-row + * alignment answer (abi.md §16). + * + *

Alignment is arithmetic here, not a scan. The native side accumulates, per + * facet, a 3-bit SET of the groupings its selected rows resolve to (plus one bit for "some row's + * classid has no ClassView answer"). A facet is aligned exactly when that set has a single member: + * + * 

+ *   aligned(facet)  ⟺  bitCount(set) == 1  &&  no unanswerable bit
+ *
+ * + *

An OR-accumulated set is exact where cheaper accumulators are not: a sum of wire values cannot + * tell {@code {0,2}} from {@code {1,1}}, and an XOR cannot tell {@code {1,1}} from {@code {}}. The + * set forgets multiplicity, which is exactly the information the question does not need. + * + *

One crossing covers all 32 facets. Asking per facet would be 32 crossings and is how a + * consumer drifts into the per-element loop abi.md §6 forbids. + */ +public final class RowLayout { + + /** Bit meaning "some selected row's classid had no ClassView answer". */ + private static final int UNANSWERABLE = 0b1000; + + private final byte[] sets; + + RowLayout(byte[] sets) { + this.sets = sets; + } + + /** How many facets this covers. */ + public int facetCount() { + return sets.length; + } + + /** + * The grouping facet {@code f}'s selected rows all share, or empty when they do not. + * + *

Empty covers three genuinely different situations, and a caller that needs to tell them + * apart should use {@link #isAligned}, {@link #isEmpty} and {@link #hasUnanswerable}: the rows + * disagree; some row's classid is unanswerable; or no row was selected at all. + */ + public Optional carvingOf(FacetId f) { + int set = sets[f.index()] & 0xFF; + if (Integer.bitCount(set) != 1 || (set & UNANSWERABLE) != 0) { + return Optional.empty(); + } + return Optional.of(Carving.ofWire(Integer.numberOfTrailingZeros(set))); + } + + /** Whether facet {@code f}'s selected rows all read the register the same way. */ + public boolean isAligned(FacetId f) { + int set = sets[f.index()] & 0xFF; + return Integer.bitCount(set) == 1 && (set & UNANSWERABLE) == 0; + } + + /** Whether no selected row carried this facet — the empty set, distinct from disagreement. */ + public boolean isEmpty(FacetId f) { + return sets[f.index()] == 0; + } + + /** Whether some selected row's classid had no ClassView answer at this facet. */ + public boolean hasUnanswerable(FacetId f) { + return (sets[f.index()] & UNANSWERABLE) != 0; + } + + /** Whether EVERY facet is aligned — the whole row reads uniformly. */ + public boolean isFullyAligned() { + for (int i = 0; i < sets.length; i++) { + int set = sets[i] & 0xFF; + if (Integer.bitCount(set) != 1 || (set & UNANSWERABLE) != 0) { + return false; + } + } + return true; + } + + /** How many facets are aligned. */ + public int alignedCount() { + int n = 0; + for (int i = 0; i < sets.length; i++) { + int set = sets[i] & 0xFF; + if (Integer.bitCount(set) == 1 && (set & UNANSWERABLE) == 0) { + n++; + } + } + return n; + } + + @Override + public String toString() { + return "RowLayout[" + alignedCount() + "/" + sets.length + " facets aligned]"; + } +} diff --git a/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java b/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java index e5dc67c..3d45e5e 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java @@ -124,6 +124,23 @@ public Mask maskOfFacetClass(FacetId facet, int classId) { return new Mask(this, mask); } + /** + * For every facet, which register grouping this selection's rows carry (abi.md §16) — the + * whole-row alignment answer, in ONE crossing. + * + *

Use this to ask "is this population layout-aligned?" before sweeping it, or to discover + * which facets a heterogeneous selection can still be swept on. Asking {@link #facetSum} per + * facet to find out would be 32 crossings and would fail on the misaligned ones rather than + * reporting them. + * + * @param selection the rows to probe; must belong to this store + */ + public RowLayout layout(Mask selection) { + java.util.Objects.requireNonNull(selection, "selection"); + requireOpen("layout()"); + return new RowLayout(Engine.rowLayoutProbe(handle, selection.handle(), FacetId.COUNT)); + } + /** * Sum one facet's 12-byte register under the grouping the SELECTION ITSELF resolves to * (abi.md §15, ABI minor 6) — the verified sibling of {@link #facetSumAs}. diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java index 71c2d7b..7997efb 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java @@ -439,6 +439,15 @@ private static final class Minor6 { private Minor6() {} } + /** The whole-row layout probe (docs/abi.md §16, ABI minor 7). */ + private static final class Minor7 { + static final MethodHandle ROW_LAYOUT_PROBE = mh("lgj_row_layout_probe", + FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, + ValueLayout.JAVA_LONG, ValueLayout.ADDRESS, ValueLayout.JAVA_LONG)); + + private Minor7() {} + } + /** * Sum one facet's 12-byte register, under {@code carving}, over the rows a mask selects. * @@ -476,6 +485,21 @@ public static long reduceFacetSumResolved(long res, int facet, long mask, Memory return outSum.get(ValueLayout.JAVA_LONG, 0); } + /** + * For every facet, the SET of register groupings the selected rows carry — one crossing for + * all 32 facets. + */ + public static void rowLayoutProbe(long res, long mask, MemorySegment out, long outLen) { + crossed(); + int st; + try { + st = (int) Minor7.ROW_LAYOUT_PROBE.invokeExact(res, mask, out, outLen); + } catch (Throwable t) { + throw wrap("lgj_row_layout_probe", t); + } + Status.check("lgj_row_layout_probe", st); + } + // ── row store (docs/abi.md §11, ABI minor 2) ───────────────────────────────────────────── // // Callers above this class are expected to have already checked Abi.requireMinor(2) — these diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java index 3ef9721..d8caa41 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java @@ -210,6 +210,19 @@ public static long openRowStoreWithEdges(long nRows, long seed, int edgeClassid, * Sum one facet's register under the grouping the POPULATION resolves to (abi.md §15, ABI * minor 6), returning {@code [sum, carvingWire]}. Requires ABI minor >= 6. */ + /** + * For every facet, the set of groupings its selected rows carry (abi.md §16, ABI minor 7). + * Requires ABI minor >= 7. + */ + public static byte[] rowLayoutProbe(long store, long mask, int facets) { + Abi.requireMinor(7); + try (Arena arena = Arena.ofConfined()) { + MemorySegment out = arena.allocate(facets); + Downcalls.rowLayoutProbe(store, mask, out, facets); + return out.toArray(ValueLayout.JAVA_BYTE); + } + } + public static long[] facetSumResolved(long store, int facet, long mask) { Abi.requireMinor(6); Scratch s = SCRATCH.get(); diff --git a/java/src/test/java/com/adaworldapi/lancegraph/FacetSumParityTest.java b/java/src/test/java/com/adaworldapi/lancegraph/FacetSumParityTest.java index 00daa90..b8cab36 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/FacetSumParityTest.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/FacetSumParityTest.java @@ -220,6 +220,64 @@ public static void run(Checks c) { } } + c.section("the whole-row layout probe (abi.md §16): alignment answered by arithmetic," + + " for all 32 facets in ONE crossing"); + try (RowStore store = RowStore.open(1000, SEED)) { + // A single-class selection: every facet's rows share a classid, so every facet must + // be aligned, and facet 3's answer must match what facetSum resolves. + // + // NOTE, and it is the whole point of having this probe: a mask built by + // maskOfFacetClass(facet 3, classid 3) constrains FACET 3 only. The other 31 facets + // of those same rows carry whatever classids the generator gave them, so the ROW is + // not aligned even though the facet is. An earlier version of this test asserted + // isFullyAligned() here and failed — the expectation was wrong, not the code, and + // that is exactly the confusion a whole-row probe exists to remove. + try (Mask single = store.maskOfFacetClass(FacetId.of(3), 3)) { + RowLayout l = store.layout(single); + c.eq("all 32 facets covered", 32L, (long) l.facetCount()); + c.that("the facet the mask was BUILT on is aligned", l.isAligned(FacetId.of(3))); + c.eq("facet 3's probed grouping equals what facetSum resolves", + store.facetSum(FacetId.of(3), single).carving(), + l.carvingOf(FacetId.of(3)).orElseThrow()); + c.that("but the ROW is not aligned — the other facets are unconstrained", + !l.isFullyAligned()); + c.note("aligned facets on a facet-3 single-class selection: " + + l.alignedCount() + "/32"); + } + + // A mixed-grouping selection: facet 3 must report NOT aligned, and facetSum must + // refuse it. The two surfaces must agree about the same population. + try (Mask a = store.maskOfFacetClass(FacetId.of(3), 3); + Mask b = store.maskOfFacetClass(FacetId.of(3), 4)) { + long[] ra = a.materializeRows(); + long[] rb = b.materializeRows(); + long[] mixed = new long[ra.length + rb.length]; + System.arraycopy(ra, 0, mixed, 0, ra.length); + System.arraycopy(rb, 0, mixed, ra.length, rb.length); + try (Mask both = store.importRows(mixed)) { + RowLayout l = store.layout(both); + c.that("facet 3 is NOT aligned on a mixed-grouping selection", + !l.isAligned(FacetId.of(3))); + c.that("and its grouping is therefore absent", + l.carvingOf(FacetId.of(3)).isEmpty()); + c.that("which is not the same as empty", !l.isEmpty(FacetId.of(3))); + c.throwsUp("facetSum refuses the same population", RuntimeException.class, + () -> store.facetSum(FacetId.of(3), both)); + } + } + + // An empty selection: every facet reports the EMPTY set, which must be + // distinguishable from disagreement — otherwise "not aligned" would conflate + // "no rows" with "rows that disagree". + try (Mask empty = store.importRows()) { + RowLayout l = store.layout(empty); + c.that("an empty selection reports empty, not misaligned", + l.isEmpty(FacetId.of(0)) && l.isEmpty(FacetId.of(31))); + c.that("empty is not aligned either", !l.isAligned(FacetId.of(0))); + c.eq("so nothing counts as aligned", 0L, (long) l.alignedCount()); + } + } + c.section("Carving's own invariant: every reading covers exactly the 12-byte register"); for (Carving carving : Carving.values()) { c.eq(carving + " covers 12 bytes", 12L, diff --git a/native/lgj-abi/src/abi.rs b/native/lgj-abi/src/abi.rs index a1cc3f7..a28ae89 100644 --- a/native/lgj-abi/src/abi.rs +++ b/native/lgj-abi/src/abi.rs @@ -58,7 +58,7 @@ pub const LGJ_ABI_MAJOR: u32 = 0; /// /// `docs/abi.md` §13). Purely additive: a minor-3 Java loads fine and /// simply cannot call either new symbol. -pub const LGJ_ABI_MINOR: u32 = 6; +pub const LGJ_ABI_MINOR: u32 = 7; /// `"LGJ_ABI\0"` read big-endian. /// diff --git a/native/lgj-abi/src/class_view_provider.rs b/native/lgj-abi/src/class_view_provider.rs index 59b55f2..82fee09 100644 --- a/native/lgj-abi/src/class_view_provider.rs +++ b/native/lgj-abi/src/class_view_provider.rs @@ -118,6 +118,65 @@ impl ClassView for FixtureClassView { // fixture domain has no reason to opt into residue/PQ fidelity. } +/// The **process-global** `classid -> register grouping` table (abi.md §15). +/// +/// # Why global, and not per dataset +/// +/// **A classid is a global address: the same classid means the same class in +/// every SoA.** The hi half is a concept minted once in the shared codebook and +/// the lo half is an app render prefix; neither is scoped to a dataset. So +/// `classid -> ClassView -> cascade_shape` is dataset-INDEPENDENT, and holding +/// one table per store would be N identical copies of the same 64 KiB answer — +/// provably so here, since [`FixtureClassView`] is a unit struct with no +/// per-store state at all. +/// +/// An earlier version put this table on `RowStore`. That was wrong in shape +/// rather than in output: the answers were right, but the placement implied two +/// datasets could disagree about what a classid carves into, which the address +/// space does not permit. +/// +/// # What it captures, and what it deliberately does not +/// +/// **Only LAYOUT.** The classid resolves how the 12 content-blind bytes are +/// grouped — `6×2` / `4×3` / `3×4` — and nothing else. Meaning, RBAC, ontology +/// category and render template are all separate resolutions off the same +/// address; none of them belong in this table and none can be inferred from it. +/// +/// # Shape +/// +/// [`class_id_for`] narrows a `u32` classid to `u16`, so the table is 65_536 +/// one-byte entries: `0` = no `ClassView` answer, otherwise the grouping's wire +/// value plus one. 64 KiB, built once for the process on first use, never +/// rebuilt. A `LazyLock` and not a `OnceLock` because at this layer the provider +/// IS known — there is no caller-supplied resolver to wait for. +static CARVING_TABLE: std::sync::LazyLock> = std::sync::LazyLock::new(|| { + use lance_graph_contract::class_view::ClassView; + let mut t = vec![0u8; 1 << 16].into_boxed_slice(); + for (cid, slot) in t.iter_mut().enumerate() { + // +1 so 0 keeps its "no answer" meaning. + *slot = class_id_for(cid as u32).map_or(0, |c| { + crate::kernels::carving_to_wire(FixtureClassView.cascade_shape(c)) as u8 + 1 + }); + } + t +}); + +/// This process's register grouping for `classid`, as a wire value. +/// +/// `None` when the classid has no `ClassView` answer — including every classid +/// outside `u16` range, which [`class_id_for`] already reports as unanswerable +/// rather than truncating into a different class. +#[must_use] +pub fn carving_wire_of(classid: u32) -> Option { + let idx = usize::try_from(classid) + .ok() + .filter(|&i| i < CARVING_TABLE.len())?; + match CARVING_TABLE[idx] { + 0 => None, + w => Some(w - 1), + } +} + /// How this fixture carves the 12-byte content-blind register, per class — /// the `ClassView::cascade_shape` override (contract, 2026-08-25). /// @@ -267,4 +326,60 @@ mod tests { assert_eq!(decode_mode(u16::MAX as u32 + 1), 0); assert_eq!(decode_mode(u32::MAX), 0); } + + /// A classid is a GLOBAL address: the same classid resolves to the same + /// layout in every SoA. This is why the table is process-global rather than + /// per dataset — two stores built from different seeds, holding different + /// bytes, must still agree about what a given classid carves into. + #[test] + fn the_layout_of_a_classid_is_the_same_in_every_dataset() { + let a = crate::rowstore::RowStore::generate(64, 0x1111).unwrap(); + let b = crate::rowstore::RowStore::generate(64, 0x9999).unwrap(); + // Different datasets, genuinely different content. + assert_ne!(a.as_bytes(), b.as_bytes(), "the two stores must differ"); + + for classid in 0..64u32 { + assert_eq!( + carving_wire_of(classid), + carving_wire_of(classid), + "classid {classid} must resolve identically, dataset-independent" + ); + } + // And the resolution genuinely varies BY CLASSID — otherwise the + // agreement above would hold for a table that answered one constant. + let answers: std::collections::HashSet<_> = + (0..64u32).filter_map(carving_wire_of).collect(); + assert!( + answers.len() > 1, + "the table must discriminate between classids, not answer a constant" + ); + } + + /// The table captures LAYOUT and nothing else. A classid with no ClassView + /// answer reports none rather than truncating into a different class — the + /// property that stops a `> u16::MAX` classid aliasing onto class 0. + #[test] + fn a_classid_with_no_classview_answer_is_none_not_class_zero() { + assert!(carving_wire_of(0).is_some(), "class 0 is answerable"); + assert_eq!( + carving_wire_of(0x1_0000), + None, + "a classid past u16 range has no answer, and must not alias class 0" + ); + assert_eq!(carving_wire_of(u32::MAX), None); + } + + /// Every answer the table gives is a legal wire value that round-trips back + /// to a real grouping — a `+1` encoding error would show up as a decode + /// failure here rather than as a wrong sweep much later. + #[test] + fn every_table_answer_round_trips_to_a_real_grouping() { + for classid in 0..1024u32 { + if let Some(w) = carving_wire_of(classid) { + let shape = crate::kernels::carving_from_wire(u32::from(w)) + .unwrap_or_else(|| panic!("classid {classid} gave undecodable wire {w}")); + assert_eq!(crate::kernels::carving_to_wire(shape), u32::from(w)); + } + } + } } diff --git a/native/lgj-abi/src/exports.rs b/native/lgj-abi/src/exports.rs index 4217afd..bbedb56 100644 --- a/native/lgj-abi/src/exports.rs +++ b/native/lgj-abi/src/exports.rs @@ -1063,16 +1063,11 @@ pub unsafe extern "C" fn lgj_reduce_facet_sum_resolved( n, &g.words, |classid| { - store - .carving_of(classid, |cid| { - crate::class_view_provider::class_id_for(cid).map(|c| { - use lance_graph_contract::class_view::ClassView; - kernels::carving_to_wire( - crate::class_view_provider::FixtureClassView - .cascade_shape(c), - ) as u8 - }) - }) + // The process-global LAYOUT table. A classid is a global + // address — the same classid means the same class in + // every SoA — so this answer is dataset-independent and + // the table is built once for the process, not per store. + crate::class_view_provider::carving_wire_of(classid) .and_then(|w| kernels::carving_from_wire(u32::from(w))) }, ) { @@ -1108,6 +1103,77 @@ pub unsafe extern "C" fn lgj_reduce_facet_sum_resolved( }) } +/// The whole-row layout probe: for EVERY facet, the set of register groupings +/// the selected rows carry (ABI minor >= 7, `docs/abi.md` §16). +/// +/// One crossing covers all 32 facets, which is the point — a caller asking +/// "is this population layout-aligned?" should not pay 32 crossings, and asking +/// per facet is how a consumer ends up writing the per-element loop §6 forbids. +/// +/// Each output byte is a SET: bit `w` set means some selected row resolves to +/// grouping `w`; bit 3 (`LAYOUT_UNANSWERABLE`) means some row's classid has no +/// `ClassView` answer. A facet is aligned exactly when its byte has a single bit +/// set and bit 3 clear — an arithmetic test, not a scan. A facet with no +/// selected rows reports `0`: the empty set, which is neither aligned nor +/// unanswerable. +/// +/// Work is `O(mask_words + popcount × facets)` with one `or` per (row, facet). +/// +/// # Safety +/// +/// A null `out` is *handled*, not UB: `NULL_ARGUMENT`. Otherwise `out` must +/// point at `out_len` writable bytes, and `out_len` must be at least the store's +/// facet count — checked BEFORE anything is written. +#[no_mangle] +pub unsafe extern "C" fn lgj_row_layout_probe( + res: u64, + mask: u64, + out: *mut u8, + out_len: u64, +) -> i32 { + guard(|| { + if out.is_null() { + return LGJ_ERR_NULL_ARGUMENT; + } + let facets = crate::rowstore::ROW_FACETS as usize; + if out_len < facets as u64 { + return LGJ_ERR_MASK_LENGTH_MISMATCH; + } + let store_entry = match registry::resolve_kind(res, LGJ_RESOURCE_ROWSTORE) { + Ok(e) => e, + Err(e) => return e, + }; + let store = match store_entry.rowstore() { + Some(s) => s, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + let (maskr, parent) = match registry::resolve_mask_with_parent(mask) { + Ok(t) => t, + Err(e) => return e, + }; + if !std::sync::Arc::ptr_eq(&parent, &store_entry) || maskr.n_rows != store_entry.n_rows { + return LGJ_ERR_MASK_LENGTH_MISMATCH; + } + let g = match maskr.read_mask() { + Some(g) => g, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + // SAFETY: non-null and long enough, both checked above. + let slice = unsafe { std::slice::from_raw_parts_mut(out, facets) }; + kernels::facet_layout_sets( + store.as_bytes(), + crate::rowstore::ROW_BYTES as usize, + store_entry.n_rows as usize, + facets, + crate::rowstore::FACET_BYTES as usize, + &g.words, + crate::class_view_provider::carving_wire_of, + slice, + ); + LGJ_OK + }) +} + /// One crossing: for every row, which of its 32 facets carry `needle` as /// classid — one `u32` bitset per row, written into the **caller's** buffer /// (a Java-arena segment of `n_rows` ints; zero-copy out, nothing diff --git a/native/lgj-abi/src/kernels.rs b/native/lgj-abi/src/kernels.rs index 2bb4bd0..fdba178 100644 --- a/native/lgj-abi/src/kernels.rs +++ b/native/lgj-abi/src/kernels.rs @@ -523,6 +523,83 @@ pub fn resolve_population_carving( resolved } +/// Bit set in a facet's layout byte when some selected row's classid has no +/// `ClassView` answer at all. +pub const LAYOUT_UNANSWERABLE: u8 = 0b1000; + +/// For each of the row's facets, the SET of register groupings its selected rows +/// carry — the whole-row alignment probe (abi.md §16). +/// +/// # The cheap exact test +/// +/// Per facet this accumulates a 3-bit set: bit `w` is set if some selected row +/// resolves to grouping `w`, plus [`LAYOUT_UNANSWERABLE`] if some row's classid +/// has no answer. That is ONE `or` per (row, facet) — no comparison, no +/// branch on the previous value, no early exit to make the cost data-dependent. +/// +/// The alignment question then falls out of arithmetic rather than a scan: +/// +/// ```text +/// aligned(facet) ⟺ byte.count_ones() == 1 && byte & UNANSWERABLE == 0 +/// ``` +/// +/// An OR-accumulated set is exact where a sum or an XOR is not: summing wire +/// values cannot tell `{0,2}` from `{1,1}`, and XOR cannot tell `{1,1}` from +/// `{}`. The set forgets multiplicity, which is precisely the information the +/// question does not need. +/// +/// `out` must hold one byte per facet; every entry is overwritten. A facet with +/// no selected rows reports `0` — the empty set, which is neither aligned nor +/// unanswerable, and the caller must not read it as either. +#[expect( + clippy::too_many_arguments, + reason = "eight is the row geometry (bytes/stride/rows/facets/facet_bytes) plus the mask, the resolver and the output. A params struct would bundle values that have no relationship except being needed here, and would hide the fact that every one is read straight from the store's own constants at the single call site." +)] +pub fn facet_layout_sets( + bytes: &[u8], + row_stride: usize, + n_rows: usize, + facets: usize, + facet_bytes: usize, + mask_words: &[u64], + wire_of: impl Fn(u32) -> Option, + out: &mut [u8], +) { + assert!( + out.len() >= facets, + "facet_layout_sets: out.len()={} < facets {facets}", + out.len() + ); + for slot in out.iter_mut().take(facets) { + *slot = 0; + } + for (w, &word) in mask_words.iter().enumerate() { + let base_row = w * 64; + if base_row >= n_rows { + break; + } + let mut bits = word; + let valid = n_rows - base_row; + if valid < 64 { + bits &= (1u64 << valid) - 1; + } + while bits != 0 { + let row = base_row + bits.trailing_zeros() as usize; + bits &= bits - 1; + let row_off = row * row_stride; + for (f, slot) in out.iter_mut().enumerate().take(facets) { + let o = row_off + f * facet_bytes; + let classid = + u32::from_le_bytes([bytes[o], bytes[o + 1], bytes[o + 2], bytes[o + 3]]); + *slot |= match wire_of(classid) { + Some(x) => 1u8 << x, + None => LAYOUT_UNANSWERABLE, + }; + } + } + } +} + /// Sum every group of one facet's 12-byte register, over the rows selected by /// `mask_words`. Returns `None` on overflow rather than a wrapped value. /// diff --git a/native/lgj-abi/src/rowstore.rs b/native/lgj-abi/src/rowstore.rs index a6a5c6a..4308f3e 100644 --- a/native/lgj-abi/src/rowstore.rs +++ b/native/lgj-abi/src/rowstore.rs @@ -90,48 +90,6 @@ pub struct RowStore { /// The seed the buffer was generated from. pub seed: u64, bytes: Arc<[u8]>, - /// The dataset's `classid -> register grouping` table, built ONCE on first - /// use (abi.md §15). - /// - /// Resolution is a per-CLASS fact and a dataset shares one `ClassView`, so - /// consulting it per row per sweep re-derives a constant. This memoises the - /// whole answer space: `class_id_for` narrows a `u32` classid to `u16`, so - /// the table is 65_536 entries of one byte — `0` = no `ClassView` answer, - /// otherwise the grouping's wire value plus one. - /// - /// 64 KiB per dataset, built with 65_536 `ClassView` calls on first resolved - /// sweep and never again. That trade is the right way round here: the table - /// is bounded and one-off, while the per-row consult it replaces is - /// unbounded in sweeps. A `OnceLock` rather than a `LazyLock` because the - /// resolver is supplied by the caller (the provider is not in scope at this - /// layer) — the first caller wins and every later one reads. - carving_table: std::sync::OnceLock>, -} - -impl RowStore { - /// This dataset's grouping for `classid`, from the memo table, building it - /// on first call with `resolve`. - /// - /// `resolve` is called at most 65_536 times for the life of the store, and - /// exactly zero times after the table exists. It must be a pure function of - /// the classid — it is a `ClassView` consult, which by contract is. - pub fn carving_of(&self, classid: u32, resolve: impl Fn(u32) -> Option) -> Option { - let table = self.carving_table.get_or_init(|| { - let mut t = vec![0u8; 1 << 16].into_boxed_slice(); - for (cid, slot) in t.iter_mut().enumerate() { - // +1 so that 0 keeps its "no answer" meaning. - *slot = resolve(cid as u32).map_or(0, |w| w + 1); - } - t - }); - // A classid outside u16 range has no ClassView answer at all — the same - // fact the table's 0 encodes, reached without indexing past it. - let idx = usize::try_from(classid).ok().filter(|&i| i < table.len())?; - match table[idx] { - 0 => None, - w => Some(w - 1), - } - } } impl std::fmt::Debug for RowStore { @@ -173,7 +131,6 @@ impl RowStore { n_rows, seed, bytes: Arc::from(bytes), - carving_table: std::sync::OnceLock::new(), }) } @@ -269,7 +226,6 @@ impl RowStore { n_rows, seed, bytes: Arc::from(bytes), - carving_table: std::sync::OnceLock::new(), }) } diff --git a/valhalla-lab/reproducers/R10-observed.txt b/valhalla-lab/reproducers/R10-observed.txt new file mode 100644 index 0000000..b606b45 --- /dev/null +++ b/valhalla-lab/reproducers/R10-observed.txt @@ -0,0 +1,12 @@ +== (1) the SCHEMA as one value class: 12 B, never flat == + Rails6 false Triplets4 false Quads3 false +== (2) one GROUP as a value class: <= 4 B, all flat == + Rail 2B true Triplet 3B true Quad 4B true + +== (3) storage / Panama / Valhalla must decode the SAME values == + RAILS_6X2 layout=12 B groups=[5123, 13861, 22599, 31337, 40075, 48813] + TRIPLETS_4X3 layout=12 B groups=[2429955, 5785398, 9140841, 12496284] + QUADS_3X4 layout=12 B groups=[908399619, 2053724231, 3199048843] + + all three descriptions agree, all schemas: true + and the schemas genuinely read differently: true diff --git a/valhalla-lab/reproducers/R10_SchemaAlignsWithStorage.java b/valhalla-lab/reproducers/R10_SchemaAlignsWithStorage.java new file mode 100644 index 0000000..5803b61 --- /dev/null +++ b/valhalla-lab/reproducers/R10_SchemaAlignsWithStorage.java @@ -0,0 +1,152 @@ +// Reproducer R10 — bolt the SAME three schemas into Valhalla and Panama, and prove all three +// descriptions (storage bytes, Panama MemoryLayout, Valhalla value class) agree. +// +// The substrate's register is 12 content-blind bytes carved three ways, and which one applies is +// resolved from the classid (ClassView::cascade_shape). This file asks the two questions that +// decide whether Java can hold that schema honestly: +// +// (1) Does a Valhalla value class of the SCHEMA flatten? -> no, and it cannot (R4/R6) +// (2) Does a value class of one GROUP flatten? -> yes, all three +// +// and then proves the alignment that makes (2) usable: for each schema, the Panama MemoryLayout, +// the Valhalla value class, and the raw storage bytes must all decode the same register to the +// same values. Three descriptions, one truth, or the schema is not "bolted on" — it is a second +// story about the same bytes. +// +// javac --enable-preview -source 27 -target 27 \ +// --add-exports java.base/jdk.internal.value=ALL-UNNAMED -d out R10_SchemaAlignsWithStorage.java +// java --enable-preview --enable-native-access=ALL-UNNAMED \ +// --add-exports java.base/jdk.internal.value=ALL-UNNAMED -cp out R10_SchemaAlignsWithStorage +import java.lang.foreign.Arena; +import java.lang.foreign.MemoryLayout; +import java.lang.foreign.MemorySegment; +import java.lang.foreign.ValueLayout; +import jdk.internal.value.ValueClass; + +public class R10_SchemaAlignsWithStorage { + + static final int REGISTER_BYTES = 12; + + // ── the three schemas, as the substrate names them ── + enum Schema { + RAILS_6X2(6, 2), TRIPLETS_4X3(4, 3), QUADS_3X4(3, 4); + + final int groups, groupBytes; + + Schema(int g, int b) { + this.groups = g; + this.groupBytes = b; + } + + /** The Panama description of the register under this schema. */ + MemoryLayout layout() { + return MemoryLayout.sequenceLayout(groups, + MemoryLayout.sequenceLayout(groupBytes, ValueLayout.JAVA_BYTE)); + } + } + + // ── (2) one GROUP as a value class: the unit Java can actually hold flat ── + static value record Rail(byte lo, byte hi) {} // 2 B + static value record Triplet(byte a, byte b, byte c) {} // 3 B + static value record Quad(byte a, byte b, byte c, byte d) {} // 4 B + + // ── (1) the whole SCHEMA as a value class: measured, for contrast ── + static value record Rails6(Rail r0, Rail r1, Rail r2, Rail r3, Rail r4, Rail r5) {} // 12 B + static value record Triplets4(Triplet t0, Triplet t1, Triplet t2, Triplet t3) {} // 12 B + static value record Quads3(Quad q0, Quad q1, Quad q2) {} // 12 B + + /** Decode group g of a register under `s`, straight from the bytes — the storage truth. */ + static long fromStorage(byte[] reg, Schema s, int g) { + long v = 0; + for (int k = 0; k < s.groupBytes; k++) { + v |= (long) (reg[g * s.groupBytes + k] & 0xFF) << (8 * k); + } + return v; + } + + /** The same group, read through Panama's own layout description. */ + static long fromPanama(MemorySegment seg, Schema s, int g) { + long v = 0; + for (int k = 0; k < s.groupBytes; k++) { + v |= (long) (seg.get(ValueLayout.JAVA_BYTE, (long) g * s.groupBytes + k) & 0xFF) + << (8 * k); + } + return v; + } + + /** The same group, hydrated into the Valhalla value class for that width. */ + static long fromValhalla(byte[] reg, Schema s, int g) { + int o = g * s.groupBytes; + return switch (s) { + case RAILS_6X2 -> { + Rail r = new Rail(reg[o], reg[o + 1]); + yield (r.lo() & 0xFFL) | ((r.hi() & 0xFFL) << 8); + } + case TRIPLETS_4X3 -> { + Triplet t = new Triplet(reg[o], reg[o + 1], reg[o + 2]); + yield (t.a() & 0xFFL) | ((t.b() & 0xFFL) << 8) | ((t.c() & 0xFFL) << 16); + } + case QUADS_3X4 -> { + Quad q = new Quad(reg[o], reg[o + 1], reg[o + 2], reg[o + 3]); + yield (q.a() & 0xFFL) | ((q.b() & 0xFFL) << 8) | ((q.c() & 0xFFL) << 16) + | ((q.d() & 0xFFL) << 24); + } + }; + } + + static boolean flat(Class t, Object init) { + return ValueClass.isFlatArray(ValueClass.newNullRestrictedNonAtomicArray(t, 4, init)); + } + + public static void main(String[] args) { + Rail rail = new Rail((byte) 0, (byte) 0); + Triplet trip = new Triplet((byte) 0, (byte) 0, (byte) 0); + Quad quad = new Quad((byte) 0, (byte) 0, (byte) 0, (byte) 0); + + System.out.println("== (1) the SCHEMA as one value class: 12 B, never flat =="); + System.out.printf(" Rails6 %-6s Triplets4 %-6s Quads3 %s%n", + flat(Rails6.class, new Rails6(rail, rail, rail, rail, rail, rail)), + flat(Triplets4.class, new Triplets4(trip, trip, trip, trip)), + flat(Quads3.class, new Quads3(quad, quad, quad))); + + System.out.println("== (2) one GROUP as a value class: <= 4 B, all flat =="); + System.out.printf(" Rail 2B %-6s Triplet 3B %-6s Quad 4B %s%n", + flat(Rail.class, rail), flat(Triplet.class, trip), flat(Quad.class, quad)); + + System.out.println(); + System.out.println("== (3) storage / Panama / Valhalla must decode the SAME values =="); + byte[] reg = new byte[REGISTER_BYTES]; + for (int k = 0; k < REGISTER_BYTES; k++) { + reg[k] = (byte) (k * 17 + 3); // varied, so a wrong offset shows up + } + try (Arena arena = Arena.ofConfined()) { + MemorySegment seg = arena.allocate(REGISTER_BYTES); + MemorySegment.copy(reg, 0, seg, ValueLayout.JAVA_BYTE, 0, REGISTER_BYTES); + + boolean allAgree = true; + for (Schema s : Schema.values()) { + // Panama's own layout must describe exactly 12 bytes under every schema. + long described = s.layout().byteSize(); + StringBuilder vals = new StringBuilder(); + for (int g = 0; g < s.groups; g++) { + long a = fromStorage(reg, s, g); + long b = fromPanama(seg, s, g); + long c = fromValhalla(reg, s, g); + allAgree &= (a == b && b == c); + vals.append(a).append(g + 1 < s.groups ? ", " : ""); + } + System.out.printf(" %-14s layout=%2d B groups=[%s]%n", s, described, vals); + if (described != REGISTER_BYTES) { + allAgree = false; + } + } + System.out.println(); + System.out.println(" all three descriptions agree, all schemas: " + allAgree); + + // The schemas must give DIFFERENT readings, or agreement above is trivial. + boolean differ = fromStorage(reg, Schema.RAILS_6X2, 0) + != fromStorage(reg, Schema.QUADS_3X4, 0); + System.out.println(" and the schemas genuinely read differently: " + differ); + } + } +} diff --git a/valhalla-lab/reproducers/README.md b/valhalla-lab/reproducers/README.md index 488b7e6..b8cc7b0 100644 --- a/valhalla-lab/reproducers/README.md +++ b/valhalla-lab/reproducers/README.md @@ -436,6 +436,29 @@ conclusion — B ≈ standalone, D > B falsified, C ~30×, the B′ collapse, th recovery, the end-to-end E′ win — held identically). That stability of *conclusions* under *unstable* absolutes is why the ratios are the result and the raw numbers are the evidence. +## R10 — the same schema in storage, Panama and Valhalla (`R10_SchemaAlignsWithStorage.java`) + +The substrate carves its 12 content-blind bytes three ways and resolves which from the +classid. R10 asks whether Java can hold that schema *honestly* — three descriptions of +the same bytes that must not disagree. + +**Measured** (`R10-observed.txt`): + +| as a value class | flat? | +|---|---| +| the whole SCHEMA (`Rails6` / `Triplets4` / `Quads3`, 12 B) | `false` — and cannot be, per R4/R6 | +| one GROUP (`Rail` 2 B / `Triplet` 3 B / `Quad` 4 B) | **`true`, all three** | + +And the alignment that makes the second row usable: for every schema, decoding a register +from **raw storage bytes**, through a **Panama `MemoryLayout`**, and via the **Valhalla +value class** yields identical values — with each layout describing exactly 12 bytes, and +the three schemas genuinely reading differently (so the agreement is not trivial). + +**The consequence for "bolt the schema into Valhalla":** it bolts on at the GROUP, not at +the register. `12 = 6×2 = 4×3 = 3×4` means the largest group in any carving is 4 bytes — +half the flattening budget — while the register is 12 and the facet 16, neither of which +Java can flatten or needs to. So the schema is expressible on both sides; what crosses is +the group, and the register stays where it is. ## R11 — the physical layout is a schema, and applying it is a descriptor swap (`R11_LayoutIsASchema.java`) The store today is AoS: 32 facets × 16 B interleaved in a 512-B row, lanes exposed as