diff --git a/.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md b/.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md new file mode 100644 index 00000000..3cfa4420 --- /dev/null +++ b/.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md @@ -0,0 +1,476 @@ +# HHTL Substrate — GridLake Pre-Sprint Prompt (PR-X1 + PR-X2) + +> Date: 2026-05-19 (drafted post-PR #162 merge at master tip `c8f4af68`) +> Status: Pre-sprint kickoff prompt — slots before / inside PR-X10 in the +> 8-week Phase 2 substrate arc. Companion to +> `hhtl-substrate-execution-prompt.md` (W1-W8 master schedule). +> +> **Q-NEW-1 marker present** — the plan-review savant decides path (a) vs +> path (b) at preflight. See § "Schedule slot" below. + +## Why this exists — the column-substrate identity is load-bearing + +GridLake is the in-process materialisation of the Lance column. The +`MultiLaneColumn` carrier (`Arc<[u8]>`-backed byte storage with typed +multi-width lane views) is not a convenience type; it is the substrate +identity from PR #162 made concrete at the Rust ownership layer. + +**Verbatim, from `.claude/knowledge/stack-consolidation-bardioc-to-hhtl.md` +§ "Column-substrate identity — Lance ≡ Arrow ≡ ndarray SoA"** (master +commit `c8f4af68`, lines 126-220): + +``` +Lance dataset (single physical store) + │ + ▼ +Lance column ≡ Arrow column buffer ≡ ndarray SoA + (one representation, all the way down) + │ + ├──→ lance-graph: XOR-cascade lookups, cognitive-shader cycles + │ (ndarray SIMD ops directly over the column bytes; + │ no copy, no serde, no marshal — the "in-RAM Thought" + │ IS the Lance column slot) + │ + ├──→ SurrealDB: SurrealQL parses → reads the same column + │ LIVE subscription = a watch on column-state predicates + │ + ├──→ sea-orm: SQL via Lance backend → reads the same column + │ (Zone-3 egress is materialise-once into PG-shape for the + │ legacy surface; the source bytes are unchanged) + │ + ├──→ Databend: analytic SQL → reads the same column + │ (ndarray::simd kernel swap → operates on the same bytes + │ the cognitive cascade just operated on) + │ + └──→ Tantivy: FTS index → built over the same column +``` + +> **One physical representation, end to end.** The Lance column layout, the +> Arrow column buffer layout, and the ndarray SoA layout are the same bytes +> viewed through three names. The four dialect surfaces (lance-graph cascade, +> SurrealDB, sea-orm, Databend, Tantivy) all parse their respective query +> languages down to operations on those same bytes. + +> **ndarray amortises the SIMD primitive across the whole stack.** The same +> kernel that runs the cognitive cascade, that Databend's filter pushdown +> invokes, that Tantivy's indexer reads, that sea-orm projects to legacy +> egress — they are the same kernel on the same bytes. ndarray pays for the +> SIMD primitive once and the entire stack collects rent. No transcode tier, +> no copy boundary, no format conversion at any zone. + +And the load-bearing summary (consolidation doc lines 211-215): + +> **The column IS the SoA IS the ndarray buffer.** The cognitive cascade, +> the analytic scan, the FTS index build, and the graph traversal all +> operate on the same bytes through the same SIMD kernels. ndarray::simd +> is the common substrate because the substrate is genuinely one thing, +> not four parallel things wearing the same uniform. + +`MultiLaneColumn` is what makes this literal in Rust — a single `Arc<[u8]>` +that the cascade reads as `F32x16` slices, that the codec reads as +`U64x8` slices, that the SurrealDB FFI reads as Arrow buffers, that +sea-orm projects to PG-shape rows. **Same bytes. Different lane width. +No copy.** If this primitive doesn't exist before PR-X4 / PR-X9 / PR-X12 +ship, those sprints either (a) reintroduce a per-consumer copy boundary +that the consolidation doc explicitly forbids, or (b) reach for raw +`std::slice::from_raw_parts` violating the W1a consumer contract. + +That is the gap this pre-sprint fills. + +## What GridLake is, mechanically + +```rust +// PR-X1: MultiLaneColumn carrier +pub struct MultiLaneColumn { + bytes: Arc<[u8]>, // 64-byte-aligned, padded to lane boundary + elem_count: usize, // logical length (pre-padding) + elem_width_bytes: u8, // 1, 2, 4, 8 (u8/u16/u32/u64/f32/f64) +} + +impl MultiLaneColumn { + pub fn iter_u8x64(&self) -> impl Iterator + '_ { ... } + pub fn iter_u16x32(&self) -> impl Iterator + '_ { ... } + pub fn iter_u32x16(&self) -> impl Iterator + '_ { ... } + pub fn iter_u64x8(&self) -> impl Iterator + '_ { ... } + pub fn iter_f32x16(&self) -> impl Iterator + '_ { ... } + pub fn iter_f64x8(&self) -> impl Iterator + '_ { ... } + pub fn iter_bf16x32(&self) -> impl Iterator + '_ { ... } + + pub fn len(&self) -> usize { self.elem_count } // logical, NOT byte-padded + pub fn byte_capacity(&self) -> usize { self.bytes.len() } // padded + pub fn as_bytes(&self) -> &[u8] { &self.bytes } // for Arrow / Lance FFI +} + +// PR-X2: #[soa(pad_to_lanes=N)] proc-macro derive +#[derive(SoA)] +#[soa(pad_to_lanes = 64)] // pad each column to multiple of 64 elements +struct Belief { + fingerprint: u64, + truth_f: f32, + truth_c: f32, + committed: bool, +} +// derives: BeliefSoa { fingerprint: MultiLaneColumn, truth_f: MultiLaneColumn, ... } +// plus per-column typed accessors and a `slice(range)` returning row-tuples +``` + +The `Arc<[u8]>` is the column. The lane iterators are the views. The +derive emits column-typed accessors at compile time (which is what the +"What survives — JITson / Cranelift, cleaner than before" passage in the +consolidation doc names as the load-bearing compile-time pipeline). + +## Schedule slot — Q-NEW-1 for plan-review savant + +The GridLake sprint is not currently in the 8-week schedule from +`hhtl-substrate-execution-prompt.md` (master commit `c8f4af68`, lines +76-91): + +| Week | Sprints | Workers | +|---|---|---| +| W1-W2 | PR-X10 (linalg-core foundation) | 12 (max fan-out: A1 → A2-A12) | +| W3 | PR-X11 (jc consolidation) + PR-X13 (OGIT bridge) | 6 + 4 | +| W4-W5 | PR-X12 (codec) + PR-X4 (splat cascade) | 8 + 5 | +| W6-W7 | PR-X9 (basin-codebook) | 6 | +| W8 | Integration + canary | 3 | + +PR-X1 and PR-X2 are needed before PR-X4 / PR-X9 / PR-X12's SIMD-staged +inner loops, because those consumers expect the typed column surface to +already exist. Two viable insertion points exist; the plan-review savant +must choose one at preflight: + +### Path (a): W2.5 prerequisite slot — clean but extends schedule + +GridLake lands as a dedicated 0.5-week sprint after PR-X10 merges and +before W3 spawns PR-X11 + PR-X13. The new schedule: + +| Week | Sprints | +|---|---| +| W1-W2 | PR-X10 (linalg-core, 12 workers) | +| **W2.5** | **PR-X1 + PR-X2 (GridLake, 4 workers, ~3 days)** | +| W3 | PR-X11 + PR-X13 | +| W4-W5 | PR-X12 + PR-X4 | +| W6-W7 | PR-X9 | +| W8 | Integration + canary | + +- ✅ Clean scope — GridLake's surface (`MultiLaneColumn` + `#[derive(SoA)]`) + is the entire PR, no entanglement with linalg primitives +- ✅ Locks `crate::simd::*` surface before X11 / X13 spawn +- ✅ Q-NEW-1 trade-off is visible to W3+ coordinators +- ❌ Extends the 8-week arc to 8.5 weeks +- ❌ Adds a coordinator + specialist savant cycle (extra ~0.5 day overhead) + +### Path (b): X10-A13/A14 absorption — cheaper but pollutes PR-X10 + +GridLake workers become A13 (MultiLaneColumn) and A14 (#[derive(SoA)]) +inside PR-X10. The PR-X10 sprint becomes 14 max-fan-out workers instead +of 12; A13/A14 spawn parallel with A2-A12 after A1 lands. + +- ✅ No schedule extension — fits inside W1-W2 +- ✅ Single coordinator, single specialist-savant preflight covering A1-A14 +- ❌ Pollutes PR-X10's scope (linalg primitives + storage carrier in one PR) +- ❌ A13/A14 don't share dependency surface with A2-A12 — they're a + different concern under the same coordinator +- ❌ Already exists tension: A6 absorbs heel_f64x8 distance kernels per + the merged salvage note; adding A13/A14 doubles the absorption load + on PR-X10's review surface + +**Recommendation (informational, plan-review savant decides):** Path (a). +The scope cleanliness matters more than 0.5 weeks of schedule when the +deliverable is a typed substrate surface every downstream sprint +consumes. Path (b) is acceptable if the W1-W2 window has slack after A1 +unblocks (rare — coordinators historically use the slack for tier-3 +optional workers). + +**Q-NEW-1**: choose path (a) or (b) before sprint kickoff. Recorded as an +open question for the plan-review savant; do not start workers until the +answer is committed to this doc. + +## Worker decomposition (4 workers, max fan-out) + +``` +A1 (sequential) — `simd_soa/multi_lane.rs` — MultiLaneColumn carrier + lane iterators +A2 (parallel) — `simd_soa_derive/` — #[derive(SoA)] proc-macro crate +A3 (parallel) — `simd_soa/bench/` — zero-copy / no-alloc bench harness +A4 (parallel) — `simd_soa/integration/` — splat3d + nars consumer probes +``` + +### A1 — `crate::simd_soa::MultiLaneColumn` + +Surface (mandatory): +- `MultiLaneColumn::new_from_slice(slice: &[T], pad_to_lanes: usize) -> Self` + — allocates one `Arc<[u8]>`, 64-byte-aligned, padded to lane-multiple +- `iter_u8x64`, `iter_u16x32`, `iter_u32x16`, `iter_u64x8`, `iter_f32x16`, + `iter_f64x8`, `iter_bf16x32` — each returns `impl Iterator + '_` +- `as_bytes(&self) -> &[u8]` — for Arrow / Lance FFI +- `len()` returns LOGICAL element count (pre-padding); `byte_capacity()` + returns padded byte count +- All lane iterators are zero-copy: no `Vec` allocation, no `collect`, + no temporary buffers — verified by A3 bench harness + +Backing the W1a consumer contract: +- All lane types are `crate::simd::*` re-exports (`U8x64`, `F32x16`, …) +- All three backends (AVX-512, AVX2, scalar) work because the underlying + types are already polyfilled +- Every `unsafe` block in `iter_*` carries a `// SAFETY:` comment that + cites the multiple-of-64 alignment invariant + lane-multiple + padding invariant + +Out of scope for A1 (defer to PR-X4 / PR-X9 / PR-X12 consumers): +- Sliced row-tuple iteration (the `#[derive(SoA)]` macro generates this) +- Cross-column joins or column-level math (consumers compose iterators) +- Lance dataset I/O (PR-X13 OGIT bridge owns FFI; GridLake exposes the + Arrow-compatible byte buffer, doesn't read or write Lance files) + +### A2 — `simd-soa-derive` proc-macro crate + +```rust +#[derive(SoA)] +#[soa(pad_to_lanes = N)] // N ∈ {8, 16, 32, 64}, default 64 +struct Belief { fingerprint: u64, truth_f: f32, truth_c: f32, committed: bool } +``` + +Generates: +- `struct BeliefSoa { fingerprint: MultiLaneColumn, truth_f: MultiLaneColumn, + truth_c: MultiLaneColumn, committed: MultiLaneColumn }` +- `impl BeliefSoa { pub fn from_aos(rows: &[Belief]) -> Self; pub fn len(&self) -> usize; }` +- `pub fn slice(&self, range: Range) -> BeliefSoaSlice<'_>` + returning a `&` to each underlying `MultiLaneColumn` over the range +- Per-column zero-copy accessors typed to the column's lane width + +Mandatory invariants: +- `#[soa(pad_to_lanes = N)]` MUST NOT change `BeliefSoa::len()` semantics — + `len()` always returns the logical row count, not the padded byte count + (gates Correctness.1 "bit-exact" — padded slots cannot leak into + consumer output) +- Compile error if a field type is not `Pod` or not `repr(C)` +- Compile error if `bool` columns are used without explicit byte-packing + hint (`#[soa(bool_repr = "byte" | "bitpack")]`) + +Out of scope for A2: +- Schema-evolution hooks (OGIT bridge / DeriveEntityModel in PR-X13) +- Cranelift JIT specialisation (named as "what survives" in PR #162 doc; + out of scope here, future work) + +### A3 — bench harness `simd_soa/bench/` + +Mandatory benches (criterion-based): +- `bench_no_alloc` — runs lane iterators over a 1M-row column, asserts + zero heap allocations under `dhat::Profiler` or equivalent +- `bench_f32x16_throughput` — measures `iter_f32x16` rate vs raw + `std::slice::chunks_exact(16)`; required ratio ≥ 0.98 (i.e., the + typed wrapper costs less than 2% vs raw slices) +- `bench_lane_width_swap` — same column, iterated as `iter_u64x8` then + as `iter_f64x8` in alternation; measures cache friendliness of + the `Arc<[u8]>`-as-shared-substrate model +- `bench_arc_clone_cost` — `MultiLaneColumn::clone()` (which clones the + `Arc`, not the bytes) must be O(1) — assert ≤ 50 ns + +These benches gate the W2.5 → W3 handoff. + +### A4 — integration probes `simd_soa/integration/` + +Probe 1 — splat3d backward compat: `splat3d::tile::TileBinning`'s +existing 16×16 fixed-tile arrays can be expressed as `MultiLaneColumn` +slices without changing splat3d's public API. Probe ships as a feature +flag (`simd-soa-splat3d-probe`), opt-in. + +Probe 2 — nars revise compat: `hpc::nars::Belief` (the canary's +operand type) can be `#[derive(SoA)]`-ified. Probe shows the SoA layout +delivers the same `revise()` output as the AoS layout, bit-exact, on +10,000 randomly-seeded inputs (mirrors the canary Correctness gate 1). + +Probe 3 — Arrow round-trip: A `MultiLaneColumn::as_bytes()` slice can +be reconstituted into an `arrow::array::PrimitiveArray` (zero-copy) +and back. Validates the "Lance column ≡ Arrow column buffer" claim at +the Rust type-system layer. + +## Forbidden (negative constraints) + +1. **Do NOT re-introduce L1 / L2 / L∞ distance kernels.** PR-X10 A6 + absorbs them under `crate::hpc::linalg::distance` per the merged + salvage note in `pr-x10-linalg-core-design.md` § "Distance kernels" + (master commit `c8f4af68`): + + > Absorbs the `heel_f64x8::l1/l2/linf` kernels from PR #160 (lance-graph) + > — the code is correct, the framing was wrong (it was filed as "Sprint + > 0a of a four-repo integration arc"; the right home is here, alongside + > polar / matfn in the linalg core). Bench parity vs the PR #160 + > implementation is part of the A6 acceptance gate, not a separate + > worker. + + If a GridLake SIMD-staged inner loop wants a distance, **call + `crate::hpc::linalg::distance::l2_f64_simd`** (which lands as part of + PR-X10 A6 in W2). Do not implement distance primitives in + `crate::simd_soa::*`. + +2. **Do NOT introduce a new SIMD type.** All lane types are + `crate::simd::*` re-exports. If GridLake's iterators need a lane + width that doesn't exist (unlikely — `crate::simd` covers + u8x64/u16x32/u32x16/u64x8/f32x16/f64x8/bf16x32 already), file an + extension proposal against `vertical-simd-consumer-contract.md` + before adding it. PR-X1 must not be the introduction point for a + new SIMD type. + +3. **Do NOT add Lance file I/O.** The GridLake column surface is the + in-process materialisation. Lance dataset open / commit / scan + belongs to PR-X13 OGIT bridge. GridLake exposes `as_bytes()` for + FFI; the FFI itself is downstream. + +4. **Do NOT add cross-column compute.** A `MultiLaneColumn` is one + column. Joins, projections, predicates over multiple columns are + composed by consumers iterating multiple columns in lockstep. + GridLake does not own a query layer. + +5. **Do NOT add a runtime schema registry.** The `#[derive(SoA)]` macro + is compile-time. The "ontology evolution → next compile cycle" + pipeline from the consolidation doc lives at the OGIT-schema-compile + layer, not at runtime in GridLake. + +## Acceptance gates — inherit canary gates, map to GridLake primitives + +The canary gates from `hhtl-canary-inhabitance-plan.md` (master commit +`c8f4af68`, lines 64-127) are the substrate's final-form gates. GridLake's +done criteria are the per-primitive contributions that *let* those gates +pass. + +### Correctness (binary, from canary lines 69-89) + +> **1. Revision output matches scalar reference**: `Fingerprint` (u64) +> bit-exact match against `src/hpc/nars.rs::revise`; `TruthValue` (f, c) +> within ULP ≤ 4 of scalar reference; 10,000 randomly-seeded revisions, +> zero divergences allowed. + +GridLake contribution: +- A2's `#[soa(pad_to_lanes=N)]` MUST NOT change `len()` semantics — + padded slots cannot leak into `revise()` input +- A4 integration probe 2 runs the 10,000-seed scalar-vs-SoA parity test + on the canary's `Belief` type, asserts bit-exact match before + PR-X1/PR-X2 ship + +> **5. Typed surfaces at zone boundaries** + +GridLake contribution: `MultiLaneColumn::as_bytes()` is the Zone-1↔Zone-2 +typed surface (Arrow buffer = Lance column = ndarray SoA = same bytes, +per the column-substrate identity). No `serde_json::Value`, no +`HashMap>` — the buffer is the contract. + +### Performance (numeric, from canary lines 91-109) + +> **6. Working set per worker thread**: ≤ **1 MB** (fits L2 cache on Zen4/SPR) + +GridLake contribution: `MultiLaneColumn::iter_u8x64()` zero-copy +semantics verified by A3 `bench_no_alloc` — no hidden allocations on +the hot path. The 1 MB budget assumes the SoA layout fits in L2; if +iterators allocate, the budget is blown. + +> **7. ndarray::simd primitive coverage**: 100% of hot-path SIMD ops route +> through `ndarray::simd::*` — zero raw intrinsics in the cognitive path +> (enforced by clippy lint and the W1a consumer contract gate). + +GridLake contribution: every lane iterator returns a `crate::simd::*` +type. Consumers cannot route around `ndarray::simd` because +`MultiLaneColumn` doesn't expose typed-slice-of-T accessors — only +lane iterators. This is the *enforcement mechanism* for gate 7 across +PR-X4 / PR-X9 / PR-X12. + +### Inhabitance (qualitative, from canary lines 110-126) + +> **3. The canary survives a sentinel-qa audit with zero P0 SAFETY +> findings on the new code.** + +GridLake contribution: all `unsafe` blocks in `MultiLaneColumn::iter_*` +audited by sentinel-qa with `// SAFETY:` comments citing the +multiple-of-64 invariant + lane-multiple-padding invariant. PR-X1 must +not merge with any P0 finding. + +> **2. No "Bardioc-shaped" code in the canary path.** + +GridLake contribution: the `#[derive(SoA)]` macro emits direct lane +accessors, not a runtime query DSL. A consumer that wants to filter a +column writes `column.iter_f32x16().filter(|lane| lane.gt(threshold))`, +not `query("SELECT * WHERE x > ?")`. The substrate identity is preserved. + +## Cross-references + +- `.claude/knowledge/stack-consolidation-bardioc-to-hhtl.md` § "Column-substrate identity" (lines 126-220, master commit `c8f4af68`) — load-bearing architectural justification +- `.claude/knowledge/stack-consolidation-bardioc-to-hhtl.md` § "Salvage from the 2026-05-19 cross-repo rollback" — heel_f64x8 absorption note (forbidden #1) +- `.claude/knowledge/hhtl-substrate-execution-prompt.md` § "Sprint sequencing" (lines 76-91) — 8-week schedule that Q-NEW-1 must patch +- `.claude/knowledge/hhtl-substrate-execution-prompt.md` § "W1-W2: PR-X10 kickoff" (lines 91-144) — kickoff block format mirrored below +- `.claude/knowledge/hhtl-canary-inhabitance-plan.md` § "Measurement gates" (lines 64-127) — canary correctness / performance / inhabitance gates GridLake inherits +- `.claude/knowledge/pr-x10-linalg-core-design.md` § "Distance kernels — `linalg::distance`" (master commit `c8f4af68`) — A6 owns L1/L2/L∞, GridLake must not duplicate +- `.claude/knowledge/vertical-simd-consumer-contract.md` — W1a contract every `crate::simd_soa::*` public fn obeys +- `.claude/rules/data-flow.md` — Rule #3 (no `&mut self` during computation) + +--- + +## § Sprint kickoff — W2.5 (path a) or W1-W2 inline (path b): PR-X1 + PR-X2 (GridLake substrate) + +```text +You are coordinator for PR-X1 + PR-X2 (GridLake substrate), the +in-process column-substrate-identity pre-sprint of the HHTL substrate +arc. 4 max-fan-out workers; 0.5-week window (path a) or 1-week inline +window (path b); produces the `crate::simd_soa::*` surface that every +downstream sprint consumes alongside `crate::hpc::linalg::*`. + +PREFLIGHT Q-NEW-1: plan-review savant MUST choose path (a) or path (b) +and commit the answer to `hhtl-gridlake-pre-sprint-prompt.md` before +spawning workers. Default recommendation: path (a). Do not start +without an answer. + +READ FIRST: +- `.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md` — this doc +- `.claude/knowledge/stack-consolidation-bardioc-to-hhtl.md` § "Column-substrate identity" — load-bearing reason GridLake exists +- `.claude/knowledge/hhtl-substrate-execution-prompt.md` — Protocol A 7-step cadence + sprint sequencing GridLake patches +- `.claude/knowledge/hhtl-canary-inhabitance-plan.md` § "Measurement gates" — gates GridLake inherits +- `.claude/knowledge/vertical-simd-consumer-contract.md` — W1a gate +- `.claude/knowledge/pr-x10-linalg-core-design.md` § "Distance kernels" — A6 absorption (do not duplicate) +- `.claude/rules/data-flow.md` — Rule #3 + +WORKER DECOMPOSITION (4 max-fan-out): +- A1 (sequential) — `simd_soa/multi_lane.rs` MultiLaneColumn carrier + 7 lane iterators +- A2 (parallel) — `simd-soa-derive/` proc-macro crate (#[derive(SoA)] + #[soa(pad_to_lanes=N)]) +- A3 (parallel) — `simd_soa/bench/` no-alloc + throughput + lane-swap + arc-clone benches +- A4 (parallel) — `simd_soa/integration/` splat3d probe + nars probe + arrow round-trip + +PROTOCOL A — execute the 7 steps in `hhtl-substrate-execution-prompt.md`. +The 6 specialist savants for the preflight review are listed there. +Specialist savants of particular interest for this sprint: +- data-flow (Rule #3 enforcement on MultiLaneColumn API) +- SAFETY (every unsafe block in iter_* needs justified // SAFETY:) +- W1a-consumer-contract (every public fn obeys the contract) +- naming-collision (simd_soa vs simd; multi_lane vs lane_view) + +ACCEPTANCE GATES: +- A1-A4 mandatory items merged with green tests, green clippy, green + codex P0 audit, SHIP verdict from P2 savant +- `cargo test --workspace --features simd-soa` passes +- A3 bench_no_alloc passes (zero hidden allocations on iter_* hot path) +- A4 nars-probe passes 10,000-seed scalar-vs-SoA parity test bit-exact +- A4 arrow round-trip passes zero-copy (no buffer allocation) +- W1a consumer contract honored for every new public SIMD-touching fn +- All unsafe blocks audited zero-P0 by sentinel-qa +- L1/L2/L∞ distance kernels NOT present in simd_soa::* (forbidden #1) + +PR FORMAT: open one PR per worker (A1..A4), all targeting a single +integration branch `pr-x1-x2/gridlake-substrate`. Coordinator merges the +integration branch as one PR to master after Protocol A step 7. + +BUDGET (path a): 0.5 weeks. If A1 slips past day 2, A2-A4 slip — A2 +needs MultiLaneColumn types, A3 needs lane iterators to bench, A4 needs +the carrier to probe. + +BUDGET (path b): inline with PR-X10's 2-week window. A1 spawns parallel +with A2-A12 after PR-X10's A1 (MatN) lands. A3/A4 spawn after GridLake's +A1 (MultiLaneColumn) lands. + +NEXT SPRINTS: +- (path a) W3 spawns PR-X11 + PR-X13 — both must use crate::simd_soa::* + for any column-shape work +- (path b) W3 spawns as normal; GridLake done-state goes in PR-X10's + merge message +- W4 PR-X12 codec uses MultiLaneColumn for rANS / leaf storage +- W5 PR-X4 splat cascade uses #[derive(SoA)] for tile / splat columns +- W6 PR-X9 basin codebook uses MultiLaneColumn for the lookup table +- W8 canary uses #[derive(SoA)] on Belief; the canary's bit-exact + parity test against scalar revise is GridLake's final acceptance +``` diff --git a/.claude/knowledge/hhtl-pr-x14-substrate-contract-prompt.md b/.claude/knowledge/hhtl-pr-x14-substrate-contract-prompt.md new file mode 100644 index 00000000..877771bc --- /dev/null +++ b/.claude/knowledge/hhtl-pr-x14-substrate-contract-prompt.md @@ -0,0 +1,424 @@ +# HHTL Substrate — PR-X14′ Pre-Sprint Prompt (lance-graph-contract::column + bridge) + +> Date: 2026-05-19 (drafted alongside `hhtl-gridlake-pre-sprint-prompt.md` +> on branch `claude/gridlake-pre-sprint-prompt`, parent commit `ade8edb2`) +> +> Status: Pre-sprint kickoff — slots concurrent with PR-X1 + PR-X2 (GridLake) +> at W2.5, OR immediately after at W3. Companion to: +> - `hhtl-gridlake-pre-sprint-prompt.md` — the carrier (`MultiLaneColumn` + +> `#[derive(SoA)]`) that this contract wraps +> - `hhtl-substrate-execution-prompt.md` — W1-W8 master schedule +> - `stack-consolidation-bardioc-to-hhtl.md` — § "Column-substrate identity" +> +> **Q-NEW-2 marker present** — placement choice (concurrent W2.5 vs sequential +> W3 after GridLake lands) is the plan-review savant's call at preflight. + +## Why this exists — four duplicate column-access patterns + +Hot paths across the lance-graph workspace each implement their own column +access today. Four distinct patterns coexist; none share a substrate carrier. +**The carrier the GridLake prompt names (`MultiLaneColumn`) needs a contract +crate as its consumer-facing surface, or each consumer rebuilds the same +bridge.** + +### Pattern 1 — `Box<[u64]>` flat owned buffers (BindSpace, SIMD-hottest reader) + +`cognitive-shader-driver/src/bindspace.rs`:36-44 (verbatim): + +```rust +pub struct FingerprintColumns { + pub content: Box<[u64]>, // len * 256 u64 = flat SoA + pub cycle: Box<[f32]>, // len * 16_384 f32 = Vsa16kF32 carrier + pub topic: Box<[u64]>, + pub angle: Box<[u64]>, + pub sigma: Box<[u8]>, // 1 byte per row +} +``` + +Comment at line 51 explicit: *"row-major `Box<[u64]>` gives O(1) +`chunks_exact(256)` iteration which LLVM autovectorises cleanly."* + +This pattern is **already MultiLaneColumn-shaped** — flat byte storage, +multi-row chunked SIMD access, separate column per field. But it is owned not +`Arc`-shared, and bindspace rolled it by hand because no shared carrier +exists in the workspace. + +### Pattern 2 — `HashMap` (lance-graph query + SQL path) + +`lance-graph/src/query.rs`:243-245 (verbatim): + +```rust +pub async fn execute( + &self, + datasets: HashMap, + ... +) -> Result +``` + +`lance-graph/src/sql_query.rs`:69 mirror: + +```rust +pub async fn execute(&self, datasets: HashMap) -> Result +``` + +Cypher path normalizes via `normalize_record_batch` at line 97; SQL path runs +through datafusion's `SessionContext`. **The datafusion dep lives here +(`lance-graph/Cargo.toml`:35-39) and stays untouched** — SQL is deferred per +the 2026-05-19 architecture decision. + +Every consumer downcasts `Arc` from `batch.columns()` itself — +duplicate downcast/extract logic in each Cypher operator and each +DataFusion-backed SQL caller. + +### Pattern 3 — `Morsel { columns: Vec }` placeholder enum (the planner) + +`lance-graph-planner/src/physical/mod.rs`:37-58 (verbatim) — **explicit +placeholder, not yet wired to Arrow**: + +```rust +/// A morsel of data (batch of rows) flowing through the pipeline. +/// In the real implementation, this wraps Arrow RecordBatch. +#[derive(Debug, Clone)] +pub struct Morsel { + pub num_rows: usize, + /// Column data (placeholder — real impl uses Arrow arrays). + pub columns: Vec, +} + +/// Column data in a morsel (placeholder for Arrow integration). +#[derive(Debug, Clone)] +pub enum ColumnData { + Int64(Vec), + Float64(Vec), + String(Vec), + Fingerprint(Vec>), + TruthValue(Vec<(f64, f64)>), +} +``` + +The planner has the IR (`ir/{expr.rs, logical_op.rs, schema.rs}`) and the +cost model (`plan/{cost.rs, dp_enumerator.rs}`); its physical executor is +awaiting a real Arrow bridge. **The comment "placeholder for Arrow +integration" is the gap PR-X14′ closes.** + +### Pattern 4 — `lance::Dataset.scan()` direct (optional-feature consumers) + +Two feature-gated consumers each call lance directly: + +- `crates/holograph/Cargo.toml`:38 — `lance = { version = "=4.0.0", optional = true, default-features = false }` +- `crates/lance-graph-ontology/Cargo.toml`:35-47 — `lance = { version = "=4.0.0", optional = true }` behind feature `lance-cache` + +Each opens `lance::Dataset` and calls `.scan()` returning +`Stream`. Each rolls its own scan-bridge per feature flag. A +third candidate (`bgz-tensor` with optional `lance-graph-contract` dep) is +positioned to need the same bridge. + +### ndarray's dep direction (clarification) + +`grep` confirms **ndarray has zero direct dep on `lance` or `lancedb`**. The +"ndarray crates depending on lance" you named are lance-graph crates (jc, +lance-graph-planner, cognitive-shader-driver, …) that depend on **ndarray +↑** (for SIMD primitives via `crate::hpc::*` + `crate::simd::*`). The +dep direction is **ndarray-as-substrate**, not ndarray-needs-lance. + +PR-X14′ preserves this direction. The contract crate lives in the lance-graph +workspace; the bridge crate lives in the lance-graph workspace; ndarray +remains substrate-pure with zero lance/arrow surface. + +## The contract crate already exists — but lacks a column module + +`crates/lance-graph-contract/src/lib.rs`:1-12 (verbatim): + +``` +//! # lance-graph-contract — The Single Source of Truth +//! +//! Zero-dependency trait crate that defines the contract between: +//! - **lance-graph-planner** (implements these traits) +//! - **ladybug-rs** (calls Planner + CamPq + OrchestrationBridge) +//! - **crewai-rust** (calls ThinkingStyleContract + MulContract) +//! - **n8n-rs** (calls JitContract + OrchestrationBridge) +``` + +10+ workspace crates already depend on it (jc, sigma-tier-router, +lance-graph-ontology, lance-graph-archetype, lance-graph-planner, +cognitive-shader-driver, bgz-tensor, lance-graph-rbac, lance-graph-cognitive, +lance-graph-callcenter). + +But its 44 modules are **cognitive contracts** (`thinking`, `mul`, `cam`, +`jit`, `nars`, `qualia`, `collapse_gate`, `cycle_accumulator`, `splat`, +`vsa`, …). **There is no `column`, no `multi_lane_column`, no `soa`, no +`lance_source`.** The column-substrate contract is the missing module. + +## What PR-X14′ is, mechanically + +Two pieces. One module set added to the existing `lance-graph-contract` +crate, plus one new sibling bridge crate. + +### Piece 1 — `lance-graph-contract::column` module set (ADD) + +Submission target: existing crate `crates/lance-graph-contract/src/`. +Preserves the "zero-dependency trait crate" invariant from lib.rs:1. + +``` +crates/lance-graph-contract/src/ (existing, ADD column/) +└── column/ + ├── mod.rs // re-exports + ├── multi_lane_column.rs // MultiLaneColumn carrier — owns the surface + │ // PR-X1 (GridLake) emits this type + ├── source.rs // GridLakeSource trait + │ // pub trait GridLakeSource { + │ // fn column(&self, name: &str) -> Result; + │ // fn columns(&self) -> impl Iterator; + │ // fn schema(&self) -> &SchemaRef; + │ // } + ├── soa.rs // SoaSource trait surface + │ // PR-X2 (#[derive(SoA)]) emits impls to this + ├── lane_view.rs // typed lane iterator surfaces + │ // F32x16Iter / U64x8Iter / Bf16x32Iter handles + │ // (delegating to ndarray::simd types) + └── schema.rs // SchemaRef alias + minimal column-metadata + // (kept thin; do NOT pull arrow-schema here — + // re-export a slim view-shaped type that the + // bridge crate adapts to/from arrow_schema) +``` + +Crate-level Cargo.toml change (`crates/lance-graph-contract/Cargo.toml`): +- Add `ndarray = { path = "../../../ndarray", default-features = false, features = ["std", "hpc-extras"] }` (already a transitive dep via lance-graph-planner; here it becomes direct because `lane_view` needs `ndarray::simd::F32x16` etc.) +- Do NOT add `arrow`, `arrow-array`, `arrow-schema`, `lance` — those live in the bridge crate. + +**Zero-dependency invariant**: the contract crate after this change depends +on `ndarray` (path) + `serde` (already present) + `thiserror` (already +present) + `glob` (already present, build-script only). No arrow, no lance, +no datafusion. + +### Piece 2 — `lance-graph-contract-bridge` (NEW sibling crate) + +``` +crates/lance-graph-contract-bridge/ (NEW) +├── Cargo.toml (deps: lance-graph-contract, +│ lance = "=4.0.0", +│ arrow = "57", +│ arrow-array = "57", +│ arrow-schema = "57", +│ ndarray (path)) +└── src/ + ├── lib.rs + ├── record_batch_to_mlc.rs // RecordBatch → MultiLaneColumn + │ // Zero-copy where alignment permits: + │ // if Arrow buffer is 64-byte aligned + │ // and length is a lane-multiple, wrap + │ // Arc<[u8]> over the existing buffer. + │ // Otherwise: one-shot pad+align copy. + ├── lance_dataset_to_mlc.rs // lance::Dataset.scan() → MultiLaneColumn stream + │ // pub async fn scan_as_mlc( + │ // dataset: &Dataset, + │ // columns: &[&str], + │ // ) -> impl Stream>> + ├── arrow_array_view.rs // Arc → typed lane view + │ // Type-by-type bridge: PrimitiveArray + │ // → MultiLaneColumn { width: 4 } with F32x16 lane iterator + └── soa_record_batch.rs // impl SoaSource for RecordBatch + // (the dual: take a RecordBatch shaped per the + // derive macro's schema declaration, project + // to typed SoA struct) +``` + +Crate ownership: lives at `crates/lance-graph-contract-bridge/` in the +lance-graph repo. Opt-in by anyone needing Lance/Arrow access; pure-cognitive +consumers (cognitive-shader-driver hot path, jc math kernels) do not pull +lance/arrow through. + +## Migration — four patterns collapse to one + +| Current pattern | Location | After PR-X1/X2/X14′ | +|---|---|---| +| **1.** `FingerprintColumns { content: Box<[u64]>, cycle: Box<[f32]>, ... }` | `cognitive-shader-driver/src/bindspace.rs`:36-44 | `FingerprintColumns { content: MultiLaneColumn, cycle: MultiLaneColumn, ... }` — same SIMD-hot reader, but bytes can now come from BindSpace-owned (via `MultiLaneColumn::from_box`) OR from a Lance scan (via bridge) OR from a `RecordBatch` (via bridge). Bindspace stops being a source-of-truth duplicate. | +| **2.** `HashMap` (Cypher + SQL execute) | `lance-graph/src/query.rs`:243, `sql_query.rs`:69 | **Cypher path**: `HashMap` via bridge crate; Cypher executor gets typed lane views directly. **SQL path: UNCHANGED** — keeps `RecordBatch + datafusion::SessionContext`. Two paths, one substrate. | +| **3.** `Morsel { columns: Vec }` | `lance-graph-planner/src/physical/mod.rs`:37-58 | `Morsel { columns: Vec }`. Drop the placeholder enum entirely. The planner is finally wired to real Arrow bytes via the bridge crate; physical operators receive typed lane iterators directly. | +| **4.** `lance::Dataset.scan()` rolled per consumer | `holograph` (`lancedb` feature), `lance-graph-ontology` (`lance-cache` feature), `bgz-tensor` (proposed) | All three call `lance_dataset_to_mlc::scan_as_mlc()` from the bridge crate. Single scan-bridge impl; the three feature-gated callers share one path. | + +**Net clutter-replacement count**: 4 duplicate column-access bridges → 1 +typed surface (in the contract crate) + 1 reusable bridge impl (in the new +sibling crate). The SQL/datafusion surface stays untouched as a deliberate +deferred decision; the cognitive/SIMD-hot/planner/optional-Lance surfaces all +share one substrate. + +## Schedule slot — Q-NEW-2 for plan-review savant + +The 8-week schedule from `hhtl-substrate-execution-prompt.md`:76-91 has not +been amended to include PR-X14′. Two viable insertion points: + +### Path (α): Concurrent with GridLake at W2.5 + +PR-X1 + PR-X2 + PR-X14′ all land in the same 0.5-week slot. Workers fan out: +- A1 (chain dep): `MultiLaneColumn` type signature + `Arc<[u8]>` + alignment + invariants — must land first, parsing target for all others +- A2-A4 (parallel): SoA derive macro, lane iterator implementations, + alignment+padding tests +- A5-A7 (parallel, blocked on A1 only): `column/source.rs` + `column/soa.rs` + + `column/lane_view.rs` in lance-graph-contract +- A8-A10 (parallel, blocked on A1 + A5): bridge crate impls + (`record_batch_to_mlc`, `lance_dataset_to_mlc`, `arrow_array_view`) + +Pros: +- Single coordinator + savant cycle covering GridLake + contract + bridge +- All three pieces land together as one substrate "block" +- W3+ consumers (X11, X12, X13, X4, X9) start with the full substrate + already in place + +Cons: +- 10 workers in W2.5 (heavier than the GridLake-only 4) +- Coordinator's fan-out math is tighter; A1 critical-path slip cascades +- A 0.5-week slot becomes ~5 days at heavy fan-out + +### Path (β): Sequential at W3 (after GridLake lands) + +PR-X1 + PR-X2 ship at W2.5 (4 workers). PR-X14′ takes W3's first half as a +dedicated 3-day sprint with 4 workers, blocking W3's PR-X11 + PR-X13 to +W3.5+. + +Pros: +- Clean dependency: PR-X1 + PR-X2 land before X14′ starts; no chain-dep risk +- 4-worker fan-out matches GridLake shape (one coordinator, one cycle) +- A1 critical-path stays inside GridLake; X14′ workers have a stable target + +Cons: +- W3's PR-X11 + PR-X13 shift right by 3 days → 8-week arc becomes 8.5+ weeks +- Two preflight cycles (GridLake savant cycle + X14′ savant cycle) instead + of one + +**The plan-review savant decides α vs β at preflight.** Both paths reach the +same end-state; the choice is fan-out heat vs schedule tightness. + +## Worker spawn shape — path (α) detail + +If path α is chosen, the worker DAG: + +``` +A1: MultiLaneColumn carrier (chain dep) + │ + ├──→ A2: #[derive(SoA)] proc-macro + ├──→ A3: lane iterators (F32x16/U64x8/Bf16x32 typed handles) + ├──→ A4: alignment + padding test harness + │ + ├──→ A5: lance-graph-contract::column::source (GridLakeSource trait) + ├──→ A6: lance-graph-contract::column::soa (SoaSource trait) + ├──→ A7: lance-graph-contract::column::lane_view (typed iterator surfaces) + │ │ + │ └──→ A8: bridge::record_batch_to_mlc + │ └──→ A9: bridge::lance_dataset_to_mlc + │ └──→ A10: bridge::arrow_array_view + soa_record_batch +``` + +A1 is the only chain dep. A2-A10 are all parallel after A1 lands. Critical +path is A1 → (A5 ‖ A7) → A8/A9/A10; typical depth ~3 days. + +## Done criteria + +The sprint is done when ALL of the following hold: + +1. **Contract module set** lands in `crates/lance-graph-contract/src/column/` + - All five files present (`mod.rs`, `multi_lane_column.rs`, `source.rs`, + `soa.rs`, `lane_view.rs`) + - **Zero new external deps** in `lance-graph-contract/Cargo.toml` except + `ndarray` (path) + - `cargo build -p lance-graph-contract --no-default-features` passes + (proves zero-dep invariant preserved) + +2. **Bridge crate** lands at `crates/lance-graph-contract-bridge/` + - `record_batch_to_mlc::wrap_zero_copy` returns `MultiLaneColumn` from an + already-64-byte-aligned, lane-padded Arrow buffer with **0 allocations** + (verified by `dhat` or `cargo bench` heap-counter) + - `record_batch_to_mlc::pad_and_align` returns `MultiLaneColumn` from an + unaligned/short Arrow buffer with exactly **1 allocation** (the padded + `Arc<[u8]>`) + - `lance_dataset_to_mlc::scan_as_mlc` round-trips a Lance dataset → MLC + stream → reassembled `RecordBatch` byte-identical to the source + +3. **Parity tests against the four current patterns**: + - **BindSpace migration parity**: `FingerprintColumns` rebuilt over + `MultiLaneColumn` produces byte-identical SIMD sweep output to the + current `Box<[u64]>` impl on a 1M-row fixture + - **lance-graph Cypher parity**: a representative Cypher query over a + dataset routed through `MultiLaneColumn` (via bridge) produces identical + result to the current `RecordBatch`-HashMap path + - **Planner Morsel parity**: a representative `Morsel`-shaped pipeline + (BROADCAST → SCAN → ACCUMULATE → COLLAPSE) executes against + `Vec` and produces identical row counts + identical + fingerprint output to the placeholder-enum path + - **`lance::Dataset.scan()` parity**: holograph + ontology call + `scan_as_mlc` and produce identical row data to the current per-feature + scan implementation + +4. **Benchmarks**: + - Zero-copy MLC wrap of pre-aligned Arrow buffer: **<50 ns** (one Arc + atomic + struct construction) + - Lane iterator over `MultiLaneColumn::iter_f32x16` on 16K-row column: + **within 5%** of the equivalent `chunks_exact(16).map(F32x16::from)` + loop over `&[f32]` (proves no abstraction tax) + - BindSpace SIMD sweep over `MultiLaneColumn::iter_u64x4` for fingerprint + hamming: **within 5%** of the current `Box<[u64]>` chunks_exact path + (proves bindspace migration is free) + +5. **Workspace compile**: `cargo check --workspace` clean. All 10+ existing + `lance-graph-contract` consumers compile without source changes (the + `column` module is purely additive). + +6. **Out of scope (must NOT be touched by this sprint)**: + - `lance-graph/src/sql_query.rs` and `sql_catalog.rs` — SQL path stays on + `RecordBatch + datafusion` per the 2026-05-19 deferred-SQL decision + - `lance-graph/Cargo.toml`'s datafusion deps (lines 35-39) — left + untouched per the 2026-05-19 "don't evict datafusion" decision + - The Cypher parser (`lance-graph/src/parser.rs`) — works as-is; only the + execution path's data interface shifts to MLC + - ndarray crate — receives zero changes; this PR is consumer-side only + +## Forbidden constraints + +Two invariants the sprint MUST NOT violate: + +1. **The zero-dependency invariant of `lance-graph-contract`** stated in + lib.rs:1: *"Zero-dependency trait crate that defines the contract + between [...]"*. Adding `ndarray` (path dep, already transitively + present) is acceptable because ndarray is the substrate; adding `arrow`, + `arrow-schema`, `arrow-array`, `lance`, `lancedb`, or `datafusion` to + this crate **violates the invariant and gates the sprint**. + +2. **No write to lance-graph upstream storage format**. The bridge crate + reads `lance::Dataset` and `arrow::RecordBatch` as-is. It does NOT define + new Arrow extension types, NOT register custom Lance encoders, NOT + modify schemas in place. Wrapping via `Arc<[u8]>` over existing buffers + is allowed; in-place mutation of Arrow/Lance buffers is forbidden. + +## Cross-references + +- `.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md` — PR-X1 + PR-X2 + carrier (`MultiLaneColumn` + `#[derive(SoA)]`) that this contract wraps +- `.claude/knowledge/hhtl-substrate-execution-prompt.md` — W1-W8 master + schedule (PR-X14′ slots at W2.5-α or W3-β) +- `.claude/knowledge/stack-consolidation-bardioc-to-hhtl.md` § "Column- + substrate identity — Lance ≡ Arrow ≡ ndarray SoA" lines 126-220 +- `.claude/knowledge/pr-arithmetic-inventory.md` § "Shopping-list addendum" + — needs 2026-05-19 update with PR-X14′ entry +- File evidence cited above: + - `crates/lance-graph-contract/src/lib.rs`:1-12 (zero-dep claim) + - `crates/cognitive-shader-driver/src/bindspace.rs`:36-44 (Pattern 1) + - `crates/lance-graph/src/query.rs`:243-245 (Pattern 2 Cypher) + - `crates/lance-graph/src/sql_query.rs`:69 (Pattern 2 SQL) + - `crates/lance-graph-planner/src/physical/mod.rs`:37-58 (Pattern 3 placeholder) + - `crates/holograph/Cargo.toml`:38 (Pattern 4 holograph) + - `crates/lance-graph-ontology/Cargo.toml`:35-47 (Pattern 4 ontology) + - `crates/lance-graph/Cargo.toml`:35-39 (datafusion deps to leave alone) + +## TL;DR + +PR-X14′ adds a `column/` module set to the existing `lance-graph-contract` +crate (preserving its zero-dep invariant) and a new sibling +`lance-graph-contract-bridge` crate that bridges Lance datasets + Arrow +RecordBatches into `MultiLaneColumn`. Four current duplicate column-access +patterns (BindSpace `Box<[u64]>`, lance-graph query `HashMap`, planner `Morsel` placeholder enum, `lance::Dataset.scan()` +rolled per feature) collapse to one. SQL path + lance-graph's datafusion +deps are deliberately untouched. The plan-review savant chooses path α +(concurrent with GridLake at W2.5, 10 workers) vs path β (sequential at W3, +4 workers + half-week schedule extension) at preflight. diff --git a/.claude/knowledge/pr-arithmetic-inventory.md b/.claude/knowledge/pr-arithmetic-inventory.md index a559af2b..1307ceb7 100644 --- a/.claude/knowledge/pr-arithmetic-inventory.md +++ b/.claude/knowledge/pr-arithmetic-inventory.md @@ -341,6 +341,84 @@ PR-X10 ships the ndarray-side canonical surface; jc agents pick up the consolidation against it. PR-X10 is **independent** of PR-X4 / PR-X9 / PR-Z1 (no file overlap), can ship concurrently from a separate branch. +## Shopping-list addendum (2026-05-19 substrate-pair update) + +The 2026-05-18 entry above (PR-X10 + jc consolidation) is the **arithmetic** +substrate. A 2026-05-19 cross-cutting design session identified the +**storage-and-contract** substrate as the missing companion piece. Two +pre-sprint prompts capture it: + +- `.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md` — **PR-X1 + PR-X2** + (GridLake carrier: `MultiLaneColumn` + `#[derive(SoA)]` proc-macro) +- `.claude/knowledge/hhtl-pr-x14-substrate-contract-prompt.md` — **PR-X14′** + (`lance-graph-contract::column` module set + new + `lance-graph-contract-bridge` sibling crate) + +### Cost-of-ownership framework adopted this session + +Cost-of-ownership is calculated only by **how many cluttered parts a choice +replaces**. By that metric the substrate-pair is the highest-leverage move +in the Phase-2 arc: + +| Adoption | Cluttered parts replaced | +|---|---| +| PR-X1 + PR-X2 (GridLake carrier) | **5 implicit per-consumer column-buffer copies** (Databend, Tantivy, lance-graph, sea-orm, SurrealDB each maintain their own) collapse to one | +| PR-X14′ (contract + bridge) | **4 duplicate column-access patterns** (BindSpace `Box<[u64]>`, lance-graph query `HashMap`, planner `Morsel` placeholder enum, `lance::Dataset.scan()` rolled per feature) collapse to one | +| Databend (future, when integrated through the contract) | ClickHouse (1 JVM/C++ service decommissioned) | +| Tantivy (future, when integrated through the contract) | Elasticsearch + Lucene (1 JVM service decommissioned) | +| lance-graph + ndarray-over-TiKV (existing) | JanusGraph + TinkerPop/Gremlin (2 JVM services decommissioned) | + +### Decisions that pinned the scope + +Three architectural decisions narrowed the prompt's scope from earlier +proposals in the session: + +1. **Skip SQL for now**. All SQL-frontend work (sea-orm column contract, + sqlparser-rs standalone, PostgREST front, the Phase-3 "fusion parser" + sprint) is deferred. The lance-graph SQL path keeps its `RecordBatch + + DataFusion::SessionContext` pipeline unchanged. + +2. **Don't evict datafusion**. `lance-graph/Cargo.toml`:35-39 has hard deps + on `datafusion = "52"` + four sibling `datafusion-*` crates; lance-graph + upstream depends on them. PR-X14′ does NOT touch these — the datafusion + 1400-module tail stays contained inside the lance-graph crate. + +3. **Use storage format as-is**. PR-X14′ reads `lance::Dataset` and + `arrow::RecordBatch` as-is; it does NOT define Arrow extension types, + NOT register custom Lance encoders, NOT modify schemas in place. + +### Decisions superseded + +- ❌ Earlier-session proposal: PR-X14 evicts datafusion from lance-graph and + builds a parsendes Fusionskraftwerk at 240ns/parse. **Superseded** — + lance-graph upstream dep makes eviction out of scope; the planner IR + + Cypher nom parser + NSM/FSM parser already exist and PR-X14′ is the + consolidation, not a fusion-parser sprint. +- ❌ Earlier-session proposal: two new top-level crates `gridlake-contract` + + `gridlake-bridge`. **Superseded** — `lance-graph-contract` already + exists with 10+ workspace consumers and is the right home for the column + module; only a sibling bridge crate is genuinely new. + +### Two Q-markers carried forward (plan-review savant decides at preflight) + +- **Q-NEW-1** (from GridLake prompt): GridLake at W2.5 prerequisite slot + vs absorbed into PR-X10 as A13/A14 +- **Q-NEW-2** (from PR-X14′ prompt): PR-X14′ concurrent with GridLake at + W2.5 (path α, 10 workers) vs sequential at W3 (path β, 4 workers + 0.5 + week schedule extension) + +### Forbidden constraints (preserved from the prompts) + +- PR-X14′ must NOT add `arrow`, `arrow-schema`, `arrow-array`, `lance`, + `lancedb`, or `datafusion` to `lance-graph-contract`'s Cargo.toml — + preserves the lib.rs:1 zero-dependency invariant +- PR-X14′ must NOT touch `lance-graph/src/sql_query.rs`, + `lance-graph/src/sql_catalog.rs`, or `lance-graph/Cargo.toml`'s + datafusion deps — these are deliberately out of scope per decision (2) + above +- ndarray crate receives zero changes from PR-X14′ — the substrate-pair is + consumer-side only + ## Cross-references - `/root/.claude/uploads/.../7b0ea082-splat3d_sprint_prompt.md` — splat3d sprint (shipped as ndarray PR #153, 2026-05-18) @@ -351,3 +429,5 @@ consolidation against it. PR-X10 is **independent** of PR-X4 / PR-X9 / PR-Z1 - `.claude/knowledge/pr-x9-design.md` — lazy basin-codebook storage - `.claude/knowledge/pr-z1-ogit-cognitive-bootstrap.md` — OGIT Cognitive namespace bootstrap - `.claude/knowledge/pr-x10-linalg-core-design.md` — **the consolidating linalg sprint** that unblocks splat3d training + inference modules + jc Pillars simultaneously +- `.claude/knowledge/hhtl-gridlake-pre-sprint-prompt.md` — **PR-X1 + PR-X2** GridLake carrier (`MultiLaneColumn` + `#[derive(SoA)]`); 2026-05-19 session output, branch `claude/gridlake-pre-sprint-prompt` +- `.claude/knowledge/hhtl-pr-x14-substrate-contract-prompt.md` — **PR-X14′** `lance-graph-contract::column` module + `lance-graph-contract-bridge` sibling crate; 2026-05-19 session output, same branch