feat(core): the membership store — one vector row per idea-atom, occupancy as a relation (bp-152)

[ascalva]

Implements bp-152 (docs/build-plans/bp-152/plan.md) — D1/D2/D3/D8 of
docs/design-notes/vector-membership-store.md, including Amendment A2. Second of three
plans on the code-ingest track; bp-151 (canonical identity) and bp-155 (path-independence) are
already merged, which is what makes the dedup claims here true.

Objective: store one vector row per idea-atom and carry all occupancy in a membership
relation, so a version is a fiber and a re-land is idempotent.

No stored data is migrated and nothing is re-embedded. Every stored-data item in this plan is
fixture-only; the live-store rebuild is bp-153.

---

What changed

New — core/stores/memberships.py. The membership relation M ⊆ V × O in SQLite, beside the
vault catalog. Key = the occupancy's coordinates (path, blob_sha, layer, chunk_index); columns
content_id, slot, slot_line_start, slot_line_end, current, tombstoned. A version is
the fiber M(path, blob_sha). Also: derived lineage (slot_runs/slot_edges, adjacent
collapse, quantified over a chain handed in), the D3 read join (resolve_occupancies), the D5
purge (purge_atom), and the D8 repair pass (orphan_atom_ids, repair_current_any,
current_any_drift).

Outer ring, and structurally so: core/kernel/** is untouched, so sourceset stays in the
inner-ring fixed point (the C5/D3 pin) and memberships enter the kernel as data through the
existing RowSource protocol. A test scans core/kernel/** for the import and carries a negative
control so its emptiness is a fact, not a broken check.

core/ingest/code_corpus.py.

• atom_id(chunk) = f"{layer}:{content_hash}" — path-free, corpus-wide. Dropping the path from
  the id is the atom model: it promotes the old per-path dedup to corpus-wide dedup (PD-1).
• code_rows(chunks, vectors, *, current=False) emits atom rows with the occupancy columns shed.
• code_memberships(path, blob_sha, chunks) — the fiber, and the A2 translation point. It builds
  its own list, one row per chunk, so the atom-side by_id.setdefault collapse is not
  inherited.
• CodeLander.land(path, blob_sha, chunks, *, head_blob_sha=None) — D2's five steps in D8's order,
  plus reconcile() and supersede_path().
• CodeCorpusSync gained required memberships and embedder_identity fields, and its
  D-fiber state re-homed from {(source_path, digest)} to memberships.fibers().

core/stores/vectorstore.py. ATOM_ROW_SHED, is_code_atom_row, the all_rows shed guard
(below), atom_rows(), set_current_any(ids, value) (batched, chunked predicates),
delete_atom(id) (the purge path), and the current column's schema comment corrected to carry
both readings.

eval/harness/code_retrieval.py — see "the one reach beyond write_scope" below.

Nine test files repaired; one new (tests/unit/test_memberships.py, 18 tests).

---

The three things it would have been easy to get wrong

1. A re-land is idempotent BECAUSE reconciliation converges, not because the call
short-circuits. land() runs D2 step 4 even when step 3 wrote nothing. On A → B → A the fiber
for blob A already exists carrying current=false, so a lander that returns early on "fiber
exists" leaves B marked HEAD — silent corruption of every default read, nothing raised and
nothing logged. core/stores/versions.py:22-27 documents the repo learning this once already at
note grain. The same rule applies one level up: sync() reconciles every unchanged HEAD path
rather than skipping it, because "unchanged blob ⇒ skip" is the identical short-circuit.

test_a_lander_that_short_circuits_on_an_existing_fiber_leaves_b_current exhibits the buggy
lander passing every count assertion and failing the currency ones — so the currency assertions
are known to have teeth.

2. The atom side dedups; the membership side must not. Two byte-identical L0b windows in one
blob are two occupancies with distinct chunk_index (the F5 multiset pin). The test asserts
both halves at once: |V| == 1 (geometry collapsed) and len(fiber) == 2 (occupancy did not).

3. provenance STAYS on the atom row. The mirror firewall is a row prefilter
(provenance IN (...), prefilter=True), so shedding the column would not weaken the firewall —
it would remove it, with no failing call anywhere. The test asserts presence and equality to
CODE on every row, not merely "no code leaked". No firewall assertion in test_code_mirror.py or
test_code_vector_isolation.py was relaxed; both read through the new join and claim the same
thing.

---

The group_sources guard (Item 3 — the note's §3 Q5 gap)

source_sets(store) defaults to all strata by design ("a structural grouping utility, not a
mirror read"), and group_sources keys on r["digest"]. Shed atom rows all carry digest='', so
every code atom in the corpus would collapse into ONE bogus SourceSet keyed '' — and
MixedProvenanceError cannot fire, because it needs a digest spanning several provenances and
these rows are uniformly CODE. It fails with no visible error at all.

The guard lives in VectorStore.all_rows: an unscoped read excludes shed atom rows unless the
caller passes include_atom_rows=True. It is on the shed side, not in the kernel, exactly as the
plan requires. The test reproduces the falsifier first —
group_sources(vs.all_rows(include_atom_rows=True)) really does return the '' set containing
every atom — and only then asserts the guard; a second test pins that the guard keys on the shed
(is_code_atom_row), never on the provenance, so the live store's existing pre-D1 code rows keep
grouping until bp-153 rebuilds them.

Two other consumers benefit rather than break: core/curator/curator.py's prune_candidates has
the identical group-by-digest shape and would have reported the whole atom plane as one orphaned
digest; core/dreaming/evaluate.py's unscoped read becomes strictly more conservative.

---

Amendment A2, end-to-end

CodeChunk.line_start/line_end and the membership columns became
slot_line_start/slot_line_end. The vector-row Arrow columns keep their names (A2.3 — shared
with the prose lane, which has no slot concept). Stored values are unchanged; what changes is that
the misreading loses the name it was hiding behind.

The rename without the assertion would be cosmetic, since every leaf-symbol fixture passes
either way — so the fixture carries a class with methods and a module shell, and the test
asserts (i) the span equals the symbol's declared lineno..end_lineno, (ii) the atom's text
coverage is a strict subset of that span (def bar is in Foo's span, never in Foo's text),
(iii) the module shell's extent is the entire file for four lines of preamble, and (iv) — the
control that makes the rest mean anything — for the leaf symbol top span and coverage coincide
exactly.

The embedder pin

Atom presence is keyed to (layer, content_hash) and EmbeddingConfig.model + dim, recorded
in an atoms ledger table in the same SQLite file. This is not a second truth about occupancy
— it stores the one fact the vector table structurally cannot: its Arrow schema is shared with the
prose lane and has no embedder column, and a stored vector's len() recovers dim but never
model. Without it the owner's pin is unenforceable and a model change would silently mix two
geometries in one ANN space. It is also what makes an orphan observable for §8(e).

Its own test case, with the same-embedder control asserted first — a suite exercising one embedder
cannot see this bug.

---

Every criterion, and how it was made non-vacuous

criterion	green	the precondition that makes it bite
§8(a) revert (C1)	✅	A and B asserted DISTINCT before the third land; currency assertions carry the claim; a mutation test exhibits the short-circuiting lander passing every count and leaving B current
§8(b) fork	✅	|V| == 3 < Σ chunks == 4 and n_doc(shared) == 2 asserted FIRST; hand-built chunks so "exactly 1 new atom" is exact arithmetic
§8(c) purge	✅	the note-lane removal paths are shown to ACT (the note row really is deleted) while |V| is unmoved; then the purge report is asserted non-zero and the tombstoned rows asserted to still EXIST
§8(d) retrieval	✅	a superseded occupancy asserted to exist AND an atom whose every home is superseded (n_doc == 0); default search excludes it, include_superseded surfaces it, a shared atom resolves to both homes
§8(e) crash	✅	the injection point asserted: |V| grew, |M| did not, the fiber is empty, orphan_atom_ids() is non-empty and equals the landed set; the re-land then adopts them at zero embeds
§8(f) invariants	✅	one test asserts the fixture carries all four shapes before any invariant is read; A→B→A is asserted to give 3 runs / 2 edges where distinct-collapse gives 2; the fiber-sum invariant is broken by a subclass whose fiber() drops superseded rows; the drift invariant by a hand-flipped flag

One correction to the plan's phrasing, found while building: "slotted" is the layer, not a
non-empty name. The L0a module shell carries qualname='' just like L0b/L1, so reading
slottedness off slot != '' would silently drop the lineage of the one slot every file has.
slot_runs filters on layer == LAYER_CODE_AST.

---

For the reviewer: two things to rule on

1. title is shed, and D1 does not say either way. The note's shed list enumerates
source_path, digest, qualname, line_*, chunk_index; the keep list is
id, layer, text, vector, provenance, current. title is in neither. On a code row title was
set to path — it is source_path under another name — so it is shed with the occupancy
columns. Keeping it would stamp every shared atom with its first-landed path, exactly the
coordinate D0's consequence note says must never be relied on. One-line reversal if you read D1's
enumeration as exhaustive.

2. One reach beyond write_scope: eval/harness/code_retrieval.py (+~20 lines, additive).
ranked_paths reads h["source_path"] off a hit; on an atom row that is '', so the M-C3/M-C5
battery silently ranked nothing. The plan's §3 did not surface it (it is the D3 read path, one file
outside the enumerated scope). ranked_paths and run_mc3 gained an optional memberships
parameter: pass it and each hit contributes every path it currently occupies — the honest reading
of a shared atom; omit it and the pre-split behavior is byte-identical. Leaving the instrument
knowingly broken was the alternative.

tests/unit/test_code_lineage.py is also outside §5's list and was repaired for the same reason
(it constructs CodeCorpusSync and asserts stored (source_path, digest)).

Known and sequenced, not a regression: the daemon's incompleteness probe
(ops/lifecycle/launcher.py:_code_backfill_incomplete) still reads (source_path, digest) off
vector rows. ops/ is out of scope by the plan and §6's re-home is bp-153's work — but it must
land before a rebuilt store exists or it false-positives forever (finding-0166's named
falsifier). Recorded in the journal's "For bp-153" section.

---

Verification

Each gate leg run separately (never &&-chained; leg 3 exits 1 by design). Counts reported exactly
as observed.

leg	result
uv run ruff check .	All checks passed!
uv run mypy core agents eval ops scheduler scripts	Success: no issues found in 263 source files
uv run mypy (argless)	Found 69 errors in 20 files (checked 565 source files) — exits 1 at the tests/-baseline, tail == 69, unmoved
uv run python -m ops.type_gate	Tier-2 membership OK · bare-ignore scan OK · one parked, non-fatal raw-shim report in tests/unit/test_restart_trustworthy.py (pre-existing, finding-0223)
uv run python scripts/check_imports.py	Import firewall (I2) OK · worker boundary (tier 4) OK

The "core/kernel/** imports no core.stores.memberships" claim is not check_imports.py's —
that scanner is the network/zone firewall. It is
test_memberships.py::test_the_membership_store_is_never_imported_by_the_kernel, which scans the
kernel tree and carries a negative control (the scanner is shown to fire on code_corpus.py, which
does import it) so the empty result is a fact rather than a broken check.
tests/unit/test_inner_ring.py is green, so memberships.py computes OUTER and the fixed point is
unmoved — no INNER edit, and none would have been legal.
| uv run pytest -q | 5 failed, 2450 passed, 15 skipped in 369.27s |

The 5 failures are exactly the three known-red-locally classes, unchanged from the baseline —
no new red, and +23 passing:

FAILED tests/e2e/test_dream_v2_live.py::test_dream_v2_synthesizes_grounded_themes_live
FAILED tests/integration/test_worktree_enforcement.py::test_a_deny_cross_worktree
FAILED tests/integration/test_worktree_enforcement.py::test_c_unsafe_direction_narrow_not_loosened
FAILED tests/integration/test_worktree_enforcement.py::test_d_no_pointer_is_no_plan_not_main_fallback
FAILED tests/unit/test_core_self_containment.py::test_core_imports_nothing_outside_core

Baseline on the clean base (68d8d39), before any edit: 5 failed, 2427 passed, 15 skipped —
exactly the three known-red-locally classes: the finding-0103 core-self-containment ratchet,
tests/e2e/test_dream_v2_live.py, and tests/integration/test_worktree_enforcement.py ×3
(issue #13 / finding-0280). Both runs used the same worktree and the same uv sync --extra dev
environment, so the two numbers are directly comparable.

Refs #34 (the A2 coordinate reading, now an assertion rather than a docstring), #27 (the design's
blessing and the embedder sub-confirmation). Files #37 as a consequence this plan surfaced but was
scoped out of fixing.

🤖 Generated with Claude Code

https://claude.ai/code/session_011LZZQPyGsoeGL73cbbEp3U