feat(core): user-defined schemas via SchemaSpec and GenericStore

[beinan]

Part 2 of #214, on top of the consolidated StorageBase from #215–#217.

Users can now declare their own schema instead of picking one of the three built-in ones — while add and the WAL merge behave identically to RolloutStore, because they are literally the same code path, not a reimplementation.

Three pieces

schema_spec — schema as a value

The built-in stores hard-code an Arrow schema and then repeat its field list across five to seven places (schema fn, builders, decoder, projection, filters, merge key, index name). A SchemaSpec is declared once at creation and persisted with the store.

Validation enforces what the storage layer actually assumes:

• id is mandatory — Utf8, non-nullable, tagged unenforced-primary-key. It is always the LSM merge key and the indexed column, which is what makes a retried append idempotent and a crashed merge safe to redo.
• Reserved names rejected — anything starting with _, so future Lance internals can't collide with a user column.
• Vector columns carry dim + metric, so an index can be built and the metric round-trips on open without being re-specified.
• Nested lists and lists-of-vectors rejected rather than silently mis-encoded.

generic_codec — schema-driven encode/decode

JSON rows ↔ Arrow, matched by name in both directions. Two deliberate choices:

• An undeclared key is an error, not a silent drop — a field-name typo fails at the write instead of becoming missing data discovered much later.
• Projected batches decode without their missing columns, which is what lets a blob-excluding scan round-trip.

This also fixes by construction a bug class the built-in stores have: they decode nested structs (relationships, state_metadata) positionally, so reordering fields silently mis-assigns data.

generic_store — the store

Delegates all storage behavior to StorageBase: resident writer, fence retry, surgical WAL drain, compaction with defer_index_remap, id index, LSM reads. seal_on_add picks rollout's deferred-seal profile (default) or read-your-write.

How users write it

store = GenericStore.open(uri, schema={
    "id":        {"type": "string", "nullable": False},
    "user_id":   {"type": "string"},
    "score":     {"type": "float32"},
    "embedding": {"type": "vector", "dim": 768, "metric": "cosine"},
    "video":     {"type": "binary", "blob": True},
})

store.add([{"id": "r1", "user_id": "u1", "score": 0.9}])   # nullable cols may be omitted
store.list()                                  # no video — never materialized
store.get("r1", columns=["video"])            # fetch the blob for one row

Blobs are inline, and that's deliberate

Per your point that this project exists to store big blobs: inline LargeBinary, never blob-v2 offload. The LSM read path has no blob-materialization step, so an offloaded column reads back as None — which is exactly why RolloutStore stores artifacts inline today.

blob: true doesn't change where bytes live, it changes who pays for them: blob columns are dropped from default scan projections, so list never materializes them, and get(id, columns) fetches them per row.

I measured this path directly before designing around it:

size	write	read	survives WAL merge
5 MB	80 ms	42 ms	✓
40 MB	483 ms	262 ms	✓
120 MB	1.39 s	760 ms	✓

The test suite pins a 4 MB round-trip through a merge (kept small so CI stays fast).

⚠️ Worth knowing for the 100 MB+ case: max_memtable_size defaults to 256 MB, so two such rows trigger a seal. That's not breakage — the count/time merge triggers bound the resulting generations — but it does raise seal frequency. ShardWriterConfig exposes the knob; no store surfaces it yet. Follow-up if it bites.

Schema is immutable in v1

Reopening with a different schema is rejected, not silently reinterpreted over existing data. Evolution needs a versioning story and is out of scope here.

Verification

28 new tests: 8 spec validation, 9 codec round-trip, 11 store behavior — including that duplicate ids dedup to the newest write (proving id really is the merge key), deferred seal matches rollout's visibility semantics, and generations merge into the base table.

209 core lib tests pass (up from 181), cargo clippy --workspace --all-targets clean.

Not in this PR

The server/client/Python surface — CreateContextRequest carrying a schema, generic record DTOs, deduping the six duplicated conversion functions, explicit column names in search/retrieve. This PR is the core engine; the API layer is the next one, and it's large enough to want its own review.

🤖 Generated with Claude Code