feat: expose user-defined schemas across the API, server, client and Python

[beinan]

Completes #214 Part 2. GenericStore was core-only after #218; this wires it through every layer so users can actually declare and use their own schema.

How it looks

from lance_context import GenericStore

store = GenericStore.open(uri, schema={
    "id":     {"type": "string", "nullable": False},   # required
    "user":   "string",                                 # shorthand
    "score":  "float32",
    "tags":   {"type": "list", "item": {"type": "string"}},
    "vec":    {"type": "vector", "dim": 768, "metric": "cosine"},
    "video":  {"type": "binary", "blob": True},
})

store.add([{"id": "r1", "user": "u1", "score": 0.9}])   # nullable cols may be omitted
store.list(filter="score > 0.5")                        # no video — never materialized
store.get("r1", columns=["video"])                      # fetch the blob for one row

Same over REST:

POST   /api/v1/generic                    create with a schema
GET    /api/v1/generic                    list
GET    /api/v1/generic/{name}             fetch identity + schema
DELETE /api/v1/generic/{name}
POST   /api/v1/generic/{name}/rows        append
GET    /api/v1/generic/{name}/rows        list (?filter, ?limit, ?offset)
GET    /api/v1/generic/{name}/rows/{id}   point read (?columns=video)
POST   /api/v1/generic/{name}/flush
POST   /api/v1/generic/{name}/merge-wal

SchemaSpec moves to the API crate

A schema is part of the wire contract — the server accepts one at creation, the client needs the same type to send it — so it belongs with the other request/response types rather than in the storage crate. It depends only on arrow-schema and serde, not on Lance. Core re-exports it, so existing paths are unchanged.

Its one tie to core (DistanceMetric::parse) becomes a local identifier check, with a comment noting the two must stay in sync.

No per-column DTO, on purpose

The schema is declared at runtime, so a static struct can't describe it. Rows are serde_json maps in both directions — the wire form is the row form, so there's no conversion layer to drift.

Two bugs the survey turned up

Nine duplicated conversion functions, not the six #214 counted. dto_to_relationship and relationship_to_dto each had a third copy in routes/rollouts.rs. All were byte-for-byte identical — pure copy-paste that happened to still be in sync. Now defined once in core and re-exported.

shutdown only closed rollout stores. Datagen and generic stores hold resident MemWAL writers too, so an unsealed memtable was left to the best-effort detached close in Drop — rows could stay durable-but-invisible until the next WAL replay. All three kinds now close deterministically. This one pre-existed for datagen; adding a fourth store kind would have quietly widened it.

Registry stores name and URI only

A generic store's schema lives in its own dataset. A second writable copy in the registry is exactly the divergence this whole refactor exists to remove, so I didn't add one. The cost is that listing can't report schemas without opening each dataset — hence GenericStoreInfo::schema is None in list responses and populated for single-store lookups. Called out explicitly rather than left to be discovered.

Verification

21 new tests — 6 server route tests (create/list/add/read round-trip, invalid schema rejected before anything is created, duplicate-create conflicts, undeclared column rejected, delete unregisters, deferred seal + WAL merge) and 15 Python tests.

Full suite: 201 core + 57 server + 18 api Rust tests, 197 Python tests, cargo clippy --workspace --all-targets clean, typos clean.

Follow-ups

• seal_on_add is not persisted, so a reopened store takes the server default. Worth persisting alongside the schema if it turns out to matter.
• The sweepers (spawn_global_sweeper, spawn_flush_sweeper) still only visit rollout stores — pre-existing for datagen, now also true for generic. Generic stores default to deferred seal, so they rely on an explicit flush or the merge-wal route until that's addressed.

🤖 Generated with Claude Code