Uniqueness validation full-scans the dataset on every write (O(n²) for incremental/bulk appends)

[dcfocus]

Summary

Every write validates id / external_id uniqueness by listing and deserializing the entire dataset, even though an id scalar index (id_idx) already exists and is used for point lookups elsewhere. This makes each add / add_many / upsert / update O(N) in store size, so incremental or streaming appends are O(N²). Validation should use indexed lookups instead of a full scan.

Where

• validate_unique_ids (store.rs:670) — called by add (store.rs:369), so it runs on every add / add_many. After O(batch) within-batch checks it does list_with_options(None, None, LifecycleQueryOptions::new(true, true)) (store.rs:699) — a full scan over all records (including expired/retired/tombstoned), deserializing each via batch_to_records, then linear-scans for id / external_id collisions.
• validate_new_record_id (store.rs:724) — called by the upsert (store.rs:478) and update (store.rs:632) paths; same full-scan via list_with_options(...) (store.rs:726).

Impact

• Each write is O(N) time and O(N) memory (it materializes the whole dataset into a Vec<ContextRecord>).
• Adding N records incrementally, or streaming in many small commits (see Ingest raw interaction logs into lance-context records (batch + streaming) #97), is O(N²). This is the validation-cost constraint already flagged in Ingest raw interaction logs into lance-context records (batch + streaming) #97's streaming design notes.
• Cost grows with total history, not just live rows, because the scan includes expired/retired/tombstoned records (include_expired + include_retired = true).

The fix

An id scalar index already exists (id_idx / ID_INDEX_NAME, created via ensure_id_index, store.rs:349, store.rs:1246) and is used for indexed point lookups in get (store.rs:792: scanner.filter("id = '...'").limit(1)). Validation should reuse that path:

• For each candidate, do an indexed existence check (the get / get_by_id filtered-lookup style), or
• Issue a single filtered scan with id IN (...) / external_id IN (...) over only the candidate keys, projecting just the id / external_id columns (no full-record deserialize).
• Keep the existing within-batch duplicate detection (already O(batch), fine as-is).
• external_id has no dedicated index today — either add a scalar index on external_id, or validate it via a projected filtered scan rather than a full deserialize.

Acceptance criteria

• add / add_many / upsert / update validate uniqueness without listing + deserializing the entire dataset.
• Uniqueness-validation cost is sublinear in store size for a bounded batch (indexed lookups or one projected IN scan), not O(N) per call.
• Existing behavior preserved: duplicate id / external_id within a batch, and against existing records (including expired/retired/tombstoned), are still rejected with the same errors.
• A benchmark/test shows append cost no longer grows linearly with store size (appending into a large store is ~constant per call rather than O(N)).

Notes

• Semantics to preserve (or decide explicitly): validation currently considers expired/retired/tombstoned records, so an external_id collision against a tombstoned/superseded record is rejected today. Keep that, or call out any change.
• Unblocks efficient bulk + streaming ingestion (Ingest raw interaction logs into lance-context records (batch + streaming) #97) and a future bulk-upsert-by-external_id path.