You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a bulk / batch upsert (insert-or-replace by external_id) API. Today every upsert path is single-record, and add_manyrejects duplicate external_ids rather than merging — so there is no way to idempotently upsert a batch of records in one operation.
General callers syncing a batch from an upstream system want the same insert-or-replace semantics upsert_by_external_id already provides, but for many records at once without N round-trips.
core: upsert_many_by_external_id(records) that applies insert-or-replace per external_id in one logical operation, reusing the existing append-only supersession semantics (supersedes_id set on the successor, original hidden from default reads).
Define batch semantics: duplicate external_ids within the batch (reject vs. last-wins — recommend reject for parity with add_many), partial-failure behavior (all-or-nothing vs. per-record results), and a returned per-record UpsertResult (inserted vs. replaced, new id).
Plumb through Python (upsert_many), REST (batch endpoint or array body), and the client.
A user can upsert a batch of records keyed by external_id in one call: new records inserted, existing visible records replaced (superseded), in a single version bump where possible.
Per-record results indicate inserted vs. replaced and the resulting id.
Within-batch duplicate external_id handling is defined and tested.
Parity across Python / core / REST / client.
Tests cover insert, replace, mixed batches, and idempotent re-application.
Notes
Enabler for #97 (idempotent bulk + streaming ingestion). Builds on upsert_by_external_id (store.rs:453) and the batch-write path (#54). Best landed together with or after #100 (indexed uniqueness validation) to avoid O(n²) cost.
Summary
Add a bulk / batch upsert (insert-or-replace by
external_id) API. Today every upsert path is single-record, andadd_manyrejects duplicateexternal_ids rather than merging — so there is no way to idempotently upsert a batch of records in one operation.Motivation
add_manyerrors on existingexternal_ids (validate_unique_ids,store.rs:688), so adapters currently have no batch path to "insert new, replace existing."upsert_by_external_idalready provides, but for many records at once without N round-trips.Current state (all single-record)
upsert_by_external_id(store.rs:453)Context.upsert(api.py:510)upsert_record(crates/lance-context-server/src/routes/records.rs:65)api/src/lib.rs:37,core/src/api_impl.rs:40,unified.rs:125, clientlib.rs:61Proposed work
upsert_many_by_external_id(records)that applies insert-or-replace perexternal_idin one logical operation, reusing the existing append-only supersession semantics (supersedes_idset on the successor, original hidden from default reads).external_ids within the batch (reject vs. last-wins — recommend reject for parity withadd_many), partial-failure behavior (all-or-nothing vs. per-record results), and a returned per-recordUpsertResult(inserted vs. replaced, new id).upsert_many), REST (batch endpoint or array body), and the client.Acceptance criteria
external_idin one call: new records inserted, existing visible records replaced (superseded), in a single version bump where possible.external_idhandling is defined and tested.Notes
Enabler for #97 (idempotent bulk + streaming ingestion). Builds on
upsert_by_external_id(store.rs:453) and the batch-write path (#54). Best landed together with or after #100 (indexed uniqueness validation) to avoid O(n²) cost.