Skip to content

Add bulk / batch upsert (insert-or-replace by external_id) #102

Description

@dcfocus

Summary

Add a bulk / batch upsert (insert-or-replace by external_id) API. Today every upsert path is single-record, and add_many rejects duplicate external_ids rather than merging — so there is no way to idempotently upsert a batch of records in one operation.

Motivation

  • Bulk/streaming ingestion (Ingest raw interaction logs into lance-context records (batch + streaming) #97) needs idempotent re-ingestion: add_many errors on existing external_ids (validate_unique_ids, store.rs:688), so adapters currently have no batch path to "insert new, replace existing."
  • General callers syncing a batch from an upstream system want the same insert-or-replace semantics upsert_by_external_id already provides, but for many records at once without N round-trips.

Current state (all single-record)

  • core: upsert_by_external_id (store.rs:453)
  • python: Context.upsert (api.py:510)
  • REST: upsert_record (crates/lance-context-server/src/routes/records.rs:65)
  • trait/dispatch: api/src/lib.rs:37, core/src/api_impl.rs:40, unified.rs:125, client lib.rs:61

Proposed work

  • core: upsert_many_by_external_id(records) that applies insert-or-replace per external_id in one logical operation, reusing the existing append-only supersession semantics (supersedes_id set on the successor, original hidden from default reads).
  • Define batch semantics: duplicate external_ids within the batch (reject vs. last-wins — recommend reject for parity with add_many), partial-failure behavior (all-or-nothing vs. per-record results), and a returned per-record UpsertResult (inserted vs. replaced, new id).
  • Plumb through Python (upsert_many), REST (batch endpoint or array body), and the client.
  • Should compose with the indexed-validation fix (Uniqueness validation full-scans the dataset on every write (O(n²) for incremental/bulk appends) #100) so batch upsert does not full-scan per record.

Acceptance criteria

  • A user can upsert a batch of records keyed by external_id in one call: new records inserted, existing visible records replaced (superseded), in a single version bump where possible.
  • Per-record results indicate inserted vs. replaced and the resulting id.
  • Within-batch duplicate external_id handling is defined and tested.
  • Parity across Python / core / REST / client.
  • Tests cover insert, replace, mixed batches, and idempotent re-application.

Notes

Enabler for #97 (idempotent bulk + streaming ingestion). Builds on upsert_by_external_id (store.rs:453) and the batch-write path (#54). Best landed together with or after #100 (indexed uniqueness validation) to avoid O(n²) cost.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions