feat: add BatchCommitTables API for atomic multi-table commit operations#315
Conversation
github-actions
bot
added
enhancement
|
you should edit the model in |
XuQianJin-Stars
force-pushed
the
feat/delete-table-versions-multi-table
branch
from
c63ab77 to
48a71e8
Compare
XuQianJin-Stars
changed the title
docs/src/rest.yaml
Outdated
| Supported operation types: | ||
| - `declare_table`: Declare (reserve) a new table | ||
| - `create_table_version`: Create a new version entry for a table | ||
| - `delete_table_versions`: Delete version ranges from a table |
There was a problem hiding this comment.
BatchDeleteTableVersions
docs/src/rest.yaml
Outdated
| single atomic transaction at the metadata layer. | ||
|
|
||
| Supported operation types: | ||
| - `declare_table`: Declare (reserve) a new table |
There was a problem hiding this comment.
nit: these should be PascalCase, like DeclareTable
docs/src/rest.yaml
Outdated
| - `deregister_table`: Deregister (soft-delete) a table | ||
|
|
||
| All operations are committed atomically: either all succeed or none are applied. | ||
| Physical file operations (e.g., writing manifest files, deleting version files) |
There was a problem hiding this comment.
this is not true, we still want files to be the eventual source of truth. Let's just remove this line.
docs/src/rest.yaml
Outdated
| Supports deleting ranges of versions for efficient bulk cleanup. | ||
| required: | ||
| - ranges | ||
| This request supports two modes: |
There was a problem hiding this comment.
these changes seem wrong? Looks like they come from the last PR for BatchDeleteTableVersions
docs/src/rest.yaml
Outdated
| description: | | ||
| A single operation within a batch commit. | ||
| Uses a discriminator on the `type` field to determine the operation kind. | ||
| oneOf: |
There was a problem hiding this comment.
we try to avoid the use of oneOf, because it causes a ton of issues in codegen. Prefer just simple case that lists all operations directly, see for example: https://github.com/lance-format/lance-namespace/blob/main/docs/src/rest.yaml#L2811
vector:
type: object
nullable: true
description: Query vector(s) for similarity search. Provide either single_vector or multi_vector, not both.
properties:
single_vector:
type: array
items:
type: number
format: float
description: Single query vector
multi_vector:
type: array
items:
type: array
items:
type: number
format: float
description: Multiple query vectors for batch search
vector can only be one of single_vector or multi_vector, but we still list them all as just optional properties, and we just check at client side that only one of them is set.
docs/src/rest.yaml
Outdated
| A single operation within a batch commit. | ||
| Uses a discriminator on the `type` field to determine the operation kind. | ||
| oneOf: | ||
| - $ref: "#/components/schemas/CommitTableOperationDeclareTable" |
There was a problem hiding this comment.
why not directly reference DeclareTableRequest? (similar question to the next few lines)
jackye1995
left a comment
jackye1995
left a comment
There was a problem hiding this comment.
looks good to me! Can you also update the Java and Python LanceNamespace interfaces? Also we just added LanceNamespaceAsync in Java as well. All these high level interfaces need to be updated manually. Once that is done we should be good!
Edit rest.yaml to add DeleteTableVersionsEntry schema and entries field to BatchDeleteTableVersionsRequest. All language clients (Rust, Java, Python) are regenerated using make gen. This enables transactional deletion of table versions across multiple tables in a single operation. When entries is provided, the legacy single-table id + ranges fields are ignored.
Add BatchCommitTables endpoint (POST /v1/table/batch-commit) to support atomic multi-table transactions with discriminated union types: - CommitTableOperation: declare_table, create_table_version, delete_table_versions, deregister_table - CommitTableResult: corresponding result types for each operation Generated Rust, Python, and Java client code via make gen.
- Add batchCommitTables to Java sync LanceNamespace interface - Add batchCommitTables to Java async LanceNamespaceAsync interface - Add batch_commit_tables to Python LanceNamespace ABC - Regenerate async-client to include BatchCommitTables models
XuQianJin-Stars
force-pushed
the
feat/delete-table-versions-multi-table
branch
from
e61c7a0 to
7cd14d3
Compare
2 participants
Summary
Add
BatchCommitTablesAPI (POST /v1/table/batch-commit) for atomic multi-table commit operations,and add
DeleteTableVersionsEntrymodel withentriesfield toBatchDeleteTableVersionsRequestfor multi-table version deletion.New API:
BatchCommitTablesEndpoint:
POST /v1/table/batch-commitAtomically commit a batch of mixed table operations within a single transaction at the metadata layer.
This is a generalized version of
BatchCreateTableVersionsthat supports mixed operation types.All operations are committed atomically: either all succeed or none are applied.
Physical file operations are best-effort — metadata is the source of truth.
New Models
Request:
BatchCommitTablesRequest— contains a list ofCommitTableOperationCommitTableOperation— discriminated union (viatypefield) supporting 4 operation types:CommitTableOperationDeclareTable(type: declare_table) — declare/reserve a new tableCommitTableOperationCreateTableVersion(type: create_table_version) — create a new version entry for a tableCommitTableOperationDeleteTableVersions(type: delete_table_versions) — delete version ranges from a tableCommitTableOperationDeregisterTable(type: deregister_table) — soft-delete a tableResponse:
BatchCommitTablesResponse— contains a list ofCommitTableResult(same order as request)CommitTableResult— discriminated union (viatypefield) with result types:CommitTableResultDeclareTableCommitTableResultCreateTableVersionCommitTableResultDeleteTableVersionsCommitTableResultDeregisterTableModified:
BatchDeleteTableVersionsRequestAdded multi-table support with optional
entriesfield:entries:Optional<List<DeleteTableVersionsEntry>>— multi-table transactional delete entriesDeleteTableVersionsEntrywith required fields:id(table identifier) andranges(version ranges)entriesis provided, legacy single-tableid+rangesfields are ignoredFiles Changed
docs/src/rest.yamlbatch_commit_tables_request.rs,batch_commit_tables_response.rs,commit_table_operation.rs,commit_table_operation_*.rs(4),commit_table_result.rs,commit_table_result_*.rs(4),metadata_api.rs,table_api.rs,transaction_api.rsBatchCommitTablesRequest.java,BatchCommitTablesResponse.java,CommitTableOperation.java,CommitTableOperation*.java(4),CommitTableResult.java,CommitTableResult*.java(4),MetadataApi.java,TableApi.java,TransactionApi.javaBatchCommitTablesRequest.java,BatchCommitTablesResponse.java,CommitTableOperation*.java(5),CommitTableResult*.java(5),TableApi.javabatch_commit_tables_request.py,batch_commit_tables_response.py,commit_table_operation*.py(5),commit_table_result*.py(5)delete_table_versions_entry.rsDeleteTableVersionsEntry.javaDeleteTableVersionsEntry.javadelete_table_versions_entry.pymod.rs,__init__.py,README.md,openapi.yaml,BatchDeleteTableVersionsRequest.*files updatedTotal: 141 files changed, 14015 insertions(+), 190 deletions(-)
All generated code produced via
make genfrom the updatedrest.yamlspec.