Skip to content

feat: support ingest mode for external blob writes#6356

Merged
Xuanwo merged 9 commits intomainfrom
xuanwo/external-blob-ingest-mode
Apr 2, 2026
Merged

feat: support ingest mode for external blob writes#6356
Xuanwo merged 9 commits intomainfrom
xuanwo/external-blob-ingest-mode

Conversation

@Xuanwo
Copy link
Copy Markdown
Collaborator

@Xuanwo Xuanwo commented Mar 31, 2026

This adds external_blob_mode="ingest" for blob v2 writes so datasets can import external blob bytes into Lance-managed storage instead of always persisting URI references. The write path now streams external sources through lance-io reader streams for packed and dedicated blobs while preserving inline materialization for small payloads.

Closes #6321.

@github-actions github-actions bot added enhancement New feature or request python labels Mar 31, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 31, 2026

@wjones127
Copy link
Copy Markdown
Contributor

It looks like there's a few Windows and linting issues to fix, but otherwise looks good

@Xuanwo Xuanwo merged commit 8b59eb6 into main Apr 2, 2026
28 checks passed
@Xuanwo Xuanwo deleted the xuanwo/external-blob-ingest-mode branch April 2, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support materializing external blob references on write

2 participants