BlobV2Descriptor Structured Extension — Fine-Grained Random Access for Multimodal Blob

[wenxuanguan]

Background & Motivation

Blob v2 already provides efficient storage and access for large binary objects. However, the current BlobV2Descriptor only records the blob's overall position and size (kind / position / size / blob_id / blob_uri) and is unaware of the blob's internal structure.

This limits Lance's applicability in multimodal scenarios:

• Video: extracting the N-th frame necessitates a decoding utility (e.g., ffmpeg) to access both the target frame and its reference frames, incurring multiple I/O accesses and precluding batch retrieval of the target frame sequence within a single I/O operation.
• Image: cannot access a specific page of a multi-page image or a specific region of a huge image in one I/O pass.
• Audio: cannot on-demand extract a segment for a given time range within a single I/O operation.

Building on prior discussions, #7174 points out that the Blob v2 descriptor only expresses physical storage location and cannot support richer logical objects such as multiple rows sharing the same object, delta encoding, or chunked blobs. It proposes an Object Layer abstraction that decouples physical location from logical representation through a unified resolve → plan → read → decode flow, enabling domain-specific blob layouts (e.g., GOP-level video storage) and cross-row object sharing (I/O coalescing). #4320 points out that the Blob API treats video as unstructured binary data and cannot perceive video metadata (e.g., frame rate); it proposes a video encoding extension that pluggably supports "encode-on-write" and "background progressive rewrite (at GOP level)" while retaining the native strengths of columnar storage.

I'd like to suggest a minimal, version-gated (file version >= 2.3) BlobV2Descriptor extension on top of these discussions, so that Lance can perceive a blob's internal structure and support fine-grained random access, while keeping the Lance core free of any codec dependency.

Core Design: A Minimal Extension to BlobV2Descriptor

On top of the existing 5 native fields, the only addition is a nullable sub-field blob_info, which carries the metadata and internal-structure index for various multimodal blobs. data_storage_version >= 2.3 allows the six-field layout. The existing 5 fields are unchanged. The full structure is defined as follows:

BlobV2Descriptor  
├── kind : u8 (unchanged)  
├── position : u64 (unchanged)  
├── size : u64 (unchanged)  
├── blob_id : u32 (unchanged)  
├── blob_uri : string (unchanged)  
└── blob_info : struct<...> (nullable) ← new  
    ├── blob_type : u8 (1=video / 2=audio / 3=image / ...)  
    ├── stream_info : binary (serialized metadata) 
    └── entries : binary (serialized position index)

blob_info is a fixed Arrow envelope in the format specification (stream_info : binary, entries : binary), not a client-defined nested Struct. The binary contents of these two children are user-defined. Lance is unaware of their content or semantics; it only stores, validates, projects, and copies them, while their parsing is defined by a user-provided plugin. For video, this might include stream-level metadata (e.g., fps) and GOP/Frame offsets. For image, it could contain page-level metadata such as page count, and index entries for a multi-page document.

Architecture: How It Maps Onto the Object Layer

This proposal follows the implementation path of #7174 Object Layer. It adheres to resolve → plan → read → decode, using video frame extraction as an example.

Mapping of the blob read path:

Object Layer stage	This proposal's implementation	Core value
resolve	Read blob_info and other metadata from the BlobV2Descriptor	Obtain video metadata plus GOP/frame info
plan	Extract the GOP/frame index from blob_info.entries, compute the physical read range — from the GOP's I-frame up to the target frame, including all decode dependencies	Cross-row I/O coalescing: multiple overlapping read ranges on the same GOP are merged into 1 Range GET
read	Issue the Range GET, return physical bytes	I/O volume shrinks from the full MP4 to a single GOP
decode	Call the user-provided plugin decoder implementation	Codec isolation: Lance core has no codec dependency

blob_info is inlined in the descriptor, so the number of I/O reads stays the same.

Custom Plugins

Media processing capabilities are implemented by user-provided plugins. Lance passes the source blob and opaque configuration through a unified plugin interface, and receives the processed byte stream, stream_info, and entries produced by the plugin. Lance is unaware of the data type handled by the plugin and does not parse the contents of the two blob_info children.

On reads, the plugin parses stream_info and entries according to its own definitions, plans the required byte ranges, and decodes the bytes returned by Lance. lance-media-runtime is responsible for loading the plugin; Lance core only handles data storage and range reads, with no built-in codec dependencies such as FFmpeg.

On writes, do not extend write_dataset. Add write_dataset_with_blob_info(..., media_options={column_path: (library_path, config_binary)}). Configured Packed/Dedicated columns (version >= 2.3) go through MediaProvider::ingest: write the processed byte stream (e.g. GOP) and return stream_info / entries. Unconfigured columns and Inline keep the original path with blob_info = null. No video frame decoding or re-encoding is involved.

The boundary of changes is clear:

Layer	Change type	Details	Core API change?
File Format	✅ Version-gated extension (2.3)	BlobV2Descriptor gains 1 nullable sub-field blob_info (stream_info : binary, entries : binary). Used only when file version >= 2.3. The existing 5 fields keep their semantics. Old readers must fail closed on six fields.	Format revision; 5-field contract unchanged
Table Format	❌ Zero change	Manifest/Fragment/DataFile are not touched at all; the existing 5 fields of BlobV2Descriptor keep their semantics; no new feature gate; no new Arrow extension type.	No core API change
Index Formats	❌ Zero change	The frame/GOP position index is descriptor sub-field data (row-level data) and does not belong to the table-level index spec; existing vector/scalar indexes are fully compatible.	No core API change
Catalog Specs	❌ Zero change	Reuse the existing _blobs/ directory and .blob files; no new directory/file type.	No API change
Namespace Client Spec	✅ New APIs	Do not change write_dataset / take_blobs / read_blobs. Add write_dataset_with_blob_info and extract_frame.	API addition

---

API Description

Using video as an example, this proposal adds the following two user-facing APIs:

• write_dataset_with_blob_info(..., media_options=...): a generic blob_info write API that is not tied to video or any specific media type. For Packed/Dedicated columns configured with a plugin, it invokes the plugin to process the input bytes, stores the plugin's output byte stream, and writes the returned stream_info and entries into blob_info.
• extract_frame(...): extracts frames from video blobs with blob_info. It invokes the plugin to parse stream_info and entries, reads the required byte ranges according to the index, and returns the frames decoded by the plugin.

The existing write_dataset, take_blobs, read_blobs, and BlobFile APIs gain no new parameters and retain their existing semantics.

Compatibility

Taking video frame-extraction scenarios as an example. Stored bytes, blob_info, and decoded frames are three different logical values: take_blobs / read_blobs return stored bytes only; extract_frame returns frames.

• write_dataset: unchanged. 5-field, no blob_info, no ingest.
• write_dataset_with_blob_info: requires version >= 2.3; configured columns write ingest bytes + blob_info; unconfigured / Inline keep blob_info = null.
• take_blobs / read_blobs / BlobFile: version <= 2.2 or blob_info null → native logic; blob_info != null → explicit error (not an MP4).
• extract_frame: only rows written by this proposal with blob_info != null; historical MP4 (<= 2.2 or blob_info null) is not supported.
• Old readers without 2.3 must fail explicitly on six-field / GOP bytes; they must not treat them as ordinary blobs.
• Compaction copies blob_info with the descriptor and does not re-ingest.

The proposal lets the plugin decompose the source container and stores the processed byte stream (e.g. GOP) in the Lance Dataset. Since original container metadata is stripped during ingest, legacy interfaces cannot correctly interpret these bytes or reproduce a valid MP4, so they error instead of returning them.

Performance Gains

Taking video frame extraction as an example, the blob_info extension reduces the number of I/O operations by an order of magnitude compared to the BlobFile-based random access approach, thereby significantly lowering the I/O cost for multimodal frame-extraction workloads.

End-to-End Latency Comparison

Test setup: MP4 video stored on MinIO (133.4 MB, H.264, fps 25, 40 min duration); Ten 500 ms time windows are randomly generated, and ten corresponding keyframe extraction operations are performed.The test implementation follows the official Lance lazy video decoding example: example-decode-video-frames-lazily. In that example, the process_frame function performs RGB conversion and materializes the frame as an ndarray.

This proposal's extraction flow: extract_frame locates and retrieves the target GOP from the blob according to the time interval, and then decodes to all keyframes using the media plugin.

Baseline extraction flow: Using take_blobs to fetch the BlobFile, a container is built from it. The container is then seeked to the closest keyframe position based on the given start time, and all keyframes are subsequently extracted in sequential order.

Results:

Metric	This proposal	Baseline	Improvement
Avg latency	17.4 ms	67.3 ms	2.86x

Performance Breakdown

This proposal decodes from the GOP's keyframe sequentially to the target frame; the baseline computes a seek to the preceding keyframe from the frame number and fps (and related info), then decodes sequentially to the target frame. Decode times are close — the performance gap mainly comes from blob data reads.

Comparing the I/O traffic of reading blob data, the difference is significant:

Metric	This proposal	Baseline	Read reduction
Total Input Size	~1.98M	~13.91M	~6x
Number of Reads	10	183	~17x

Other approaches

Other approaches can achieve similar results, such as storing the internal-structure index in a separate user-table column; this adds one sequential projection and requires users to keep the blob and index column synchronized across append / compact / merge. Inlining blob_info in the descriptor keeps physical location and internal index as one piece of row-level metadata, reducing logical reads from 3 to 2.

Test setup: on remote MinIO, 30 real H.264 MP4s (about 7 MB each, 30 s duration, fps 25, 720P/1080P) with 30 real frame PTS values. The full flow is: project descriptor / blob_info → rebuild BlobEntry → plan GOP range → read GOP payload; timing stops when the payload is in memory, with no frame decode. The inlined layout obtains descriptor + blob_info in one projection; the separate-column layout projects the index column first, then the descriptor.

Metric	This proposal	Separate column	Latency reduction
Avg latency (full flow)	1018.631 ms	1079.998 ms	5.68%
Avg latency (excl. GOP payload read)	97.876 ms	124.185 ms	21.18%

[Xuanwo]

Precomputing the media index makes sense to me. Could we compare this with keeping the index in a regular column and projecting it alongside the blob descriptor? We already have read_blob_ranges for batching and coalescing, so I’d like to understand what moving the index into the descriptor buys us.

If we do need storage-level support, could this fit into the Object Layer design? That seems like a natural home for representation-specific metadata.

[wenxuanguan]

Thanks a lot for the review.

Could we compare this with keeping the index in a regular column and projecting it alongside the blob descriptor? We already have read_blob_ranges for batching and coalescing, so I’d like to understand what moving the index into the descriptor buys us.

This is indeed an important comparison dimension. What I need to clarify is read_blob_ranges solves read-time I/O coalescing (merging multiple blob range requests), but it does not solve two problems:

1. Cross-column consistency. If the index lives in a separate column, any mutation to the media data (e.g., re-encoding) requires coordinating updates across two columns. Lance currently has no cross-column transactional consistency guarantee, so a failed or partial index update would leave the dataset in an inconsistent state. By embedding the index in the descriptor, the index is committed with the DataFile write, so the index and data are consistent without any additional synchronization mechanism.

2. Extra I/O overhead. With a separate column, accessing a fine-grained media unit(for example, video frame extraction) requires two reads: first the media index column to get offsets, parse the offsets, and then the media column to fetch the frame. In contrast, inlining the index in the blob descriptor means the reader gets both the offset and the blob location in a single metadata read — zero additional I/O. Our benchmarks in the proposal (Section Other approaches) shows the latency comparison for the video frame extraction case:

Metric	This proposal	Separate column	Latency reduction
Avg latency (full flow)	1018.631 ms	1079.998 ms	5.68%
Avg latency (excl. video frame decode)	97.876 ms	124.185 ms	21.18%

If we do need storage-level support, could this fit into the #7174? That seems like a natural home for representation-specific metadata.

Yes, absolutely — #7174 is indeed the natural home for this extension. Here's how the pieces align:

Resolve stage: Read blob_info from the BlobV2Descriptor
Plan stage: Extract the index, compute the physical read range, and perform I/O coalescing.
Read stage: Issue the Range GET.
Decode stage: Decoding is completed by a user-provided plugin.

The current design and implementation (including benchmarks) already follow the #7174 direction, and I'm happy to work with you to ensure this fits smoothly into the #7174 design.