Lance Blob v2 Object Layer Proposal

[Xuanwo]

Abstract

Blob v2 already decouples a blob’s logical value from its physical placement: a blob can be stored as Inline, Packed, Dedicated, or External. This proposal adds a more general Object Layer on top of these placements.

The Object Layer defines how a row-level blob descriptor resolves to a logical blob object:

row descriptor -> object reference -> read plan -> physical reads -> logical bytes

This allows Lance to express exact sharing, delta encoding, chunked blobs, and future domain-specific blob representations with a single model. As a first step, we only need to implement a minimal Raw Pack under the Packed placement; other pack types should be designed separately.

Problem

Today the Blob v2 descriptor is largely interpreted as a physical location:

kind, position, size, blob_id, blob_uri

For raw packed blobs this means:

kind     = Packed
blob_id  = packed sidecar id
position = raw byte offset
size     = raw byte length

This is sufficient for raw bytes, but there is no clean place to express richer blob representations:

• Multiple rows sharing the same logical blob object.
• A blob reconstructed from a base object plus a delta.
• A blob composed from reusable chunks.
• Workloads that need dedicated blob layouts and metadata.

These are object representation concerns, not new placements. They should not turn BlobKind into a codec enum, nor should they be modeled as row-to-row dependencies or new page layouts.

Design

Introduce an internal Object Layer into the Blob v2 read/write path. The descriptor is first resolved into a placement-specific object reference, then planning and reading happen through a uniform contract.

flowchart TD
    R["Row"] --> D["Blob v2 descriptor"]
    D --> O["BlobObjectRef"]
    O --> P{"Placement backend"}
    P --> I["Inline"]
    P --> K["Packed"]
    P --> X["Dedicated"]
    P --> E["External"]
    I --> B["logical bytes"]
    K --> B
    X --> B
    E --> B

Loading

The uniform contract can be expressed as:

resolve(descriptor) -> BlobObjectRef
plan(BlobObjectRef) -> BlobReadPlan
read(BlobReadPlan) -> physical bytes
decode(BlobReadPlan, physical bytes) -> logical bytes

A key invariant is:

plan_before_payload = true

If a representation requires dependencies, chunks, or codec metadata, that metadata must be available before reading the main payload. A reader must not read a delta payload first and only then discover it needs another object.

Placement boundaries

The Object Layer spans all Blob v2 placements, but the role of each placement differs.

Placement	Role	Boundary
Inline	data-file-local object	Keep simple for small/raw values; complex representations promote to Packed.
Packed	multi-object backend	Primary home for exact sharing, delta, chunking, and domain-specific packs.
Dedicated	single-object sidecar	Suitable for large independent objects; can exact-share, but not suited for delta chains.
External	foreign object reference or ingest source	Reference mode does not own the bytes; ingest mode re-enters the placement policy.

A critical boundary: BlobKind continues to express only placement. Representation details belong to the Object Layer backends.

Packed Backend and Raw Pack

The first concrete backend should be the Raw Pack for Packed.

In legacy raw packed blobs, position means a byte offset:

kind     = Packed
blob_id  = raw packed sidecar id
position = raw byte offset
size     = raw byte length

In object-backed packed blobs, position should be interpreted as an opaque backend reference:

kind     = Packed
blob_id  = packed backend object id
position = opaque backend reference
size     = logical decoded size
blob_uri = empty

The Raw Pack only needs to prove the new boundary holds:

entry_ref -> raw payload -> logical bytes

The Raw Pack can be implemented as an internal Lance file, or as a smaller indexed sidecar format. This proposal does not require all future packs to share the same schema.

The Raw Pack must not store its internal payload recursively as Blob v2. The pack itself is already a blob container.

Future Delta Pack, Chunked Pack, exact-share policy, source-revision pack, video-frame pack, and custom pack types should each have their own independent designs.

Compatibility

The Object Layer itself is an internal abstraction. A required feature is only needed when a backend changes the interpretation of existing descriptor fields.

For object-backed Packed, Lance will need a feature gate similar to:

blob_object_pack

Compatibility requirements:

• New readers continue to read existing Inline, raw Packed, Dedicated, and External blobs.
• Old readers must not silently interpret object-backed Packed data as raw packed byte slices.
• Object-backed Packed data must fail explicitly on readers that lack the required feature.
• Each backend representation can declare its own type, version, and required features.

[westonpace]

I think I get the gist of what you are saying. You are proposing a plugin extension point (the object layer?) whose responsibility is to take in a description and request and return data. As a result we should not always assume that position and size mean "offset into a file and number of bytes in that file". However, readers can be confident they will get the 5-value description struct and that there is an API to covert that struct into bytes at a later date.

I assume this means we will need metadata of some kind (pack file metadata, etc.) to convert from position to instructions into object storage?

To test my understanding let me think about the video case. With the existing capabilities I would just have one row per video and a column of type blob representing the video bytes. I could have another row maybe of type list which is all the "segment ids" in the video and another column of type list<list> which is all the frame ids per segment. To read a single frame I would do something like SELECT decode_video_frame(videos, <segment_id>, <frame_id>) FROM tbl. The logic for decoding the video would be in decode_video_frame.

With this new proposal a blob could be a frame in the video (output is an image) or it could be a segment of the video (output is a smaller video). I suppose I could even have three columns (video, segment, frame) all with type blob (all backed by the same data) and one row per frame. Reading a single frame would just be SELECT read_blob(frames) FROM tbl WHERE frame_id=<frame_id>. The logic for decoding the video would be in the object layer. I'm not suggesting we adopt this approach, just exploring it as an example.

I'm +1 on the proposal.

I think the main advantage of the object layer is that we move the decoding logic into this plugin layer and the user (whoever is writing the queries) doesn't have to keep track of complicated UDFs. They just use the same old read_blob function they've always used. This is a no-brainer I think for things like delta or chunked. I'm not entirely sold on the video case (I think UDFs probably work fine there) but being able to avoid something like the list and list<list> data types makes queries a lot easier to write.

[Xuanwo]

However, readers can be confident they will get the 5-value description struct and that there is an API to covert that struct into bytes at a later date.

Bingo! Giving a new meaning to offset and size to make it compatible to blob v2.

I assume this means we will need metadata of some kind (pack file metadata, etc.) to convert from position to instructions into object storage?

Yes! the packed blob will need to store some metadata in header or footer.

[hpvd]

+1

would be great to think this really universal/plug-able to enable other usecases where

1. individual objects are simply connected (belong to a group) or
2. objects are/could be derived from each other

like

• 1 main image and image variants with different aspect ratios (derived simply deterministic)
• 1 main image and different cropped variants (AI based selections)
• 1 main image (raw) and Variants in the usage format and resolutions (jpeg)
• 3D Model and images from different view-angels (jpeg)
• 3D-Model of a component and its different subparts (also 3D Models)
• 3D-Model and its slices (only save angel and offset)
• Vector based drawing and its derived jpeg representation (for further usage)
• ...

[DanielMao1]

+1 on this proposal, and I'd be happy to contribute PRs for it.

I want to add some context on the video case, since I've been working on storing video data in Lance (GOP-level blobs
instead of whole MP4 containers, mainly for robotics/embodied-AI workloads). In my experience the video case is
considerably more complex than "a video and its frames":

• Codec diversity matters. Different codecs and encoding parameters (GOP size, keyframe interval, intra-only vs. long-GOP)
  suit different access patterns, and frame-level read latency is dominated by keyframe spacing rather than file size. A
  single decode_video_frame UDF can't reasonably cover this space.
• Domain-specific formats are real. Robotics datasets often come with custom containers and per-sensor encodings, not just
  standard MP4/H.264. A pluggable decoding layer is exactly the right place to absorb that variety, instead of pushing it
  into query-side UDFs that every user has to get right.
• Low latency requires planning below the UDF boundary. This is the key point for me: a Python UDF only sees one row at a
  time and can't optimize I/O across rows. With the Object Layer's plan -> read -> decode contract, the reader can coalesce
  and deduplicate S3 range requests across many frame reads (e.g. many frames resolving to the same GOP/base object should
  trigger one fetch, not N). That kind of request batching is essential for low-latency random frame access over object
  storage, and it's only possible if the representation logic lives inside the read path rather than in user-space UDFs.

So while I agree the Raw Pack is the right minimal first step, I think the video/frame pack is a strong motivating case for
the Object Layer itself — happy to help design and implement that part when the time comes.