Skip to content

Support version metadata when writing a dataset #514

Description

@changhiskhan

Motivation

Version metadata can be used for debugging and lineage tracking

Proto definition

  1. Deprecate https://github.com/lancedb/lance/blob/cb092db625baade0b5568c1a98039e5a991d48e8/protos/table.proto#L134-L139
  2. Make it an optional field instead

Write

  1. Add Rust struct for VersionAuxData (https://github.com/eto-ai/lance/blob/main/protos/format.proto#L79) along with pb<>rust conversion traits
  2. Write the aux data section similar to how index metadata is written (https://github.com/eto-ai/lance/blob/main/rust/src/io/writer.rs#L79). write_manifest should take additional version metadata as parameter
    note make sure the version_aux_data file position is tested

Read

  1. Add fn to read version aux data in the Version struct (https://github.com/eto-ai/lance/blob/main/rust/src/dataset.rs#L66)
  2. Instead of returning a PyDict for each version (https://github.com/eto-ai/lance/blob/main/python/src/dataset.rs#L182), create a pyo3 struct Version that is exposed to python. This struct should have a function to read the aux data and cache it in memory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-pythonPython bindingsenhancementNew feature or requesthelp wantedExtra attention is neededrustRust related tasks

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions