Table Layout

Data model

lance-nd stores the data model of nd-arrow-array. That repo defines the encoding. In short:

• An nd arrow array is one N-dimensional array. As an Arrow column, it is the extension type nd_arrow.array over a struct:

  Struct {
    values:    List<T>,       // the flat values, in C order
    dim_names: List<Utf8>,    // the name of each dimension
    dim_sizes: List<UInt32>,  // the size of each dimension
    attrs:     Struct { .. }, // optional attributes, such as units
  }
  

• An nd record batch is a set of named nd arrow arrays on one shared grid, plus optional global attributes in the struct column __global_attrs__.

In Rust, these are NdArrowArray and NdRecordBatch. See the nd-arrow FORMAT.md for the full rules.

The lance-nd table

A lance-nd table is a Lance table. Each table row is one nd record batch: a complete one, or one chunk of a larger one.

Column	Type	Nullable
one column per array name	nd_arrow.array	yes
__global_attrs__	nd_arrow.global_attributes	yes

For example, a table with three nd record batches:

Row	x	y	temperature	pressure	__global_attrs__
0	(x=3)	(y=4)	(x=3, y=4)	absent	{id: "a"}
1	absent	absent	absent	(station=3)	{id: "b"}
2	(x=2)	(y=4)	(x=2, y=4)	absent	{id: "c"}

Array columns

• The table has one column for each array name in any row.
• A null in a row means that the nd record batch of that row does not have that array. A reader leaves it out of the row.
• One array name has one value type in a table. A write that changes the type fails, unless WriteOptions::casting allows a cast, see Casting.

Schema evolution

A write can bring new array names, new attribute keys and new global attributes. lance-nd adds them to the table schema first, then appends the rows. Older rows read the new arrays as absent.

Lance can add nested fields only with file version 2.2 or later, so lance-nd creates tables with file version 2.2.

Rows and chunks

The format does not separate a complete nd record batch from a chunk, and it has no reserved columns for names or positions. Put them in the global attributes, for example id or chunk_offset, then filter on them:

__global_attrs__.id = 'a'

See FORMAT.md for the full specification of the table.