Skip to content

Python Getting Started

Robin Kooyman edited this page Nov 10, 2026 · 3 revisions

This page makes a table, adds datasets and reads them back. Each code block shows the output of a real run.

Install

pip install lance_nd

The package needs Python 3.10 or later. It installs xarray, numpy, pyarrow and nd-arrow-array.

The terms

  • A table is a folder, such as profiles.lance, on disk or in an object store.
  • Each row of the table is one xarray.Dataset.
  • Each commit makes a new version of the table. A reader sees one version, so a reader never sees half of a commit.
  • Lance stores the rows in fragments, which are files of about 128 MiB.

The example data

The examples on these pages use CTD profiles. Each profile is one dataset: a temperature and a salinity over depth, a position and a time. The global attributes give the station and the cruise.

import numpy as np
import pandas as pd
import xarray as xr

import lance_nd


def profile(station, cruise, lat, lon, levels, time):
    depth = np.arange(levels, dtype="float64") * 10.0
    return xr.Dataset(
        {
            "TEMP": ("depth", 15.0 - depth / 100.0, {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )

The datasets of a table do not have to be the same. They can have different sizes, variables and attributes. Real files work the same way: open each file with xr.open_dataset(path).

Make a table

lance_nd.create() gives a writer. The writer commits all datasets at the end of the with block.

with lance_nd.create("profiles.lance", stats=True) as writer:
    writer.write(profile("A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01"))
    writer.write(profile("A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02"))

table = writer.table
print(table.count_rows(), table.version)
2 1

Add datasets

lance_nd.append() gives a writer for a table that exists.

with lance_nd.append("profiles.lance") as writer:
    writer.write(profile("B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01"))

table = lance_nd.open("profiles.lance")
print(table.count_rows(), table.version)
3 2

Read datasets

Table.scan() reads the datasets. to_xarray() gives each dataset back.

for batch in table.scan():
    ds = batch.to_xarray()
    print(ds.attrs["station"], ds.sizes["depth"], float(ds.TEMP.max()))
A1 50 15.0
A2 80 15.0
B1 60 15.0
print(next(iter(table.scan(limit=1))).to_xarray())
<xarray.Dataset> Size: 1kB
Dimensions:    (depth: 50)
Coordinates:
  * depth      (depth) float64 400B 0.0 10.0 20.0 30.0 ... 470.0 480.0 490.0
Data variables:
    TEMP       (depth) float64 400B 15.0 14.9 14.8 14.7 ... 10.4 10.3 10.2 10.1
    PSAL       (depth) float64 400B 35.0 35.01 35.02 35.03 ... 35.47 35.48 35.49
    LATITUDE   float64 8B 52.5
    LONGITUDE  float64 8B 4.0
    TIME       datetime64[us] 8B 2026-06-01
Attributes:
    station:  A1
    cruise:   cruise-1

Next steps

The user guide explains each task in its own chapter: create, add, read, delete, replace, compact and versions.

Clone this wiki locally