Repository navigation
Python Getting Started
This page makes a table, adds datasets and reads them back. Each code block shows the output of a real run.
pip install lance_ndThe package needs Python 3.10 or later. It installs xarray, numpy,
pyarrow and nd-arrow-array.
- A table is a folder, such as
profiles.lance, on disk or in an object store. - Each row of the table is one
xarray.Dataset. - Each commit makes a new version of the table. A reader sees one version, so a reader never sees half of a commit.
- Lance stores the rows in fragments, which are files of about 128 MiB.
The examples on these pages use CTD profiles. Each profile is one dataset: a temperature and a salinity over depth, a position and a time. The global attributes give the station and the cruise.
import numpy as np
import pandas as pd
import xarray as xr
import lance_nd
def profile(station, cruise, lat, lon, levels, time):
depth = np.arange(levels, dtype="float64") * 10.0
return xr.Dataset(
{
"TEMP": ("depth", 15.0 - depth / 100.0, {"units": "degree_Celsius"}),
"PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
"LATITUDE": ((), lat),
"LONGITUDE": ((), lon),
"TIME": ((), pd.Timestamp(time)),
},
coords={"depth": ("depth", depth, {"units": "m"})},
attrs={"station": station, "cruise": cruise},
)The datasets of a table do not have to be the same. They can have
different sizes, variables and attributes. Real files work the same way:
open each file with xr.open_dataset(path).
lance_nd.create() gives a writer. The writer commits all datasets at the
end of the with block.
with lance_nd.create("profiles.lance", stats=True) as writer:
writer.write(profile("A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01"))
writer.write(profile("A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02"))
table = writer.table
print(table.count_rows(), table.version)2 1
lance_nd.append() gives a writer for a table that exists.
with lance_nd.append("profiles.lance") as writer:
writer.write(profile("B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01"))
table = lance_nd.open("profiles.lance")
print(table.count_rows(), table.version)3 2
Table.scan() reads the datasets. to_xarray() gives each dataset back.
for batch in table.scan():
ds = batch.to_xarray()
print(ds.attrs["station"], ds.sizes["depth"], float(ds.TEMP.max()))A1 50 15.0
A2 80 15.0
B1 60 15.0
print(next(iter(table.scan(limit=1))).to_xarray())<xarray.Dataset> Size: 1kB
Dimensions: (depth: 50)
Coordinates:
* depth (depth) float64 400B 0.0 10.0 20.0 30.0 ... 470.0 480.0 490.0
Data variables:
TEMP (depth) float64 400B 15.0 14.9 14.8 14.7 ... 10.4 10.3 10.2 10.1
PSAL (depth) float64 400B 35.0 35.01 35.02 35.03 ... 35.47 35.48 35.49
LATITUDE float64 8B 52.5
LONGITUDE float64 8B 4.0
TIME datetime64[us] 8B 2026-06-01
Attributes:
station: A1
cruise: cruise-1
The user guide explains each task in its own chapter: create, add, read, delete, replace, compact and versions.
Concepts
Rust
Python