Python Getting Started

This page makes a table from NetCDF files, adds a file and reads the datasets back. Each code block shows the output of a real run.

Install

pip install lance_nd

The package needs Python 3.10 or later. It installs xarray, numpy, pyarrow and nd-arrow-array. To open NetCDF files, also install netCDF4.

The terms

• A table is a folder, such as profiles.lance, on disk or in an object store.
• Each row of the table is one xarray.Dataset, for example the content of one NetCDF file.
• Each commit makes a new version of the table. A reader sees one version, so a reader never sees half of a commit.
• Lance stores the rows in fragments, which are files of about 128 MiB.

What you can write

You write xarray.Dataset objects. Each dataset becomes one row:

• A dataset from a file: xr.open_dataset("profile.nc"). Each format that xarray opens works, such as NetCDF, HDF5, Zarr or GRIB.
• A dataset that you make in memory with xr.Dataset(...).
• A list of datasets.

The datasets of one table do not have to be the same:

• They can have different variables, different dimensions and different sizes. A dataset without a variable has no value for it.
• They can have different attributes.
• One variable must have the same value type in all datasets, for example float64. See Add datasets.

The table stores the variables, the coordinates, their attributes and the global attributes. It does not store the encoding of a dataset.

The example files

The examples on these pages use small NetCDF files of CTD profiles. Each file has a temperature and a salinity over depth, a position and a time. The global attributes give the station and the cruise.

profiles/A1.nc  profiles/A2.nc  profiles/A3.nc   cruise-1
profiles/B1.nc  profiles/B2.nc  profiles/B3.nc   cruise-2
new/C1.nc       new/C2.nc       new/C3.nc        cruise-3

Use your own NetCDF files, or run this block to make the example files:

Make the example files

from pathlib import Path

import numpy as np
import pandas as pd
import xarray as xr


def write_profile(path, station, cruise, lat, lon, levels, time, temp_type="float64", extra=None):
    """Write one CTD profile as a NetCDF file."""
    depth = np.arange(levels, dtype="float64") * 10.0
    ds = xr.Dataset(
        {
            "TEMP": ("depth", (15.0 - depth / 100.0).astype(temp_type), {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )
    if extra:
        ds["DOXY"] = ("depth", np.full(levels, 250.0), {"units": "umol/kg"})
    Path(path).parent.mkdir(exist_ok=True)
    ds.to_netcdf(path)


# Six profiles of two cruises.
write_profile("profiles/A1.nc", "A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01")
write_profile("profiles/A2.nc", "A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02")
write_profile("profiles/A3.nc", "A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03")
write_profile("profiles/B1.nc", "B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01")
write_profile("profiles/B2.nc", "B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02")
write_profile("profiles/B3.nc", "B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03")
# Three new profiles of a third cruise.
write_profile("new/C1.nc", "C1", "cruise-3", 58.0, 6.0, 40, "2026-08-01")
write_profile("new/C2.nc", "C2", "cruise-3", 58.5, 6.5, 45, "2026-08-02")
write_profile("new/C3.nc", "C3", "cruise-3", 59.0, 7.0, 50, "2026-08-03")
# Corrected profiles, with other depth levels.
write_profile("corrected/A1.nc", "A1", "cruise-1", 52.5, 4.0, 55, "2026-06-01")
write_profile("corrected/A2.nc", "A2", "cruise-1", 53.0, 4.5, 90, "2026-06-02")
write_profile("corrected/B1.nc", "B1", "cruise-2", 54.0, 3.0, 65, "2026-07-01")
write_profile("corrected/B2.nc", "B2", "cruise-2", 55.5, 3.5, 75, "2026-07-02")
# Files that differ from the others.
write_profile("other/int_temp.nc", "D1", "cruise-4", 50.0, 1.0, 10, "2026-09-01", temp_type="int32")
write_profile("other/with_doxy.nc", "E1", "cruise-5", 51.0, 2.0, 20, "2026-10-01", extra=True)
write_profile("other/float32_temp.nc", "F1", "cruise-6", 52.0, 2.0, 10, "2026-11-01", temp_type="float32")

One of the files:

import xarray as xr

with xr.open_dataset("profiles/A1.nc") as ds:
    print(ds)

<xarray.Dataset> Size: 1kB
Dimensions:    (depth: 50)
Coordinates:
  * depth      (depth) float64 400B 0.0 10.0 20.0 30.0 ... 470.0 480.0 490.0
Data variables:
    TEMP       (depth) float64 400B ...
    PSAL       (depth) float64 400B ...
    LATITUDE   float64 8B ...
    LONGITUDE  float64 8B ...
    TIME       datetime64[ns] 8B ...
Attributes:
    station:  A1
    cruise:   cruise-1

Make a table

lance_nd.create() gives a writer. Open a file and write its dataset, then open the next file. The writer commits all datasets at the end of the with block.

import lance_nd

with lance_nd.create("profiles.lance", stats=True) as writer:
    with xr.open_dataset("profiles/A1.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/A2.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/A3.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/B1.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/B2.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/B3.nc") as ds:
        writer.write(ds)

table = writer.table
print(table.count_rows(), table.version)

6 1

• Each file is one row, so the table has 6 rows.
• All six files are one commit, so the table starts at version 1.
• Write the dataset inside the with xr.open_dataset(...) block. The writer reads the values from the file.

Add a dataset

lance_nd.append() gives a writer for a table that exists.

with lance_nd.append("profiles.lance") as writer:
    with xr.open_dataset("new/C1.nc") as ds:
        writer.write(ds)

table = lance_nd.open("profiles.lance")
print(table.count_rows(), table.version)

7 2

Read the datasets

Table.scan() reads the datasets back. to_xarray() gives each dataset as an xarray.Dataset.

for batch in table.scan():
    ds = batch.to_xarray()
    print(ds.attrs["station"], ds.sizes["depth"], float(ds.LATITUDE))

A1 50 52.5
A2 80 53.0
A3 120 60.0
B1 60 54.0
B2 70 55.5
B3 90 57.0
C1 40 58.0

print(next(iter(table.scan(limit=1))).to_xarray())

<xarray.Dataset> Size: 1kB
Dimensions:    (depth: 50)
Coordinates:
  * depth      (depth) float64 400B 0.0 10.0 20.0 30.0 ... 470.0 480.0 490.0
Data variables:
    TEMP       (depth) float64 400B 15.0 14.9 14.8 14.7 ... 10.4 10.3 10.2 10.1
    PSAL       (depth) float64 400B 35.0 35.01 35.02 35.03 ... 35.47 35.48 35.49
    LATITUDE   float64 8B 52.5
    LONGITUDE  float64 8B 4.0
    TIME       datetime64[ns] 8B 2026-06-01
Attributes:
    station:  A1
    cruise:   cruise-1

The dataset from the table is the same as the dataset from the file:

first = next(iter(table.scan(limit=1))).to_xarray()
with xr.open_dataset("profiles/A1.nc") as original:
    xr.testing.assert_identical(first, original.load())
print("identical")

identical

Next steps

The user guide explains each task in its own chapter: create, add, read, delete, replace, compact and versions.