Python Add Datasets

lance_nd.append() gives a writer that adds datasets to a table that exists. All datasets of one writer are one commit.

Example data for this page

The examples use six CTD profiles from two cruises, see Getting started. Run this block first to make the table profiles.lance.

import numpy as np
import pandas as pd
import xarray as xr

import lance_nd


def profile(station, cruise, lat, lon, levels, time):
    depth = np.arange(levels, dtype="float64") * 10.0
    return xr.Dataset(
        {
            "TEMP": ("depth", 15.0 - depth / 100.0, {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )


def profiles():
    return [
        profile("A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01"),
        profile("A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02"),
        profile("A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03"),
        profile("B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01"),
        profile("B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02"),
        profile("B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03"),
    ]


def stations(table):
    """The station of each dataset in the table, in table order."""
    return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]


with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
    writer.write(profiles())
table = lance_nd.open("profiles.lance")

Add one or more datasets

write() takes one dataset or a list of datasets.

with lance_nd.append("profiles.lance") as writer:
    writer.write(profile("C1", "cruise-3", 58.0, 6.0, 40, "2026-08-01"))
    writer.write([
        profile("C2", "cruise-3", 58.5, 6.5, 45, "2026-08-02"),
        profile("C3", "cruise-3", 59.0, 7.0, 50, "2026-08-03"),
    ])

table = lance_nd.open("profiles.lance")
print(table.version, table.count_rows())
print(stations(table))

2 9
['A1', 'A2', 'A3', 'B1', 'B2', 'B3', 'C1', 'C2', 'C3']

• The new datasets come after the old datasets, in the order of the write() calls.
• lance_nd.open() gives the table at its latest version. A table object that you opened before stays at its version.
• append() raises TableNotFoundError when the table does not exist.

Add many files

One writer can take thousands of datasets. It writes full fragments to disk while you add datasets, so the memory use stays small.

from pathlib import Path

with lance_nd.append("profiles.lance") as writer:
    for path in sorted(Path("incoming").glob("*.nc")):
        with xr.open_dataset(path) as ds:
            ds = ds.load()
        ds.attrs["source_file"] = path.name
        writer.write(ds)

• Commit once for a few thousand files, not once for each file. Each commit closes a fragment, so many small commits give many small fragments.
• A source_file attribute lets you find, delete and replace the dataset of one file later, see Delete datasets.
• ds.load() reads the file before the file closes.

Errors in one dataset

A dataset that does not fit the table raises LanceNdError. The writer skips all datasets of that write() call and stays usable.

bad = profile("D1", "cruise-4", 50.0, 1.0, 10, "2026-09-01")
bad["TEMP"] = bad.TEMP.astype("int32")

with lance_nd.append("profiles.lance") as writer:
    for ds in [profile("D0", "cruise-4", 50.0, 1.0, 10, "2026-09-01"), bad]:
        try:
            writer.write(ds)
        except lance_nd.LanceNdError as error:
            print(ds.attrs["station"], "skipped:", type(error).__name__)

print(lance_nd.open("profiles.lance").count_rows())

D1 skipped: SchemaMismatchError
10

• finish() commits the datasets that the writer took. Here D0 is added and D1 is not.
• Catch the error per file, log the file and go on.
• A failed write of a fragment to disk makes the writer fail. Then finish() commits nothing.

New variables

A dataset can have variables that the table does not have. The table gets a new column, and older datasets do not have the variable.

extra = profile("E1", "cruise-5", 51.0, 2.0, 20, "2026-10-01")
extra["DOXY"] = ("depth", np.full(20, 250.0), {"units": "umol/kg"})

with lance_nd.append("profiles.lance") as writer:
    writer.write(extra)

table = lance_nd.open("profiles.lance")
print(table.variables())
last = list(table.scan())[-1].to_xarray()
first = next(iter(table.scan(limit=1))).to_xarray()
print("DOXY" in last, "DOXY" in first)

['TEMP', 'PSAL', 'LATITUDE', 'LONGITUDE', 'TIME', 'depth', 'DOXY']
True False

Types that do not match

A variable has one type in the table. By default, a dataset with another type raises LanceNdError, as in Errors in one dataset. casting changes this:

casting	A float32 TEMP in a float64 table	A float64 TEMP in a float32 table
"no" (default)	error	error
"safe"	cast to float64	error
"same_kind"	cast to float64	cast to float32
"promote"	cast to float64	the table becomes float64

f32 = profile("F1", "cruise-6", 52.0, 2.0, 10, "2026-11-01")
f32["TEMP"] = f32.TEMP.astype("float32")

with lance_nd.append("profiles.lance", casting="safe") as writer:
    writer.write(f32)

last = list(lance_nd.open("profiles.lance").scan(project=["TEMP"]))[-1]
print(last.to_xarray().TEMP.dtype)

float64

• "safe" casts only when no value can change.
• "promote" widens the variable in the table, as numpy type promotion. The writer first commits the datasets that it wrote before.
• Table.widen("TEMP", pa.float64()) widens a variable of a table without a write.

Large datasets in chunks

A large dataset that you open with dask chunks can become one row per chunk. A reader can then load one chunk at a time.

ds = xr.open_dataset("ocean.nc", chunks={"time": 24, "depth": 100})

with lance_nd.append("ocean.lance") as writer:
    writer.write(ds, chunks=["time", "depth"], on_drop="warn")

• chunks names the chunk dimensions. Each row is one chunk.
• A variable is kept when all its dimensions are chunk dimensions, or when it has all chunk dimensions. on_drop says what happens to the other variables: "warn", "raise" or "ignore".
• The API reference gives all rules.

Other input

write() also takes an nd_arrow_array.NdRecordBatch, a list of these, or Arrow data in the nd-arrow format, such as a pyarrow.RecordBatch with one dataset per row.