Python Add Datasets

lance_nd.append() gives a writer that adds datasets to a table that exists. All datasets of one writer are one commit.

Example files and table for this page

The examples use small NetCDF files of CTD profiles, see Getting started. Run this block first. It writes the files and makes the table profiles.lance from the six files in profiles/.

from pathlib import Path

import numpy as np
import pandas as pd
import xarray as xr


def write_profile(path, station, cruise, lat, lon, levels, time, temp_type="float64", extra=None):
    """Write one CTD profile as a NetCDF file."""
    depth = np.arange(levels, dtype="float64") * 10.0
    ds = xr.Dataset(
        {
            "TEMP": ("depth", (15.0 - depth / 100.0).astype(temp_type), {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )
    if extra:
        ds["DOXY"] = ("depth", np.full(levels, 250.0), {"units": "umol/kg"})
    Path(path).parent.mkdir(exist_ok=True)
    ds.to_netcdf(path)


# Six profiles of two cruises.
write_profile("profiles/A1.nc", "A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01")
write_profile("profiles/A2.nc", "A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02")
write_profile("profiles/A3.nc", "A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03")
write_profile("profiles/B1.nc", "B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01")
write_profile("profiles/B2.nc", "B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02")
write_profile("profiles/B3.nc", "B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03")
# Three new profiles of a third cruise.
write_profile("new/C1.nc", "C1", "cruise-3", 58.0, 6.0, 40, "2026-08-01")
write_profile("new/C2.nc", "C2", "cruise-3", 58.5, 6.5, 45, "2026-08-02")
write_profile("new/C3.nc", "C3", "cruise-3", 59.0, 7.0, 50, "2026-08-03")
# Corrected profiles, with other depth levels.
write_profile("corrected/A1.nc", "A1", "cruise-1", 52.5, 4.0, 55, "2026-06-01")
write_profile("corrected/A2.nc", "A2", "cruise-1", 53.0, 4.5, 90, "2026-06-02")
write_profile("corrected/B1.nc", "B1", "cruise-2", 54.0, 3.0, 65, "2026-07-01")
write_profile("corrected/B2.nc", "B2", "cruise-2", 55.5, 3.5, 75, "2026-07-02")
# Files that differ from the others.
write_profile("other/int_temp.nc", "D1", "cruise-4", 50.0, 1.0, 10, "2026-09-01", temp_type="int32")
write_profile("other/with_doxy.nc", "E1", "cruise-5", 51.0, 2.0, 20, "2026-10-01", extra=True)
write_profile("other/float32_temp.nc", "F1", "cruise-6", 52.0, 2.0, 10, "2026-11-01", temp_type="float32")

import lance_nd


def stations(table):
    """The station of each dataset in the table, in table order."""
    return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]


with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
    with xr.open_dataset("profiles/A1.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/A2.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/A3.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/B1.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/B2.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("profiles/B3.nc") as ds:
        writer.write(ds)
table = lance_nd.open("profiles.lance")

Add files

Open a new file and write its dataset, then open the next file. Here the three files in new/ are added to the table.

with lance_nd.append("profiles.lance") as writer:
    with xr.open_dataset("new/C1.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("new/C2.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("new/C3.nc") as ds:
        writer.write(ds)

table = lance_nd.open("profiles.lance")
print(table.version, table.count_rows())
print(stations(table))

2 9
['A1', 'A2', 'A3', 'B1', 'B2', 'B3', 'C1', 'C2', 'C3']

• The new datasets come after the old datasets, in the order of the write() calls.
• write() also takes a list of datasets, or a dataset that you make in memory with xr.Dataset(...).
• lance_nd.open() gives the table at its latest version. A table object that you opened before stays at its version.
• append() raises TableNotFoundError when the table does not exist.

Add many files

One writer can take thousands of datasets. It writes full fragments to disk while you add datasets, so the memory use stays small.

from pathlib import Path

with lance_nd.append("profiles.lance") as writer:
    for path in sorted(Path("incoming").glob("*.nc")):
        with xr.open_dataset(path) as ds:
            ds.attrs["source_file"] = path.name
            writer.write(ds)

• Commit once for a few thousand files, not once for each file. Each commit closes a fragment, so many small commits give many small fragments.
• A source_file attribute lets you find, delete and replace the dataset of one file later, see Delete datasets.

Errors in one dataset

A dataset that does not fit the table raises LanceNdError. The writer skips all datasets of that write() call and stays usable.

The file other/int_temp.nc stores TEMP as int32, but the table stores TEMP as float64:

with lance_nd.append("profiles.lance") as writer:
    with xr.open_dataset("profiles/A1.nc") as ds:
        writer.write(ds)
    with xr.open_dataset("other/int_temp.nc") as ds:
        try:
            writer.write(ds)
        except lance_nd.LanceNdError as error:
            print("int_temp.nc skipped:", type(error).__name__)

print(lance_nd.open("profiles.lance").count_rows())

int_temp.nc skipped: SchemaMismatchError
10

• finish() commits the datasets that the writer took. Here the dataset of A1.nc is added a second time, and the dataset of int_temp.nc is not added.
• Catch the error per file, log the file and go on.
• A failed write of a fragment to disk makes the writer fail. Then finish() commits nothing.

New variables

A dataset can have variables that the table does not have. The table gets a new column, and older datasets do not have the variable. The file other/with_doxy.nc has an oxygen variable DOXY:

with lance_nd.append("profiles.lance") as writer:
    with xr.open_dataset("other/with_doxy.nc") as ds:
        writer.write(ds)

table = lance_nd.open("profiles.lance")
print(table.variables())
last = list(table.scan())[-1].to_xarray()
first = next(iter(table.scan(limit=1))).to_xarray()
print("DOXY" in last, "DOXY" in first)

['TEMP', 'PSAL', 'LATITUDE', 'LONGITUDE', 'TIME', 'depth', 'DOXY']
True False

Types that do not match

A variable has one type in the table. By default, a dataset with another type raises LanceNdError, as in Errors in one dataset. casting changes this:

casting	A float32 TEMP in a float64 table	A float64 TEMP in a float32 table
"no" (default)	error	error
"safe"	cast to float64	error
"same_kind"	cast to float64	cast to float32
"promote"	cast to float64	the table becomes float64

The file other/float32_temp.nc stores TEMP as float32:

with lance_nd.append("profiles.lance", casting="safe") as writer:
    with xr.open_dataset("other/float32_temp.nc") as ds:
        writer.write(ds)

last = list(lance_nd.open("profiles.lance").scan(project=["TEMP"]))[-1]
print(last.to_xarray().TEMP.dtype)

float64

• "safe" casts only when no value can change.
• "promote" widens the variable in the table, as numpy type promotion. The writer first commits the datasets that it wrote before.
• Table.widen("TEMP", pa.float64()) widens a variable of a table without a write.

Large datasets in chunks

A large dataset that you open with dask chunks can become one row per chunk. A reader can then load one chunk at a time.

ds = xr.open_dataset("ocean.nc", chunks={"time": 24, "depth": 100})

with lance_nd.append("ocean.lance") as writer:
    writer.write(ds, chunks=["time", "depth"], on_drop="warn")

• chunks names the chunk dimensions. Each row is one chunk.
• A variable is kept when all its dimensions are chunk dimensions, or when it has all chunk dimensions. on_drop says what happens to the other variables: "warn", "raise" or "ignore".
• The API reference gives all rules.

Other input

write() also takes an nd_arrow_array.NdRecordBatch, a list of these, or Arrow data in the nd-arrow format, such as a pyarrow.RecordBatch with one dataset per row.