Python Replace Datasets

A table has no update in place. To change datasets, delete the old datasets and add the new datasets.

Example data for this page

The examples use six CTD profiles from two cruises, see Getting started. Run this block first to make the table profiles.lance.

import numpy as np
import pandas as pd
import xarray as xr

import lance_nd


def profile(station, cruise, lat, lon, levels, time):
    depth = np.arange(levels, dtype="float64") * 10.0
    return xr.Dataset(
        {
            "TEMP": ("depth", 15.0 - depth / 100.0, {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )


def profiles():
    return [
        profile("A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01"),
        profile("A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02"),
        profile("A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03"),
        profile("B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01"),
        profile("B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02"),
        profile("B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03"),
    ]


def stations(table):
    """The station of each dataset in the table, in table order."""
    return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]


with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
    writer.write(profiles())
table = lance_nd.open("profiles.lance")

Replace one dataset

Here station A2 gets a corrected profile with 90 depth levels.

table.delete("__global_attrs__.station = 'A2'")
with lance_nd.append("profiles.lance") as writer:
    writer.write(profile("A2", "cruise-1", 53.0, 4.5, 90, "2026-06-02"))

table = lance_nd.open("profiles.lance")
print(table.version, table.count_rows())
print(stations(table))

3 6
['A1', 'A3', 'B1', 'B2', 'B3', 'A2']

• The delete and the append are two commits, so the table gets two versions.
• The new A2 comes at the end of the table. Datasets have no fixed position. Use an attribute, such as station, to find a dataset.

Replace many datasets

Delete all old datasets with one filter, then add all new datasets with one writer. This is two commits for any number of datasets.

corrected = [
    profile("B1", "cruise-2", 54.0, 3.0, 65, "2026-07-01"),
    profile("B2", "cruise-2", 55.5, 3.5, 75, "2026-07-02"),
]
names = ", ".join(f"'{ds.attrs['station']}'" for ds in corrected)

table.delete(f"__global_attrs__.station IN ({names})")
with lance_nd.append("profiles.lance") as writer:
    writer.write(corrected)

table = lance_nd.open("profiles.lance")
print(table.version, table.count_rows())
print([(b.to_xarray().attrs["station"], b.to_xarray().sizes["depth"]) for b in table.scan(project=["depth"])])

5 6
[('A1', 50), ('A3', 120), ('B3', 90), ('A2', 90), ('B1', 65), ('B2', 75)]

Between the two commits

• A reader that opens the table after the delete and before the append sees the table without the old datasets and without the new datasets.
• When the append fails, the old datasets stay deleted. Write the new datasets again, or undo the delete with restore().
• Do the delete first. When you append first, the filter of the delete also matches the new datasets.

A safe pattern keeps the version before the delete:

before = table.version
table.delete("__global_attrs__.station = 'A1'")
try:
    with lance_nd.append("profiles.lance") as writer:
        writer.write(profile("A1", "cruise-1", 52.5, 4.0, 55, "2026-06-01"))
except lance_nd.LanceNdError:
    table.restore(before)
    raise

print(lance_nd.open("profiles.lance").count_rows())

6

Many small replacements

Each replacement makes two versions and a small fragment. After many replacements, compact the table to merge the small fragments and to remove the deleted rows.