Python Compact

Table.compact() rewrites the fragments of a table. It merges small fragments, splits large fragments and removes deleted rows.

Example files and table for this page

The examples use small NetCDF files of CTD profiles, see Getting started. Run this block first. It writes the files and makes the table profiles.lance from the six files in profiles/.

from pathlib import Path

import numpy as np
import pandas as pd
import xarray as xr


def write_profile(path, station, cruise, lat, lon, levels, time, temp_type="float64", extra=None):
    """Write one CTD profile as a NetCDF file."""
    depth = np.arange(levels, dtype="float64") * 10.0
    ds = xr.Dataset(
        {
            "TEMP": ("depth", (15.0 - depth / 100.0).astype(temp_type), {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )
    if extra:
        ds["DOXY"] = ("depth", np.full(levels, 250.0), {"units": "umol/kg"})
    Path(path).parent.mkdir(exist_ok=True)
    ds.to_netcdf(path)


# Six profiles of two cruises.
write_profile("profiles/A1.nc", "A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01")
write_profile("profiles/A2.nc", "A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02")
write_profile("profiles/A3.nc", "A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03")
write_profile("profiles/B1.nc", "B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01")
write_profile("profiles/B2.nc", "B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02")
write_profile("profiles/B3.nc", "B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03")
# Three new profiles of a third cruise.
write_profile("new/C1.nc", "C1", "cruise-3", 58.0, 6.0, 40, "2026-08-01")
write_profile("new/C2.nc", "C2", "cruise-3", 58.5, 6.5, 45, "2026-08-02")
write_profile("new/C3.nc", "C3", "cruise-3", 59.0, 7.0, 50, "2026-08-03")
# Corrected profiles, with other depth levels.
write_profile("corrected/A1.nc", "A1", "cruise-1", 52.5, 4.0, 55, "2026-06-01")
write_profile("corrected/A2.nc", "A2", "cruise-1", 53.0, 4.5, 90, "2026-06-02")
write_profile("corrected/B1.nc", "B1", "cruise-2", 54.0, 3.0, 65, "2026-07-01")
write_profile("corrected/B2.nc", "B2", "cruise-2", 55.5, 3.5, 75, "2026-07-02")
# Files that differ from the others.
write_profile("other/int_temp.nc", "D1", "cruise-4", 50.0, 1.0, 10, "2026-09-01", temp_type="int32")
write_profile("other/with_doxy.nc", "E1", "cruise-5", 51.0, 2.0, 20, "2026-10-01", extra=True)
write_profile("other/float32_temp.nc", "F1", "cruise-6", 52.0, 2.0, 10, "2026-11-01", temp_type="float32")

import lance_nd


def stations(table):
    """The station of each dataset in the table, in table order."""
    return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]


with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
    for path in sorted(Path("profiles").glob("*.nc")):
        with xr.open_dataset(path) as ds:
            writer.write(ds)
table = lance_nd.open("profiles.lance")

Why compact

• Each commit writes at least one fragment. Many small commits give many small fragments, and a scan must open each fragment.
• A delete only marks rows as deleted. The rows stay in the files.

Here three appends of one file each and a delete leave four fragments:

for path in sorted(Path("new").glob("*.nc")):
    with lance_nd.append("profiles.lance") as writer:
        with xr.open_dataset(path) as ds:
            writer.write(ds)
table = lance_nd.open("profiles.lance")
table.delete("__global_attrs__.cruise = 'cruise-1'")
print(table.num_fragments, table.count_rows(), table.version)

4 6 5

Compact the table

stats = table.compact()
print(stats["fragments_removed"], stats["fragments_added"])
print(table.num_fragments, table.count_rows(), table.version)
print(stations(table))

4 1
1 6 7
['B1', 'B2', 'B3', 'C1', 'C2', 'C3']

• compact() returns fragments_removed, fragments_added, files_removed and files_added.
• The datasets and their order stay the same.
• A compaction makes two commits: the first gets the ids of the new fragments, the second replaces the fragments. Here the version goes from 5 to 7.
• When nothing needs work, compact() makes no commit.

What gets rewritten

The target size is 128 MiB on disk. compact() rewrites:

• neighbor fragments that are smaller than half of the target, merged into fragments of about the target size,
• fragments that are larger than twice the target, split into fragments of about the target size,
• fragments with 10% or more deleted rows, without the deleted rows.

A single small fragment with no neighbor to merge stays.

Options

stats = table.compact(target_fragment_bytes=512 << 20, deleted_fraction=0.05)
print(stats["fragments_removed"])

0

• target_fragment_bytes is the target size on disk. 512 MiB to 2 GiB suits a large table: a scan then opens fewer files.
• deleted_fraction is the fraction of deleted rows that makes a fragment get a rewrite. 1.0 rewrites only fragments where all rows are deleted.
• The sizes are estimates, so the new fragments are near the target, not exact. A second compact() evens them out.

Free the disk space

After a compaction, the old fragments still belong to the old versions. Remove them with cleanup_old_versions(), see Versions.

from datetime import timedelta

removed = table.cleanup_old_versions(older_than=timedelta(0))
print(removed["old_versions"])

6

When to compact

• After many small appends, deletes or replacements.
• When table.num_fragments is much larger than the table size divided by the target size.
• A compaction reads and writes the fragments that need work, so it takes time on a large table. Run it when few writers use the table, for example once a day.
• Readers can read the table during a compaction. They see the old fragments until the commit.