Python Compact

Table.compact() rewrites the fragments of a table. It merges small fragments, splits large fragments and removes deleted rows.

Example data for this page

The examples use six CTD profiles from two cruises, see Getting started. Run this block first to make the table profiles.lance.

import numpy as np
import pandas as pd
import xarray as xr

import lance_nd


def profile(station, cruise, lat, lon, levels, time):
    depth = np.arange(levels, dtype="float64") * 10.0
    return xr.Dataset(
        {
            "TEMP": ("depth", 15.0 - depth / 100.0, {"units": "degree_Celsius"}),
            "PSAL": ("depth", 35.0 + depth / 1000.0, {"units": "psu"}),
            "LATITUDE": ((), lat),
            "LONGITUDE": ((), lon),
            "TIME": ((), pd.Timestamp(time)),
        },
        coords={"depth": ("depth", depth, {"units": "m"})},
        attrs={"station": station, "cruise": cruise},
    )


def profiles():
    return [
        profile("A1", "cruise-1", 52.5, 4.0, 50, "2026-06-01"),
        profile("A2", "cruise-1", 53.0, 4.5, 80, "2026-06-02"),
        profile("A3", "cruise-1", 60.0, 2.0, 120, "2026-06-03"),
        profile("B1", "cruise-2", 54.0, 3.0, 60, "2026-07-01"),
        profile("B2", "cruise-2", 55.5, 3.5, 70, "2026-07-02"),
        profile("B3", "cruise-2", 57.0, 5.0, 90, "2026-07-03"),
    ]


def stations(table):
    """The station of each dataset in the table, in table order."""
    return [batch.to_xarray().attrs["station"] for batch in table.scan(project=[])]


with lance_nd.create("profiles.lance", mode="overwrite", stats=True) as writer:
    writer.write(profiles())
table = lance_nd.open("profiles.lance")

Why compact

• Each commit writes at least one fragment. Many small commits give many small fragments, and a scan must open each fragment.
• A delete only marks rows as deleted. The rows stay in the files.

Here three small appends and a delete leave four fragments:

for station in ["C1", "C2", "C3"]:
    with lance_nd.append("profiles.lance") as writer:
        writer.write(profile(station, "cruise-3", 58.0, 6.0, 40, "2026-08-01"))
table = lance_nd.open("profiles.lance")
table.delete("__global_attrs__.cruise = 'cruise-1'")
print(table.num_fragments, table.count_rows(), table.version)

4 6 5

Compact the table

stats = table.compact()
print(stats["fragments_removed"], stats["fragments_added"])
print(table.num_fragments, table.count_rows(), table.version)
print(stations(table))

4 1
1 6 7
['B1', 'B2', 'B3', 'C1', 'C2', 'C3']

• compact() returns fragments_removed, fragments_added, files_removed and files_added.
• The datasets and their order stay the same.
• A compaction makes two commits: the first gets the ids of the new fragments, the second replaces the fragments. Here the version goes from 5 to 7.
• When nothing needs work, compact() makes no commit.

What gets rewritten

The target size is 128 MiB on disk. compact() rewrites:

• neighbor fragments that are smaller than half of the target, merged into fragments of about the target size,
• fragments that are larger than twice the target, split into fragments of about the target size,
• fragments with 10% or more deleted rows, without the deleted rows.

A single small fragment with no neighbor to merge stays.

Options

stats = table.compact(target_fragment_bytes=512 << 20, deleted_fraction=0.05)
print(stats["fragments_removed"])

0

• target_fragment_bytes is the target size on disk. 512 MiB to 2 GiB suits a large table: a scan then opens fewer files.
• deleted_fraction is the fraction of deleted rows that makes a fragment get a rewrite. 1.0 rewrites only fragments where all rows are deleted.
• The sizes are estimates, so the new fragments are near the target, not exact. A second compact() evens them out.

Free the disk space

After a compaction, the old fragments still belong to the old versions. Remove them with cleanup_old_versions(), see Versions.

from datetime import timedelta

removed = table.cleanup_old_versions(older_than=timedelta(0))
print(removed["old_versions"])

6

When to compact

• After many small appends, deletes or replacements.
• When table.num_fragments is much larger than the table size divided by the target size.
• A compaction reads and writes the fragments that need work, so it takes time on a large table. Run it when few writers use the table, for example once a day.
• Readers can read the table during a compaction. They see the old fragments until the commit.