Maintenance

Delete rows with an exact predicate on the table columns, then compact the table and remove the files of old versions:

use std::time::Duration;
use lance_nd::{CompactOptions, cleanup_old_versions, compact, delete};

let deleted = delete(&mut table, "__global_attrs__.id = 'a'").await?;
compact(&mut table, &CompactOptions::default()).await?;
cleanup_old_versions(&table, Duration::from_secs(7 * 24 * 60 * 60)).await?;

• delete takes Lance SQL, and delete_expr takes a DataFusion Expr. Lance gets the predicate with no change. A delete does not use the stats, because a stats filter keeps rows that do not match. A filter on values must be exact, for example array_has(sst.values, 5.0).
• A delete marks rows in deletion files. It does not rewrite data files.
• compact aims for fragments of 128 MiB on disk. It merges neighboring fragments below half of the target, splits fragments above twice the target, and writes again each fragment with at least 10% deleted rows. Each run of these fragments becomes fragments of about the same size. CompactOptions changes these limits.
• The plan estimates the size of a row from the size of its fragment on disk, so the new sizes are not exact. A second compact evens them out. A row is never split. lance-nd holds about one new fragment in memory at a time. The row order stays.
• Each delete and compaction makes a new table version. The files of old versions stay on disk until cleanup_old_versions removes them. A reader that still uses a removed version fails, so keep a margin.

lance-nd writes the compacted fragments itself. Lance reads the arrays of an absent variable as null, but it rejects these nulls in a write, so its own compaction fails on a lance-nd table.

Versions

Each commit makes a new table version. versions(&table) lists the versions that still have their files. checkout(&table, version) gives an earlier version to read. restore(&mut table, version) makes the content of an earlier version the latest version, in a new commit.