bug: a short cleanup_older_than strands index files behind the 7-day unverified threshold

[gloryfromca]

Summary

A short cleanup_older_than window makes index files unreclaimable for a full 7 days, because the manifest that would have identified them as dead is deleted by an earlier cleanup pass. With a long-lived dataset and a short retention window this is a large, persistent overhead: 260MB of index files retained on a 19k-row table, none of it reclaimable by cleanup_old_versions until the files age past UNVERIFIED_THRESHOLD_DAYS.

The same workload with cleanup_older_than=0 reclaims everything, so the behaviour depends entirely on the retention window rather than on the data.

Mechanism

From rust/lance/src/dataset/cleanup.rs, an index file is removed when its UUID satisfies one of three conditions:

if inspection.referenced_files.index_uuids.contains(uuid.as_ref()) {
    return Ok(None);                                                       // 1. still referenced -> keep
} else if !maybe_in_progress {
    return Ok(cleanup_file(path, CleanupFileKind::Index, true, size_bytes)); // 2. >= 7 days old -> delete
} else if inspection.verified_files.index_uuids.contains(uuid.as_ref()) {
    return Ok(cleanup_file(path, CleanupFileKind::Index, false, size_bytes)); // 3. named by a manifest -> delete
}
// otherwise: keep

Branch 3 is the one that normally applies to a freshly superseded index, and it depends on some manifest — including one being deleted in this same pass — still naming the UUID. A short retention window breaks that:

1. optimize() builds index UUID N, referenced by manifest version V.
2. A later optimize() supersedes it. N is no longer in the current manifest but is still named by older retained manifests, so branch 3 applies.
3. A cleanup pass with a short window deletes those older manifests, while N's files are still minutes old.
4. From then on no manifest names N. Branch 1 does not apply (unreferenced), branch 2 does not apply (too young), branch 3 cannot apply (no manifest left to verify against). The files are kept.
5. They become deletable only once they cross the 7-day threshold.

With cleanup_older_than=0 steps 3 and 4 collapse into a single pass, so branch 3 still fires and the files go.

Reproduction

lancedb 0.34.0 (lance 8.0.0). Identical workload, only the retention window differs:

import asyncio, datetime as dt, shutil
from pathlib import Path
import lancedb
from lancedb.pydantic import LanceModel, Vector

class Doc(LanceModel):
    id: str
    text: str
    vector: Vector(8)

ROOT = "/tmp/retention_repro"

def idx_mb(name):
    idx = Path(f"{ROOT}/{name}.lance/_indices")
    return sum(f.stat().st_size for f in idx.rglob("*") if f.is_file()) // (1024 * 1024)

async def arm(db, name, retention_s):
    rows = [Doc(id=f"{name}_s{i}", text=f"alpha beta i{i}", vector=[0.1] * 8) for i in range(500)]
    t = await db.create_table(name, data=[r.model_dump() for r in rows])
    await t.create_index("text", config=lancedb.index.FTS())
    for b in range(24):
        await t.add([Doc(id=f"{name}_{b}_{i}", text=f"alpha gamma b{b} i{i}", vector=[0.2] * 8).model_dump()
                     for i in range(600)])
        await t.optimize()
        if (b + 1) % 4 == 0:
            await t.optimize(cleanup_older_than=dt.timedelta(seconds=retention_s), delete_unverified=False)
    await t.optimize(cleanup_older_than=dt.timedelta(seconds=retention_s), delete_unverified=False)
    print(f"retention={retention_s}s -> _indices {idx_mb(name)}MB after cleanup(delete_unverified=False)")
    await t.optimize(cleanup_older_than=dt.timedelta(seconds=0), delete_unverified=True)
    print(f"retention={retention_s}s -> _indices {idx_mb(name)}MB after cleanup(delete_unverified=True)")

async def main():
    shutil.rmtree(ROOT, ignore_errors=True)
    db = await lancedb.connect_async(ROOT)
    await arm(db, "ret0", 0)
    await arm(db, "ret60", 60)

asyncio.run(main())

Output:

retention=0s  -> _indices 0MB after cleanup(delete_unverified=False)
retention=0s  -> _indices 0MB after cleanup(delete_unverified=True)
retention=60s -> _indices 7MB after cleanup(delete_unverified=False)
retention=60s -> _indices 0MB after cleanup(delete_unverified=True)

Confirmation on a real dataset

A two-hour ingest soak (19k rows/table, cleanup_older_than=60s every 300s):

table	_indices	cleanup(False)	after backdating files past 7 days	cleanup(True)
episode	260MB	260MB (0 reclaimed)	—	1MB
atomic_fact	214MB	214MB (0 reclaimed)	—	1MB
atomic_fact (2nd dataset)	114MB	114MB (0 reclaimed)	0MB	—

The third row is the direct test of the age branch: the same cleanup_older_than=0, delete_unverified=False call reclaims nothing while the files are 24–41 hours old, and reclaims all 114MB once their mtimes are moved past the 7-day threshold. This is the documented behaviour working as designed; the issue is that a short retention window routes ordinary superseded indexes into that path.

Why this is awkward for callers

The retention window is chosen for a different concern — how long superseded data fragments are allowed to pile up before reclamation, where shorter is strictly better. On our side, moving it from 300s to 60s cut the transient data footprint about 5x. There is no hint that the same knob decides whether index files are reclaimed in minutes or in a week, and the two pull in opposite directions.

delete_unverified=True does reclaim them, but it is not usable on the hot path: it is exactly the flag that makes concurrent writers from another process unsafe.

Possible direction

When a cleanup pass removes a manifest, the index UUIDs that manifest exclusively referenced are known at that moment — treating them as verified-dead then (rather than leaving the decision to a later pass that no longer has the evidence) would make the outcome independent of the retention window. Sending a PR if you agree on the shape.

Environment

• lancedb 0.34.0, lance 8.0.0
• Local filesystem store, Linux (Ubuntu, ext4)
• Python 3.12