Expiring pins for ongoing work

[brendanclement]

Compaction should be able to run while long-running jobs are in progress. The Fragment Reuse Index (FRI) records row movements from compaction so those jobs can account for changes to the table’s layout. Cleanup also needs to know which source snapshots, committed changes, and FRI history those jobs still depend on.

This proposal introduces expiring pins to record those retention requirements. Async index builds are the first consumer, but the mechanism is independent of the job type. Other long-running work, such as refresh-column jobs, can use the same pins to retain the resources it still needs.

Purpose

Suppose a job starts reading files A and B, then compaction combines them into C. The job may still need both the original files and the FRI record showing where their rows moved. Keeping the files alone does not preserve that movement history.

Before publishing its result, a job may also need to check the changes committed since it started. Keeping its source files and movement mappings alone does not preserve those transaction records. Cleanup should retain the evidence the supported publication checks need.

I propose one pin that file cleanup and FRI cleanup both understand. The first implementation would reuse the snapshot-retention logic already used for tags, retaining every snapshot from the oldest active pin’s source through the current head. This preserves the intervening manifests and transactions as well as their referenced files.

A pin tells Lance what to retain. It does not establish whether a job’s result is still valid for the current table. That remains the job’s responsibility, including checks for saved index output such as those discussed in #9947.

Pin Records

Pins belong to the branch where they were registered. The latest manifest, which describes the current table version, holds the active-pin list. Copies in older manifests do not keep a released pin alive.

Each pin uses a dedicated manifest record:

Field	Purpose
id	The registration transaction’s UUID. Identifies the pin and stays the same on retries.
source_version	The snapshot the job reads. It sets the start of its snapshot-retention window and the source for FRI history retention.
source_transaction_uuid	The UUID of the transaction that created the source snapshot. Together with source_version, identifies the exact snapshot.
created_at	UTC timestamp when registration was requested.
expires_at	UTC timestamp after which cleanup may expire the pin. Passing this timestamp does not automatically remove it.
job_label	Optional description of the job, to help people inspect active pins.

Renewal changes only the expiry time. The source, creation time, and label stay fixed.

Registration and lifecycle

The format defines registration, renewal, release, and expiration transactions.

Registration

A new pin targets the branch’s current snapshot, identified by its version and source transaction UUID. Registration creates the next table version containing the pin. Registering historical versions is outside this proposal.

A request always targets the same snapshot, including on retries:

• If the same pin ID and source are already active, recover that pin without another commit. Preserve its current expires_at and label.
• If the pin has not already been registered, its requested snapshot must still be the branch’s current snapshot. If the branch has moved on or the snapshot has been replaced, registration fails.
• If the source has changed, fail. A new request for a newer source needs a new pin ID and is allowed only before work starts, when the job does not require the earlier source.

A job must wait for confirmation before relying on the pin. Confirmation must read the actual current state, verify the active pin and source identity, and check the branch ID for a named branch. A cached Dataset or a registration found in an older manifest is insufficient. If the state cannot be established, report an unknown outcome rather than success.

Each job has its own pin ID. Two pins can retain the same files; releasing one leaves the other active. Registration itself creates a version, so consecutive registrations can have different source version numbers even when the data is unchanged.

Renewal

A renewal targets an existing pin and requests a fixed absolute UTC expires_at. Retries keep that expires_at, they must not repeatedly add time. If the existing expires_at already meets or exceeds the request, succeed without a commit. Otherwise, extend it. Recheck current state after a commit conflict, and fail if the pin has been removed.

No separate renewal counter is needed because every extension moves the expires_at forward. There is no absolute lifetime cap. A stuck job that keeps renewing can retain resources indefinitely but an operator can inspect and release its pin as an escape hatch.

Release and expiration

Release removes one pin by ID. Releasing an absent pin succeeds without a commit. Data commits do not automatically release pins.

Passing the expires_at time only makes a pin eligible for expiration. Ordinary cleanup must commit its removal before treating resources as no longer retained by that pin. Releasing or expiring the oldest pin can advance the snapshot-retention boundary, unless another active pin still needs that source. Removing the last pin ends this window requirement; tags and other retention rules still apply.

Before committing a pin’s expiration, cleanup checks that expires_at has not changed since inspection. If the pin was renewed, cleanup skips it for that expiration attempt:

• If renewal commits first, cleanup must respect the updated expires_at.
• If expiration commits first, renewal fails because the pin is gone.

Without cleanup, an expires_at does not guarantee when storage is reclaimed. Jobs that may restart must save their pin ID. If registration has not yet been confirmed, they must also save the original request so they can retry it safely.

Compaction and cleanup

File cleanup resolves current pin membership and the authoritative branch head. If the oldest active source is v100 and the head is v110, retain every attached snapshot from v100 through v110, inclusive. Detached snapshots are outside this window. Preserve the manifests and any commit-store records needed to discover and read them.

Retain each snapshot’s normal referenced resources, including base data, deletion files, overlays, indices, transaction sidecars, FRI metadata, and external mapping files. Shared physical files count once. These remain readable snapshots; keeping their manifests while deleting their data is not sufficient.

Every attached commit whose result contains active pins must persist its own transaction before publication, either inline in its manifest or in the referenced transaction file. This includes registration, restore, overwrite, and ordinary writes that carry pins forward. Retain the transactions for every version after the oldest pinned source through the head so publication can inspect the committed changes in order.

This deliberately keeps intermediate data and index generations that publication may not read. We accept that storage cost for the first implementation to use existing retention machinery. A future metadata-only representation would need a separately specified compatible discovery and retention protocol. The pin record needs no additional field for this approach.

Cleanup can still delete files outside these requirements and other retention rules. It should report retention caused by pins. If a pinned source has the wrong identity, or a required manifest or transaction is missing or unreadable, stop the affected destructive cleanup and report the failure. Also reject trimming or rewriting that cannot establish the pin’s required fragment set. This does not release the pin; explicit release by its ID remains available.

FRI cleanup retains the combined requirements of active pins and existing consumers. An index catching up does not end a pin’s requirement. A consumer-check error must not cause pin-required history to be dropped; if the pin’s requirements cannot be established, refuse the affected trim.

For FRI (index version 0), a source at version V retains applicable current-history records stamped at V or later, including V. A pin-aware writer stamps each new record with the version immediately before the manifest that publishes it, and updates that stamp on retry. Existing records keep their stamps.

Version-1 history is called tagged; if it also contains legacy records, it is mixed. Tagged transitions have no dataset-version stamp, so pins use fragment references instead. Start with the fragments in the pinned source snapshots. Retain records that move rows out of those fragments, then follow the destination fragments through later records until no more records are needed. Retain whole records. If any group in a legacy record is needed, retain every group in that record and follow all their destinations.

Required history also retains its external mapping files, including mappings created after registration. Replacing FRI metadata does not make a mapping file removable while a retained snapshot or required history still references it.

Pins preserve the existing overlay restrictions. A tagged FRI entry blocks new overlay writes across the table, and a rewrite recording tagged transitions rejects overlaid source fragments. Retaining tagged history can keep those restrictions in effect longer. Supporting those combinations is separate work outside the scope of this proposal.

Compaction must record the movements active pins need. Before committing its result, compaction must recheck the active pins, including pins registered after planning. This also applies when no FRI entry existed during planning. If the result lacks movement records those pins require, it must not commit; the caller can replan. Operations that remove FRI history must also recheck retention requirements at commit. Restore and overwrite apply these rules to the resulting current history, as described below.

Before releasing a pin, a job must establish protection for its published result. An index build attaches its index, confirms that the index retains the history it needs, then releases the pin. These are separate transactions, with no gap in retention.

Pins do not protect newly written, uncommitted output. Jobs need a separate staging and publication strategy for those files.

Restore and branches

Pins do not block restore to a valid retained version. If a job pins v100 and the table is restored to v80, restore creates a new latest version containing v80’s data. The pin still retains every snapshot from v100 through the new head. Restore does not move its retention boundary forward.

Restore carries forward the current pins and expires_at times, rather than restoring the pin list saved in v80. Otherwise it could drop a running job’s pin or revive one already released. Overwrite and other data commits must also preserve current pins unless they explicitly change them.

Restore and overwrite apply FRI retention to the resulting current history. They may discard translation paths and do not have to reconstruct them. Within the resulting history, retain movement records needed to connect a pinned source to the current layout. Records needed only for a discarded intermediate layout may be removed if no other consumer requires them. Pins still retain the complete snapshot window, but cannot guarantee that a job can translate or publish its results when a mapping no longer exists or a data change invalidates them. Removing a record from the current FRI does not reclaim files still referenced by a retained snapshot.

New branches and clones start with no active pins and keep their normal snapshot protections. Jobs on them register their own pins.

Branch deletion must inspect current pins and reject deletion if any remain, including on force deletion. It must identify the blocking pins. Pins must first be released or removed by ordinary expiration cleanup, branch deletion does not expire them itself. Existing tag and child-branch protections also apply here.

Existing concurrency limits

Registering a pin does not cancel deletion that maintenance has already planned. For example:

• Cleanup can plan to delete old files, then finish after a restore makes those files current again.
• A branch deletion can check for pins before a new registration completes. That check does not lock out registration while files are deleted.
• A delayed branch deletion can resume after recovery and name reuse, damaging the replacement branch.

These are existing cleanup and branch-coordination gaps. Pins alone do not make those overlaps safe, and fixing them is outside this proposal. Maintenance must still honor pins it observes, including the commit-time FRI checks above.

Current-version registration has narrower cleanup exposure than tagging arbitrary old versions. Cleanup retains the current version it inspected, and recent files normally have a grace period. This is not a general guarantee: the grace period can elapse or be bypassed by delete_unverified, and restore can reintroduce older files.

Compatibility

The format reserves writer capability bit 19, FLAG_ONGOING_WORK_PINS. Once set, it remains set after restore or release of the last pin. New branches and clones preserve it when inherited, even though their pin lists start empty. Pins add no reader capability requirement by themselves.

Before enabling pins, writers, compaction workers, cleaners, and branch-deletion clients must understand the feature. Unsupported writers must reject flagged manifests. Clients that never check the flag must be upgraded or excluded, and incompatible operations already running must finish or be cancelled.

[wjones127]

Release removes one pin by ID. Releasing an absent pin succeeds without a commit. Data commits do not automatically release pins.

Why do data commits not automatically release pins? I was initially thinking they could, given you have the UUID of the transaction that is open. Then releasing is otherwise used for cancellation.

The nice thing about letting those data commits release pins is that they don't need an additional transaction to go through. Thus they don't push additional concurrency pressure on the commit path. The contract becomes "I need this lease until expires_at or I commit a transaction with this UUID."

But I can also understand if this can lead to unnecessary complexity.

[brendanclement]

i think we can change the contract to let the job publish its results and release the in the same transaction. I think with Transaction V2 this becomes simpler but I can work this into the existing contract.

[wjones127]

If we want to just say we’ll address that in txn v2—I’m not opposed to that. Just something I think we should weigh.

[wjones127]

Suggestion: make the source the registration version

Instead of the request naming source_version / source_transaction_uuid, define the source as the version the registration commit itself creates. A registration changes no data, so it can always rebase over concurrent commits. The pin then lands on whatever the head is when it commits. The job waits for confirmation, as it already must, and then reads the pinned version.

This removes source_transaction_uuid and the "branch has moved on" failure. A retry is just "pin ID already present → return its source." It also stops registration from failing over and over on tables with steady writes. The one requirement is that jobs pin before they plan, which the retry rules already imply.

Example:

1. A job reads head v100 and tries to commit "register pin P" as v101.
2. A concurrent append lands v101 first.
3. Today, registration fails and the job retries with a new pin ID, which can fail again the same way. With this change, registration rebases and lands as v102.
4. Pin P's source is v102, the manifest that contains P. Its data is identical to v101.
5. The job confirms P, reads v102 and builds against it. Cleanup keeps v102 through the head.
6. If the job crashes and retries registration with P, it finds P already active and gets source v102 back without another commit.

[brendanclement]

i really like this suggestion. i'll rework the design to put this in