feat(master): paginate task view and add TTL-based history cleanup - #178
Merged
Merged
Conversation
The Task Queue view returned every task and rendered them all, and terminal task history was only bounded by a count cap (TASK_HISTORY_LIMIT), so old finished tasks under the cap lived forever. - TTL cleanup: new TASK_HISTORY_TTL_SECS (default 24h, 0 disables) prunes terminal Done/Failed tasks older than the TTL. It composes with the existing count cap — whichever removes a task first wins — and reuses the existing prune trigger points (open/finish/claim), so no new background loop. Queued/Running tasks and terminal tasks a live task still depends on are never pruned. Prune selection is extracted to a pure, unit-tested `prunable_terminal_ids`. - Server-side pagination: GET /api/v1/tasks now takes limit/offset (default 50, clamped 1..=200) and returns total/limit/offset alongside the page. The TaskQueue view gains Prev/Next paging (reusing the records pager) and shows the true total in the stat strip. Co-Authored-By: Claude <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Jul 25, 2026
…ounded (#194) ## Problem `_stats.rollout.lance` accumulates a new Lance version on every stats upsert and is never compacted or version-pruned. On a long-running master it reaches hundreds of thousands of versions (observed 172,789, climbing ~1/min), which makes cold start and `GET /api/v1/experiments` progressively slower. `StatsStore::upsert` does delete-then-append (`WriteMode::Append`), so each upsert creates ≥1 version. The periodic scan upserts one row per experiment every `STATS_SCAN_INTERVAL_SECS`, and `update_stats_after_compaction` upserts again after each compaction — hundreds of versions per hour, unbounded. This is the same class of problem #178 fixed for the task table, which didn't cover `_stats`. ## Fix Proposals 1 and 2 from the issue: - **`StatsStore::maintain(older_than)`** — `compact_files` with `materialize_deletions: true` (every upsert leaves a deletion) into one large fragment, then `cleanup_old_versions` with a grace window. The handle is reloaded around both so the cleanup sees the rewritten manifest and subsequent reads see the compacted version. - **Scanner integration** — runs under the *existing* `stats-writer` coordination lock, so only one replica ever rewrites the dataset and it can't race a concurrent scan. Fires on the first round (an existing deployment shouldn't wait N intervals to reclaim a 170k chain) and every Nth round after. - Bounded by a 5-minute timeout and logged-not-propagated, so a slow object store can neither wedge the scanner loop nor fail the scan round. Cleanup never touches versions newer than the grace window, so readers on another replica holding a recent version are unaffected. ## Config | Env | Default | Meaning | |---|---|---| | `STATS_MAINTENANCE_EVERY_N_SCANS` | `12` | Maintain every Nth scan round; `0` disables. At the default 300s scan interval that's hourly. | | `STATS_HISTORY_TTL_SECS` | `3600` | Grace window; versions newer than this are never removed. | ## Metrics `master_stats_maintenance_duration_seconds`, `master_stats_versions_removed_total`, `master_stats_version` (gauge — the thing that was silently climbing). ## Tests - `maintain_bounds_versions_and_preserves_rows` — 40 upserts, assert versions actually reclaimed and fragments compacted, rows intact and correct afterwards, store still writable. - `maintain_respects_grace_window` — a wide TTL removes nothing. `cargo test -p lance-context-master` and `cargo clippy --all-targets` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two issues in the Task Queue view as task volume grows: the endpoint returned every task (refetched every 1s) and rendered them all, and terminal task history was bounded only by a count cap (
TASK_HISTORY_LIMIT), so old finished tasks under the cap accumulated forever.TTL-based history cleanup
TASK_HISTORY_TTL_SECS(env/flag, default 24h,0disables) prunes terminalDone/Failedtasks older than the TTL.open/finish/claim_next) — no new background loop. (Trade-off: a fully idle master won't prune until the next task event; the count cap already behaved this way.)depends_on.prunable_terminal_idswith unit tests.Server-side pagination
GET /api/v1/tasksnow acceptslimit/offset(default 50, clamped1..=200) and returnstotal/limit/offsetwith the page.TaskQueueview gains Prev/Next paging (reuses the existing records pager) and shows the truetotalin the stat strip (ActiverelabeledActive (page)since it's per-page).Test plan
cargo test -p lance-context-master -p lance-context-api— 4 new unit tests for prune policy (count cap, TTL cutoff, protected dependencies, TTL-disabled); all pass.cargo fmt --check+cargo clippy --all-targetsclean on both crates.tsc + vite)./taskspages and shows total; confirm old terminal tasks expire after the TTL on the next task event / restart.🤖 Generated with Claude Code