Kiro CLI token totals inflate without bound once conversations pass 90 days

[pitimon]

Summary

Kiro CLI token totals grow on every sync, forever, for any conversation older than 90 days. Confirmed by reproduction outside the test harness, not inferred.

Mechanism

readKiroCliRequests (src/lib/rollout.js:3694) reads conversations_v2 with no age filter — its SQL ends at WHERE json_extract(value, '$.user_turn_metadata.requests') IS NOT NULL.

clampAndCapKiroCliState (src/lib/rollout.js:4054-4058) prunes every per-request cursor older than KIRO_CLI_CURSOR_MAX_AGE_MS = 90 * 24 * 60 * 60 * 1000 (:4038).

That cursor is the sole idempotency guard for the SQLite path — src/lib/rollout.js:3780 notes "Per-request state replaces the old seenIds set". The read window is unbounded while the memory window is 90 days, so once a request ages past 90 days it is re-ingested on every subsequent sync.

Reproduction

Four consecutive syncs against a 91-day-old fixture: eventsAggregated=1 every time, bucket totals input = 100 → 200 → 300 → 400. An identical fixture dated 5 days ago: eventsAggregated = 1,0,0,0, totals stay at 100.

Downstream makes it worse rather than better: dedupe on (source, model, hour_start) keeps the latest row, so the inflated snapshot wins.

How it surfaced

Four parseKiroCliIncremental tests in test/rollout-parser.test.js went red with no code change. They pin fixtures to 2026-04-20T10:05:00.000Z, whose half-hour bucket crossed the 90-day cutoff on 2026-07-19T10:00Z — between a green CI run on 07-10 and today. Re-running with the clock faked back to 07-10 gives 147 pass / 0 fail. Bisecting the fake clock lands on exactly that hour.

This is why main is currently red and why #64 cannot go green.

Why the obvious fixes were rejected

• Drop the age cap, keep the 20k count cap — converts an age-triggered double-count into a volume-triggered one. Above 20,000 in-window requests the overflow re-counts every sync. Also breaks test/rollout-parser.test.js:4910, which asserts the age prune.
• Age-filter the reader to match the cursor — fail-safe and small, but makes Kiro the only source with a 90-day ingestion horizon. The dashboard offers a total period (dashboard/src/pages/DashboardPage.jsx:53) and nothing ever prunes cursors.hourly, so full history is a product commitment. A fresh install importing an older database, or any resync after cursor loss, would silently lose Kiro history beyond 90 days.

Direction

Per-bucket high-watermark ledger: cursors.kiroCli.ledger keyed by (model, bucketStart) records what this parser has already applied to each hourly bucket. Ingest recomputes the desired contribution from the full reader output and applies only the delta. State is then bounded by buckets (~48/day/model) rather than by requests, so no prune policy — age, count, or corruption — can cause a recount. The per-request cursor stays for retraction bookkeeping and eventsAggregated, demoted off the correctness path.

Writes must stay delta-only: source="kiro" is shared by three writers (:3019, :3978, :4191), so absolute assignment would erase Kiro IDE tokens.

Switching from accumulate to reconcile makes an unreadable database destructive — it reads as "everything was deleted". That prerequisite ships first (#66).

Known limitation

Adoption is net-zero: existing inflated totals are frozen, not repaired. Repairing them needs the current Kiro-CLI portion of each shared bucket, which is exactly what is unknown. An opt-in full resync is the only clean remedy and is out of scope here.

[pitimon]

Fixed in #70 (7dc7e96).

Root cause, confirmed: readKiroCliRequests reads the whole conversations_v2 table with no age filter, while cursors.kiroCli.requests was pruned at 90 days and capped at 20,000 entries. The memory window was strictly smaller than the read window, so a request evicted by either cap was re-ingested every sync. Note the issue's rejected-options list only considered dropping the age cap while keeping the count cap — dropping both was never evaluated, and that is essentially the fix.

Fix: keep the same per-request record, never prune it, and only ever add growth: delta = max(0, current - recorded). The delta is clamped at zero because attribution is only ever needed to justify a decrease — so never decreasing removed the need for a health gate, and left cursors.hourly (shared by 11 parsers, including the Kiro IDE writing the same source="kiro" buckets) untouched.

Two earlier designs were built and rejected under review before this one. A per-bucket signed-delta ledger is archived on fix/65-kiro-ledger — it is kept for its evidence, not its solution. Its failure taught the thing that shaped the final design: a signed delta forces you to answer "which producer owns this decrease?", which the data model cannot answer, so every boundary case became a new defect.

Worth carrying forward: green gates were not evidence at any point. The rejected ledger was 796/796 with ci:local passing while carrying a CRITICAL and three HIGH defects, because its new branches had no coverage. Four separate tests were caught passing for the wrong reason — two with >=/<= assertions that could not fail in the direction they existed to catch, one that stopped discriminating its own fix after a later refactor, and one that took an early-return path skipping the hazard it targeted.

Known behavior changes (in the PR body and worth release notes): a genuine downward correction never propagates; a re-bucketed request stays labelled under its original bucket; upgrading freezes prior inflation rather than repairing it; and state now grows with Kiro's own history (~194 bytes/request) with cursors.json rewritten whole on every sync.

Follow-ups noticed during this work, not part of this fix — worth separate issues: normalizeHourlyState overwrites rather than sums colliding buckets during the v1→v3 re-key (silent loss affecting all parsers); readJsonStrict exists unused in fs.js while readJson swallows SyntaxError, so a corrupt cursors.json reads as absent and triggers a full re-count for every parser.

[pitimon]

Note: #66 (readSqliteJsonRowsWithStatus) was the base of #70's branch, so its commits are included in the squash at 7dc7e96 and it is now redundant.