Skip to content

test: fix time-of-day flake in the durable-headline project-filter tests - #875

Merged
iamtoruk merged 1 commit into
getagentseal:mainfrom
GodCC6:fix/project-filter-test-time-of-day
Aug 3, 2026
Merged

test: fix time-of-day flake in the durable-headline project-filter tests#875
iamtoruk merged 1 commit into
getagentseal:mainfrom
GodCC6:fix/project-filter-test-time-of-day

Conversation

@GodCC6

@GodCC6 GodCC6 commented Aug 3, 2026

Copy link
Copy Markdown

Summary

  • The tests I added in fix: --project/--exclude are ignored by the durable headline totals #864 fail for half of every day. They seed today's live session at a fixed wall-clock hour (12:00 local) while the periods they build end at new Date(), and the suite runs under TZ=UTC — so on any run before 12:00 UTC that timestamp is in the future, the range filter correctly drops it, and the live half of the cache/live union contributes nothing.
  • With the headline then at $0, the one assertion that needs a non-zero total — the unattributedCostUSD footnote — hits renderOverview's No usage found early return and goes red. a769b50 fixed the start-of-month flake in the same file; this one survived it.
  • Fixes the fixture (seed a few minutes before now, clamped into today) and removes the footnote test's dependence on the live parse entirely.

Testing

  • I have tested this locally against real data (not just unit tests)
  • npm test passes
  • npm run build succeeds

npm test: 2,537 passed, 5 failed — none in this file; the 12 tests here are green. The 5 are parser.test.ts ×2 (the pre-existing pair you already noted), cli-durable-totals.test.ts "resolves provider filters identically" (fails on main, unrelated — parseAllSessions returns nothing for a today-only range once the on-disk session cache is warm), and cache-refresh-lock.test.ts ×2 (lock/timing, flaky here run-to-run). npx tsc --noEmit clean.

Evidence

Bisected the fixture on the unpatched test file, at a 03:38 UTC run:

seeded hour relative to now result
12:00 (as merged) future 1 failed | 11 passed
01:00 past 12 passed

Nothing else changed, so the timestamp's position relative to now is the entire cause. A direct probe of parseAllSessions confirms the mechanism — same fixture, same run:

wall-clock 12:00 fixture, range end = new Date()  -> 0 projects, $0.00
relative "10 minutes ago" fixture, same range     -> 1 project,  $1.08

I also tried to demonstrate this across timezones and got misled: TZ=... has no effect because the vitest config pins TZ=UTC, so every "timezone" run was the same 03:37 UTC morning. The bisect above is the reliable evidence.

Fix

  1. Fixture timestamps are now relative and clamped: max(today's midnight, now − 10min), so the session is always inside today and already in the past, at every hour.
  2. The footnote test no longer needs the live parse. It seeds a second, attributable cached day alongside the unattributable one, so the headline is non-zero from the cache alone. The assertion now exercises the footnote rather than the fixture's timing — which is what it was meant to test.

Sorry for the follow-up on my own tests; this was mine to catch, not yours.

…day proof

The tests added in getagentseal#864 seeded today's live session at a fixed wall-clock
hour (12:00 local). The periods they build end at `new Date()`, and the
suite runs under TZ=UTC, so for any run before 12:00 UTC that timestamp
is in the FUTURE and the range filter correctly drops it. The live half
of the cache/live union then contributes nothing, and the one assertion
that needs a non-zero headline — the unattributed-cost footnote — fell
into renderOverview's "No usage found" early return and went red. Half
of every day was a failing window; a769b50 fixed the start-of-month
flake but this one survived it.

Verified by bisecting the fixture on the unpatched test: moving the
seeded hour from 12:00 to 01:00 (past, at a 03:38 UTC run) turns the
same 12 tests green, so the timestamp's position relative to `now` is
the whole cause.

- Seed the session a few minutes BEFORE now, clamped to today's
  midnight, so it is always both inside today and already in the past.
- Stop the footnote test depending on the live parse at all: seed a
  second, attributable cached day so the headline is non-zero from the
  cache alone. The test now exercises the footnote instead of the
  fixture's timing.

@iamtoruk iamtoruk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — this fixes a real second flake that survived my #864 test fix (a769b50), not a duplicate of it. The tests seeded today's live session at a fixed 12:00 local while the periods end at new Date(), so under TZ=UTC any run before noon put that timestamp in the future, the range filter dropped it, and the live half of the cache/live union contributed $0 — which sent the unattributedCostUSD footnote test into renderOverview's "No usage found" early return.

Reproduced the mechanism deterministically (a future-dated live session → durable.data.cost=0, unattributedCostUSD=100), and confirmed your fix: relative-and-clamped live timestamps stay in the past at every hour, and seeding a second attributable cached day keeps the headline non-zero from the cache alone so the footnote assertion no longer depends on the live parse. All 12 pass; tsc clean; test-only. Good to merge.

@iamtoruk
iamtoruk merged commit e05db46 into getagentseal:main Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants