perf(web): speed up large dataset dashboards

[flamboh]

Problem

Dashboard latency grew with the product database rather than the requested window. The main indexes placed bucket_start before source and visibility filters, so SQLite scanned unrelated scopes. The page also fetched and rendered every chart on mount, repeated metadata and coverage queries, recreated Chart.js instances, and rebuilt timezone formatters in hot loops. Those costs were especially visible over an SSH tunnel.

Solution

• Add query-aligned timeseries indexes and refresh planner statistics after pipeline writes, feed passes, database publication, and the D1 migration. Keep the bucket-first indexes still used by file-detail queries and remove the two redundant indexes that only added write cost.
• Pin local SQLite handles for the lifetime of each request. Metadata caches now detect atomic database replacement and WAL-only commits without closing a handle that another request still uses.
• Reduce server work with targeted dataset lookup, authoritative source_members, shared coverage reads, parallel characteristics queries, conditional port aggregation, and smaller IP/protocol response objects.
• Add a bounded, expiring, abort-safe window cache. Failed rollover fetches leave existing cached windows intact.
• Reuse Chart.js instances, cache timezone conversion, avoid partial-coverage callbacks for complete series, and render isolated partial values as hollow points.
• Mount only the first visible chart. Deferred cards load near the viewport or through an accessible button, retain their layout height, and use the latest filters when activated.
• Add a repeatable large SQLite fixture and failure-sensitive dashboard and interaction profilers.

Measured on the generated 880 MB fixture with 16 sources and 80 ms simulated latency:

• Initial dashboard work drops from 6 API requests and 8 canvases to 1 request and 1 canvas.
• Initial decoded API data drops from 5.18 MB to 407 KB.
• Initial post-GC heap drops from about 27 MB to 6.9 MB.
• The initial visible chart is ready in about 1.84 seconds.
• Returning to a cached date range takes about 183 ms and sends no API requests.

The query-aligned traffic index reduced a 2.88 million-row synthetic query from 239.5 ms to 82.4 ms.

Review fast path

Build the large fixture and start the preview:

bun run perf:web:seed
bun run build:web
ATLANTIS_DB_DRIVER=sqlite LOCAL_SQLITE_PATH=/tmp/atlantis-web-perf.sqlite \
  bun run --cwd apps/web preview --host 127.0.0.1 --port 4173

Run the profilers from another shell:

bun run perf:web:profile
bun run perf:web:interactions

Flows to exercise:

1. Open the dataset dashboard. Only the first card should request data and mount a canvas.
2. Scroll through the remaining cards. Each endpoint should load once, with no large layout jump.
3. Reorder a deferred card to the top and reload. The persisted first card should be the only initial request.
4. Change filters before loading a deferred card. Its request should contain the new values.
5. Extend a date range, then restore it. The restore should read the window cache without another request.
6. Hover partial and complete coverage data. Partial numeric buckets remain visible; hover must not resize cards.

Important decisions and edge cases:

• Window-cache entries expire after 30 seconds and have hard entry and fill limits. Abort and failed rollover paths preserve valid cached data.
• Atomic product replacement retires the old connection only after its last request releases it.
• WAL changes invalidate cached dataset and source metadata even when the main database file is unchanged.
• D1 migration 0002 runs PRAGMA optimize after creating indexes so the planner can select them immediately.
• Coverage remains in each endpoint response. A shared wire format could save more bytes, but changing that data contract was not worth the added coupling in this PR.

Verification

• bun run format
• bun run lint
• bun run typecheck
• bun run test: 133 web tests and 133 Rust library tests, plus all Rust integration suites
• Browser suite: 6/6 tests in parallel
• bun run build:web
• bun run --cwd apps/web db:generate: no schema changes
• Final 880 MB dashboard and cache profiles at 80 ms RTT
• Four adversarial review rounds; the final fresh-context pass found no remaining change that cleared the evidence and complexity bar

Remaining manual verification: none.

---

Built with GPT-5.6 Sol in T3 Code, with parallel Codex subagents for profiling, implementation, and adversarial review.