Skip to content

feat(dash-apps): add web dash-app performance harness - #161

Closed
dkneeland wants to merge 3 commits into
mainfrom
feat/dashapp-perf-harness
Closed

feat(dash-apps): add web dash-app performance harness#161
dkneeland wants to merge 3 commits into
mainfrom
feat/dashapp-perf-harness

Conversation

@dkneeland

@dkneeland dkneeland commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a Playwright-based perf harness for web dash-apps with budgets, fixtures, and test coverage
  • run perf checks in GitHub Actions for pull requests and pushes to main when web dash-apps or the harness change
  • gate targeted apps via manifest perf metadata, including exempt apps and built-subdirectory support for web-expo
  • document local usage and tweak points for budgets and CI behavior

Verification

  • npm test (in dash-apps/perf-harness)
  • reran web-retro with the current harness against current HEAD and historical commit

@dkneeland
dkneeland force-pushed the feat/dashapp-perf-harness branch 3 times, most recently from 5f7244e to d6ddcb9 Compare July 6, 2026 23:05
@dkneeland

dkneeland commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Renamed dash-apps/perf-harness → dash-apps/_perf-harness/ so the harness folder always sorts to the top of the directory, visually separating it from the web apps.

Decided to keep it inside dash-apps/ to keep all dash-app-related work co-located. If you'd prefer it elsewhere in the project (e.g., as a sibling of dash-apps/), happy to move it before merging.

@dkneeland

dkneeland commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Added dashpilot-android/**/CarStateBridge.kt to the workflow path triggers (869631e). Without this, a PR that edits only CarStateBridge.kt never runs the drift test in scenario.test.js:109 — the workflow's existing
pm test step catches getter contract drift, but only if the workflow actually fires on bridge edits. Same rationale as the dash-apps/** paths: the contract goes both ways.

@dkneeland
dkneeland force-pushed the feat/dashapp-perf-harness branch 2 times, most recently from 6ab60cf to 6021677 Compare July 12, 2026 19:01
@dkneeland

dkneeland commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator Author

Sorry for that extra commit. This branch is clean again.

This was referenced Jul 12, 2026
Open
Merged
@wpmed92

wpmed92 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

@dkneeland Is there a way we can make this PR shorter? I mean, is there anything that can make this performance test harness feature conceptually simpler? 1600 lines plus is a lot of line for this. Any low hanging fruit? Any boilerplate that's avoidable?

@dkneeland dkneeland mentioned this pull request Jul 12, 2026
Merged
@dkneeland
dkneeland force-pushed the feat/dashapp-perf-harness branch from bcbf0ca to 802fb43 Compare July 14, 2026 03:03
@dkneeland
perf-harness: strip to 3 core metrics, drop targets/gate module, simp…
51523fa 
…lify CLI and shim

- Removed updateToPaintP95Ms, handlerMs, heapGrowthMb, scheduledMs
- Removed multi-run pick-best, CI gate selection (lib/targets.js)
- Replaced custom arg parser with node:util.parseArgs
- Duration via PERF_DURATION env var (default 60s -> 20s)
- Scenario periods compressed, warmup 5s -> 2s, budgets 14 -> 6
- Shim samples dropped from 5 fields to 2 (actualMs, injectionLagMs)
- Single run, no loop, --enable-precise-memory-info removed
- Updated CI workflow to inline bash (no targets.js)
- Updated README for 20s scenario, PERF_DURATION, workflow path
- Net -333 lines, 12 files changed
@dkneeland
dkneeland force-pushed the feat/dashapp-perf-harness branch from 55bdca6 to 51523fa Compare July 14, 2026 03:16
@dkneeland
dkneeland marked this pull request as draft July 14, 2026 03:58
@dkneeland

dkneeland commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Simplified harness — jsdom tripwire model

This replaces the Playwright-based browser gate with an in-process jsdom tripwire. Key changes:

Architecture: Instead of launching Chromium and running the app over HTTP, the harness now loads the app's HTML directly into a jsdom environment (via drive.js) with an injected ick() replacement. This eliminates network round-trips and browser overhead.

Metrics:

  • tickCpuP95Ms (Worst-Case Update Response Time): The maximum time (95th percentile) the app spends running its JavaScript code the instant a new telemetry update arrives. Spikes here directly cause visual stutters.
  • mutationsPerTickP95 (Screen Rebuild Overhead): The number of separate parts of the webpage changed during an update. A low number means efficient, surgical DOM updates; a high number (like 50+) means the app is rewriting the DOM from scratch.
  • meanProcessCpuPerTickMs (Average Overall Workload): The average processor effort spent per update. This captures delayed background tasks, animations, or framework rendering work that runs between update ticks.
  • steadyFrames (Clean Updates Analyzed): The count of healthy frames analyzed after the initial 2-second "warm-up" period. A minimum count is required to ensure the test gathered enough data to be statistically reliable.

Hardening:

  • Unhandled rejections and swallowed window.onerror both exit 2 (APP_ERROR)
  • Fail closed when the steady sample base is truncated
  • Fast-app CPU burn loop calibrated for Windows process timing
  • First 2s warmup excluded from mean-process-cpu
  • Integration test suite cut from ~36s to ~18s

What was dropped: Chromium launch, network server, Playwright dependency, whole-run processCpuMs metric, WASM/GLB/WGSL MIME shims, scenario/targets/gate modules, shim.js.

@dkneeland
dkneeland marked this pull request as ready for review July 18, 2026 20:00
@dkneeland @claude
feat(perf): replace Playwright gate with jsdom tripwire harness
cc75896 
Migrate from Chromium/Playwright browser gate to a lightweight jsdom
tripwire model. The harness now runs app code synchronously in-process
via a jsdom DOM shim, measuring per-tick CPU, DOM mutation cost, and
steady-state process CPU.

Changes:
- Replace shim.js (network-fetching Playwright helper) with drive.js
  (in-process jsdom tripwire runner)
- Add crash detection for suppressed window.onerror and unhandled
  rejections (exit 2)
- Calibrate the fast-app CPU loop for Windows process timing
- Exclude warmup (first 2s) from mean-process-cpu metric
- Fail closed when the steady sample base is truncated
- Treat jsdom environment gaps as benign, not APP_ERROR
- Trim harness cost: drop whole-run processCpuMs, Chromium-era MIME
  support, deduplicate result shape, share fast-app drive across tests
- Shorten integration-test durations (~36s -> ~18s)
- Document the single-stall blind spot and exemption rationale
- Document key performance metrics in README

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dkneeland
dkneeland force-pushed the feat/dashapp-perf-harness branch from 27a2656 to cc75896 Compare July 18, 2026 20:57
@dkneeland

dkneeland commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this as it has been deemed too complex for the value it brings

@dkneeland dkneeland closed this Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dkneeland @wpmed92