Repository navigation
Full-app A/B at every rebuild: a catch-all for regressions on a production box #2065
vpacher
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I run the fork in production on a headless mixing box (see #1078). Every time I rebuild it from master, I measure the old build and the new build on the same day, under the same load. This post shows the first five rebuilds (2026-10-18 to 2026-10-23).
Why.
port_comparisonand the crate benches measure one function at a time, and #1305 is about recording evidence on dedicated runs. This series measures the whole application under a real, long-running load. A regression in code that no bench covers (a thread that wakes too often, a per-frame copy, a lock in the render path) shows up here as more CPU or more frame time, even when nobody thought to bench it. It is a tripwire, not a way to find the cause: one rebuild covers 56 to 105 commits.Setup
Method
During both windows, a sampler on the host records the container CPU every 10 s and the top threads in the container every 60 s. A 120 s sample measures the CPU and wakeups/s of the UI (main) thread. Skipped frames come from
GetStatsover obs-websocket. The graphics thread time per frame comes from the profiler summary in the OBS log at shutdown.Results
The shutdown log gives the graphics thread time over a whole run. The median went from 1.905 ms to 1.783 ms over the last three builds, and the 99th percentile from 2.985 ms to 2.336 ms. The runs had different lengths (4 h, 9.7 h, 1.2 h), so the trend is the useful part, not the exact values.
Limits
I'll publish more results here periodically, when time allows. If a change goes clearly outside the band, I'll open an issue with the commit range. If the method or the scripts are useful to someone else, I can share them.
All reactions