Skip to content

A B stability testing

Jump to bottom
Mauricio G. edited this page Oct 8, 2026 · 1 revision

A/B stability testing

How to compare two builds (or two network setups) so that a difference you see is real, using the logs from a testing build.

1. Build the same way both times

Use ./tools/build.sh --env testing for both candidates and install each VPK on the Vita. Do not use ./tools/build.sh test: it only compiles the test binary and makes no VPK. If behaviour did not change, check that you installed a new VPK. See Build and deploy.

Keep one thing different between the two runs, and note the VPK name (VitakiForkv0.1.<N>.vpk).

2. Run the same scenario

  • Same console, same game and scene, same place and network.
  • At least 5 minutes each.
  • Do not move the router or yourself between runs.
  • Keep latency_mode and other options identical in ux0:data/vita-chiaki/chiaki.toml (see Configuration).

3. Pull each log

Each run writes a new file in ux0:data/vita-chiaki/, named <number>_vitarps5-testing.log. Copy it off (see Logging) and rename it for the run, for example A-baseline.log and B-candidate.log.

4. Check the setup lines

At the start of each log, confirm both runs used the same settings and know which build made them:

grep -a -E "PIPE/BUILD|Config loaded|Bitrate policy|Recovery profile|Video gap profile" run.log
  • [PIPE/BUILD] commit=... branch=... dirty=...: which code ran.
  • Config loaded: latency_mode=...: the settings read from the file.
  • Bitrate policy: preset_default (<kbps> kbps @ <w>x<h>): the starting bitrate.
  • Recovery profile: stable_default and Video gap profile: stable_default (...): the recovery settings (not user-adjustable).

Always use grep -a; plain grep may print nothing. And check for LOG_LINES_DROPPED: if present, the log lost lines and the comparison is weaker.

5. Compare health

grep -a "PIPE/FPS" run.log      # once per window
grep -a "AV diag" run.log
grep -a "PIPE/RECOVER" run.log
grep -a "CONGESTION/LOSS" run.log
  • PIPE/FPS: incoming= against target= frame rate over time, plus low_windows=, freeze=, display_fps=, decode_q_drops=, windowed_mbps= and rssi= (Wi-Fi signal).
  • AV diag: missing_ref, corrupt_bursts, fec_fail, sendbuf_overflow. Counts that keep climbing between windows mean damaged or missing video data.
  • PIPE/RECOVER ... action=...: the app noticed low FPS and acted. Many of these, or repeated request_stream_stop, means the stream was unstable.
  • CONGESTION/LOSS received=... raw_lost=... fec_recovered=... reported=...: once per second, how many packets were lost and how many error correction repaired.

6. Decide

A healthier run, compared with the other:

  • incoming stays near target, with few or no low_windows.
  • missing_ref, corrupt_bursts and fec_fail stay at or near zero.
  • Few PIPE/RECOVER actions and no repeated stops or reconnects.

If one run looks grainy or choppy but never disconnects, look at missing_ref and corrupt_bursts first. If the two runs look identical although you changed something, check that the right VPK is installed and that you are reading the right log file.

Run each side more than once if you can: one session is one sample, and Wi-Fi conditions change from minute to minute.

Clone this wiki locally