The ~1s IDR-wait freeze (#251): findings and what fixed it

[mauricio-gg]

Summary. The "input lag" we chased for weeks was really repeated ~1 second video freezes. After a lost frame, the client threw away every following P-frame and held the last picture while the PS5 took 818-1499 ms to answer a keyframe (IDR) request. The fix, shipped as PR #257 ("submit-through-loss"), feeds those frames to the decoder instead of dropping them, so the picture keeps moving with brief smearing. It is behind a settings toggle that defaults to off. Status today: shipped, hardware-validated with the toggle on.

WARNING: do not trust or reuse the old latency numbers. The earlier latency documents (folded in below) quote "latency", "RTT" and "jitter" figures that came from a metric later shown to be fabricated. The log readout Latency metrics ... RTT N (base N, jitter N) is the one-time Senkusha handshake RTT (measured once at connect, never updated mid-stream) plus an EWMA of the inter-arrival deviation of control-channel packets (lib/src/takion.c:2147-2180). It is not a live round trip, and not video or network jitter: video and audio packets never feed it. Every "RTT 17-180 ms", "jitter ~60 ms" or "target latency 150-200 ms" figure from that era is invalid. Only the PIPE/* measurements named in this post (frame arrival gaps, IDR round-trip times) are real. The on-screen "Latency" value in the stream-stats panel is still computed this way (vita/src/host_metrics.c:513-520, shown by vita/src/video_overlay.c:172-174), so it should not be trusted either.

The symptom

The picture freezes for about a second at a time. On a bad stretch this happened roughly every 13 seconds. It felt like input lag, but the PS5's own monitor showed the game responding instantly. The console was receiving input and rendering fine; the delay was entirely in video delivery to the Vita.

The mechanism (measured)

1. Wi-Fi drops a burst of packets. One video frame fails FEC and is lost.
2. Every later P-frame references it. With the reference gone, the client set succ = false and discarded each one: 26 consecutive frames in the worst case measured.
3. The client requests an IDR keyframe.
4. The PS5 took 818-1499 ms to deliver it. The eight recorded IDR round trips in one session: 942, 1032 (timed out, none arrived), 1183, 914, 973, 1240, 818, 1499 ms.
5. The screen held the last good frame for that whole window.

Eight recorded events in a two-minute session is several seconds of frozen video. Source logs: three hardware capture sessions; the richest was 92941169310_vitarps5-testing.log.

The client-side bug found on the way

chiaki_bitstream_slice_set_reference_frame() returns false for H.264 (lib/src/bitstream.c:457-458; rewriting is implemented for H.265 only). The Vita is H.264-only (lib/src/session.c:111 pins the codec; the decoder is SCE_VIDEODEC_TYPE_HW_AVCDEC). So the "use an older reference instead" recovery block in lib/src/videoreceiver.c never succeeded: 0 successes in 112 attempts. Making it work was rejected, because all 112 observed slices carry no ref_pic_list_modification, so redirecting a reference would mean inserting Exp-Golomb syntax and bit-shifting about 12 KB per frame on the receive thread.

What was ruled out (do not re-investigate)

Suspect	Evidence
Vita decode/display path	decode_avg_ms 1.7 at window 8 and at window 112; display_fps tracks incoming; 4 frame overwrites in 2 min; 0 decode-queue drops; stuck_streak/cascade_streak 0
Client socket backlog / recv-thread stall	Audio shares the same socket and recv thread; its queue stayed flat at 1.50 with zero catch-up events straight through a 757 ms video gap; drain_max=1
Console send pacing (VBR through a byte-rate pacer)	Gap after a large frame vs a small frame: -0.09 ms across 3,376 attributed gaps
Client Wi-Fi power-save batching	Burst-release signature (gaps under 10 ms) was 5 of 3,543 gaps (0.14%)
PS5 bitrate ratchet (#237)	Frame size 2.59 Mbps before the episodes, 2.78 Mbps after. Did not occur
Loss over-reporting (#238)	0 of 119 windows had reported_precap > raw_lost; session loss 0.17%
PS5 ignoring our IDR request / fixed GOP schedule	Keyframes appear 1-2 s after every request, and there are 16-25 second stretches with zero keyframes when we do not ask
Our IDR request being lost	Sent reliably via chiaki_takion_send_buffer_push; resend timeout 200 ms x 10 tries (lib/src/takionsendbuffer.c:13)

Steady-state delivery is ordinary link smear of about +/-10 ms and is not the problem. The largest frame in the session was 30,792 bytes, about 91 ms on the wire, so frame size does not explain the console's ~1 s either.

The fix plan (implemented as PR #257)

Research conclusions behind it:

• Slice-header rewriting is dead (see above) and also unnecessary. The lost frame was never submitted, so the decoder's reference buffer still holds the last good frame as its newest entry. A P-slice with the default reference list resolves to "newest in the buffer", so submitting the frame as-is already predicts from the last good frame.
• The Vita decoder tolerates this. vita-moonlight fix(psn): restore duid device provenance to fix PSN internet connect 403 (#204) #205 shows the same hardware decoding imperfect H.264 with artifacts but continuous motion. Across 22 hardware logs here, sceAvcdecDecode never returned an error.
• The one real hazard is the SPS flag gaps_in_frame_num_value_allowed_flag. If it is set, a conforming decoder invents phantom frames for the gap. If it is 0, a submitted frame lands in the benign "wrong reference, bounded drift" case. So the feature only activates when the parsed SPS proves the flag is 0 (chiaki_bitstream_h264_drift_safe(), lib/src/bitstream.c:463).
• No protocol alternative exists. Takion has no reference-frame-invalidation verb (the Moonlight/GameStream "RFI" approach): CORRUPTFRAME is a bare frame range with undocumented console semantics, IDRREQUEST carries no payload, and the launch spec has no error-resilience knobs. Official Sony clients reportedly freeze ~1 s the same way, so the console's IDR latency is treated as irreducible.
• The IDR request path was left untouched as the backstop.

Plan steps: parse the SPS fields; add decoder-anomaly counters; add an IDR-arrival discriminator (to tell "console slow" from "keyframe died in the same loss burst"); add the toggle and flip the discard policy at the single discard gate; run a hardware A/B. Open follow-up ideas recorded in the plan: send idr_period in the server settings (defined in takion.proto, never sent by any client), and use HEADERREQUEST(VIDEO) as an SPS/PPS refresh.

What main does today for the IDR wait

• With Settings -> "Motion during loss (artifacts)" on (submit_on_missing_ref, default off: vita/src/config.c:57), a P-frame with a missing reference is requested-IDR and then submitted anyway: lib/src/videoreceiver.c:821-846 (PIPE/DRIFT_SUBMIT, logged once per episode). With it off, the frame is dropped (videoreceiver.c:851-858) and the picture holds. The IDR request is sent immediately in both modes (videoreceiver.c:826), cooldown 100 ms, timeout 1000 ms (videoreceiver.c:49-50). After 3 consecutive failures the cascade skip kicks in (videoreceiver.c:51, :658).
• Hardware forensics (PR fix(stream): resume resync + transport-death recovery — kill the silent freeze paths (#258, #261) #260 description, log 99798574648) measured drift-ON loss episodes freezing 16-33 ms against 300-1040 ms for the OFF path. PR fix(stream): resume resync + transport-death recovery — kill the silent freeze paths (#258, #261) #260 recommended running with the toggle ON; it is still off by default.
• PR feat(video): staleness-gated presentation — freeze-then-jump for backlog drains (#262) #263 (merged) added a "freeze-then-jump" stale-backlog gate on top, after log 104199521041 showed no delay actually accumulates and the "lag with the toggle on" report was the client displaying the console's fast backlog drain. PR refactor: retire the abs= readout, the dead #262 stale-hold gate and PIPE/FPS stale= (#278) #280 later removed that whole gate (its cadence tracker, staleness_ms plumbing and hold state machine), the abs= readout, and the stale counters. What remains in vita/src/video.c is the older bounded freeze: a corrupt frame shows the last good picture for at most FREEZE_MAX_STREAK = 8 consecutive frames (video.c:174-179, :1159).

The latency documents, folded in

The two older documents (docs/LATENCY_ANALYSIS.md, docs/LATENCY_QUICK_WINS.md) were a measurement plan and a list of "quick wins" written before the above was understood. The analysis document was a checklist of areas to instrument (video, audio, network, input) with a target of "<100 ms remote, <50 ms local" and estimated current figures of "150-200 ms remote". Those estimates were guesses, not measurements, and they are invalid per the warning above. The one concrete item it recorded was the GH #208 ENOBUFS fix (see the "Tried and reverted" thread and PRs #209/#244).

Status of every item in the quick-wins list, checked against main (0e9dd13):

#	Quick win	Status today
1	VBlank wait disabled while streaming	Shipped (vita/src/video.c:1389; restored on stop, :1477)
2	FPS locked to 30	Changed: now a setting, force_30fps, default off (vita/src/config.c:54, applied at vita/src/host.c:473-475)
3	Video thread priority 64	Shipped (VITA_DECODE_THREAD_PRIORITY, vita/include/video.h:39)
4	Video thread pinned to CPU 0	Obsolete: decode is now its own thread on USER_1 (PR #199; video.c:1329-1331); recv is USER_0 (lib/src/takion.c:1278), audio USER_2 (vita/src/audio.c:208)
5	Input thread polls every 5 ms, later 2 ms	Shipped at 2 ms (vita/src/host_input.c:270)
6	Audio thread priority 64 / CPU 0	Shipped at priority 64 (vita/include/audio.h:5), but pinned to USER_2, not CPU 0
7	Input thread priority 96 and pinned CPU 1	Priority shipped (vita/include/host_input.h:11); pinning dropped (affinity mask 0 = unrestricted, host_input.c:262)
8	Input polling reduced to 2 ms	Shipped (same as 5)
9	Network queue trimmed to 8 packets	Obsolete: the reorder queue is 256 entries and carries control messages only; video/audio bypass it (lib/src/takion.c:69)
10	512 KB UDP receive buffer	Shipped (takion.c:51; see the socket-buffer thread)
11	Batch drain loop (64 packets)	Shipped, cap raised to 256 (PR #192; takion.c:80)
QW5	Disable frame pacer	Shipped, with a caveat: the old enable_frame_pacer flag is gone. A display pacing gate exists, should_drop_frame_for_pacing() (vita/src/video.c:311-330), but it does nothing unless force_30fps is on (default off)
QW6	Hardware H.264 decoder low-latency mode	Not done as written (never researched in the code); decoder is configured with numOfStreams = 1 and REF_FRAMES = 8 (video.c:712-713). Decoder never errored across 22 logs, and decode averages 1.7 ms, so the gain is unproven
QW7	Lower resolution (360p/480p)	Obsolete as framing: 720p/1080p are forced down to 540p (vita/src/config_values.c:18-29); lower is a user setting
P3-8	Buffer-pool malloc elimination	Shipped, by a different route: PR #192 removed the per-AV-packet receive malloc rather than adding a pool (takion.c:1332-1336)
chiaki-ng 1	Disable IP fragmentation after Senkusha	Rejected (see below); on Vita takion.c:429-433 only logs a warning
chiaki-ng 2	FEC error concealment	Already in the Chiaki base
chiaki-ng 3	Audio lag building over a session	Not done (no audio jitter buffer or lag monitor; lib/src/audioreceiver.c has none)
chiaki-ng 4	Report corrupt frames earlier	Partly: corrupt-frame ranges are reported with hold/force spans (videoreceiver.c:15-17) and IDR is requested on the first missing reference (PR #187)
chiaki-ng 5	More slots at high bitrate	Not done, low relevance
chiaki-ng 6	VSync off on PS5 side	Not verifiable from the client; not done

Two experiments from the same document were abandoned and are worth remembering:

• Dual callback architecture (Dec 2025). Splitting Takion's callback into an audio/video fast path and a control path was meant to save 0.6-2.5 ms per packet. It raced with initialisation (AV packets arriving before the stream info finished), needed ARM atomics to be safe, and in practice felt choppier. Abandoned; the unified callback stays. Lesson: lock-free tricks must be validated on hardware, because microbenchmark gains do not always become user-visible gains.
• IP_DONTFRAG on the Vita (Dec 2025, v0.1.417). setsockopt(IP_DONTFRAG = 28) returned -1 (option not supported). The Vita's BSD stack does not implement it, so fragmentation cannot be disabled and there is nothing to gain here.

Source: docs/ai/wip/251-handoff-prompt.md, docs/superpowers/plans/2026-08-14-idr-freeze-fix-251.md, docs/LATENCY_ANALYSIS.md and docs/LATENCY_QUICK_WINS.md (deleted from the repo in #281; recoverable from git history at commit 0e9dd13)