Motion macroblocking: cause and fixes

[mauricio-gg]

Summary. A user reported heavy macroblocking (blocky, smeared video) the moment the player started moving, absent in static scenes and unaffected by the bitrate setting. Two causes were found and both planned fixes shipped: PR #186 (the receiver's reference-frame window now matches the Vita decoder's 8 frames) and PR #187 (an IDR keyframe is requested on the first unrecoverable missing reference, not the third). Both are on main. Two further ideas from the investigation were never built.

Symptom

A Reddit user reported, on both PS4 and PS5 Remote Play:

• Static scenes stay sharp. As soon as motion starts, the whole frame macroblocks.
• Setting the maximum bitrate changes nothing.
• It typically began about 10 seconds into a session.
• Instant onset on motion and instant recovery on stillness points to a reference-chain collapse, not a bandwidth limit.

Mechanism

Static scenes produce small P-frames (a few UDP packets). Burst loss on Vita Wi-Fi rarely exceeds the FEC parity the PS5 chose for such small frames, so FEC repairs them and the reference chain stays whole. Motion produces large P-frames (20-50+ packets of ~1000 bytes). The same burst loss now exceeds the parity, FEC fails (CHIAKI_ERR_FEC_FAILED in lib/src/frameprocessor.c), the frame is dropped, and every later P-frame that references it breaks. Until the next keyframe, the output is wrong. This is the same chain collapse described in the "Why FPS drops" thread, shown as visual corruption instead of lost frames.

Candidate causes, ranked at the time, and their outcome

1. Reference frame count mismatch (strongest lead). Fixed by PR fix(video): align receiver ref-frame window to 8 slots matching Vita HW decoder DPB #186. The receiver tracked 16 reference slots, but the Vita hardware decoder keeps only 8 frames (REF_FRAMES 8, vita/include/video.h:35). A P-frame referencing a frame 9-16 back passed the receiver's check, but the decoder had already evicted that reference, so it decoded against the wrong picture with no error flag to trigger recovery.
2. Passive missing-reference handling. Fixed by PR fix(video): request IDR on first unrecoverable missing ref, not third #187. An IDR was only requested after CASCADE_SKIP_THRESHOLD (3) consecutive unrecoverable missing references. At 30 fps that is about 100 ms of certain corruption before the console is even asked for a keyframe.
3. No client-side loss/bitrate adaptation. Not done. There is no protocol way to renegotiate bitrate mid-session, and the PS5 ratchets bitrate down on any reported loss and never raises it again (PR fix(congestion): report post-FEC effective loss to stop PS5 bitrate ratchet (GH #206) #213 / GH Wi-Fi burst jitter + receive-queue overflow drive PS5 bitrate floor-throttling (post-GH #188 track) #206 fixed loss being reported before FEC, which was ratcheting it needlessly). Proposed directions (log FEC failure against motion and bitrate; cap Vita bitrate for FEC margin) were not built as described. FEC failures on key frames are counted in the PIPE/STAGE log line (fec_fail_kf=, lib/src/videoreceiver.c:968), which partly covers the data-collection idea.
4. Hardware decoder never signals corrupt output. Not done, and probably not needed. The investigation worried the decoder could emit garbage silently. Later work (the #251 IDR-wait freeze thread) found sceAvcdecDecode never returned an error across 22 hardware logs, and moved to a different strategy: decide on the receiver side which frames to submit.

The two fixes that shipped

• PR fix(video): align receiver ref-frame window to 8 slots matching Vita HW decoder DPB #186 (merged 2026-06-25, fix(video): align receiver ref-frame window to 8 slots matching Vita HW decoder DPB). reference_frames[] shrank from 16 to 8 via a new CHIAKI_VIDEO_RECEIVER_REF_SLOTS constant, used by add_ref_frame, have_ref_frame and the alternate-reference scan. A P-frame whose reference the decoder has already evicted is now seen as "missing" and enters the recovery path, instead of decoding silently against the wrong picture. Today: lib/include/chiaki/videoreceiver.h:22 (with a comment tying it to vita/include/video.h:REF_FRAMES), used in lib/src/videoreceiver.c:10, 96-120.
• PR fix(video): request IDR on first unrecoverable missing ref, not third #187 (merged 2026-06-25, fix(video): request IDR on first unrecoverable missing ref, not third). Added a video_receiver_maybe_request_idr(..., "missing_ref") call inside the not-recovered branch of the frame flush, so an IDR is requested on the first unrecoverable missing reference. The existing 100 ms cooldown prevents flooding, and the cascade skip at depth 3 remains as a backstop. Today: lib/src/videoreceiver.c:826.

The plan staged them deliberately: #186 first and validated on device, then #187, because without #186 the missing-reference path fires less often and #187 has little to act on. The plan also noted a guard to add if Requesting IDR (missing_ref) fired more than about 5 times per second in stable streaming (limit the request to consecutive_missing_ref == 1). It was not needed as far as the repo shows.

What the investigation got wrong or that has aged

• Recovery time estimate. The doc estimated 600-1100 ms of macroblocking before clean video, built from "100 ms cooldown + ~500-1000 ms network and PS5 latency". That was an estimate, not a measurement. GH Video delivery is bursty: frame arrival gaps up to 706ms while the console renders on time — felt as input lag #251 later measured the PS5's IDR answer at 818-1499 ms, which is what dominates; fix(video): request IDR on first unrecoverable missing ref, not third #187 only removes the client's own delay.
• "Frames reached the decoder with wrong references" (step 7). True only for the 16-slot window mismatch that fix(video): align receiver ref-frame window to 8 slots matching Vita HW decoder DPB #186 removed. When a reference is genuinely missing, the receiver drops the frame by default (the picture freezes), and only the opt-in toggle from PR fix(video): submit-through-loss — replace the ~1s IDR-wait freeze with motion (#251) #257 submits it (lib/src/videoreceiver.c:821-846).
• The "silently rewrite to an alternate reference" step (step 6) never worked on the Vita: the H.264 rewrite function returns false (lib/src/bitstream.c:457-458; 0 successes in 112 attempts).
• Constants table. Today the IDR cooldown is 100 ms and timeout 1000 ms (videoreceiver.c:49-50), CASCADE_SKIP_THRESHOLD is 3 (:51), the receiver reference store is 8 (not 16), and the video jitter ceiling on the Vita is 40 ms (lib/src/takion.c:85, not 100 ms). The reorder queue is 256 entries and handles control messages only (takion.c:69); video and audio bypass it. The 200 ms congestion report interval and 10% loss clamp are unchanged (lib/src/congestioncontrol.c:5, 11).
• Validation. The PR test plans listed on-device checks (macroblocking duration, PIPE/STAGE skips= not rising, Requesting IDR (missing_ref) appearing within one frame of onset). In the PRs those boxes are unchecked and I found no recorded before/after measurement in the repo, so treat the benefit as expected rather than measured.

How to check this on a log

• grep -a "Requesting IDR (missing_ref)" should show requests close to motion onset (not only cascade_skip).
• PIPE/STAGE skips= should not rise compared with before the change.
• Use grep -a, not plain grep: BSD grep treats these logs as binary and silently returns nothing.

Source: docs/ai/MOTION_MACROBLOCKING_INVESTIGATION.md and docs/superpowers/plans/2026-06-25-motion-macroblocking-fix.md (deleted from the repo in #281; recoverable from git history at commit 0e9dd13)