Remote Play smoothness: ranked userland-only ideas

[mauricio-gg]

Summary. A ranked list of 17 userland-only ideas for smoother PS5 Remote Play on the Vita, written in August 2026 when the PSN internet path was "the choppiest". Status today: 7 shipped (some in a different form), 1 partly shipped, 9 not done. The ones that moved the needle most were the thread and core layout (item 1), decoding on its own thread (item 11) and the PSN bitrate cap (item 3). The thread-priority reasoning in this post (why the feedback sender sits at priority 65 next to the AV threads at 64) is cited from a code comment in lib/src/feedbacksender.c, so it is kept in full below.

Platform facts the plan relied on

Checked against main (0e9dd13) unless marked.

• The Vita has a single ~333 MHz-class ARM Cortex-A9 budget for app code (444 MHz maximum userland clock), NEON, 256 MB RAM, 2.4 GHz 802.11b/g/n only, and a dedicated H.264 hardware decoder. Userland already runs at maximum clocks (vita/src/main.c:53-56: ARM 444, GPU 222, BUS 222, GPU crossbar 166). More needs a taiHEN kernel plugin and is out of scope.
• The Vita has exactly three usable user cores: USER_0, USER_1, USER_2. USER_3 is system-reserved and not available to homebrew without a kernel plugin. This is the fact the feedback-sender comment relies on.
• On the Vita a lower priority number means higher priority.
• The Vita is hard-locked to 540p: 720p and 1080p are forced down (vita/src/config_values.c:18-29).
• PS5 congestion response is a downward-only ratchet on any reported loss (confirmed by chiaki-ng's loss clamp and our PR fix(congestion): report post-FEC effective loss to stop PS5 bitrate ratchet (GH #206) #213 / GH Wi-Fi burst jitter + receive-queue overflow drive PS5 bitrate floor-throttling (post-GH #188 track) #206 evidence: 0.8-5% reported loss was enough to step target_bitrate down 5825k -> 2591k over 75 s). Do not over-report loss; reporting is post-FEC since PR fix(congestion): report post-FEC effective loss to stop PS5 bitrate ratchet (GH #206) #213.
• There is no mid-session bitrate renegotiation in the protocol (chiaki-ng issues #516 and #519). BitratePayload bitrate_change in takion.proto is a dev-overlay field no Remote Play client has got the PS5 to honour.
• Sony does not run a TURN relay for Remote Play. The "DERIVED" candidate in holepunch.c is a NAT-rebound UDP probe, not a relay.
• The Vita exposes SCE_NET_IP_TOS and SO_SNDBUF, but no SO_PRIORITY.
• Stale as written: the plan said the IDR constants were 100/1000 at videoreceiver.c:18-19 (now :49-50), that sceAvcdecDecode ran on the Takion receive thread (not since PR feat(video): decouple sceAvcdecDecode from Takion recv thread (GH #188) #199), and that the jitter ceiling was 100 ms (now 40 ms, lib/src/takion.c:85, PR fix(takion): break reorder-queue head-of-line deadlock on overflow #195).

Why the feedback sender is at priority 65 (the part a code comment points here for)

Thread layout on main, lower number = higher priority:

Thread	Priority	Core	Where
Takion receive	64	USER_0	lib/src/takion.c:1277-1278
Video decode (VitaDecode)	64	USER_1	vita/src/video.c:1330-1331
Audio	64	USER_2	vita/src/audio.c:207-208
Feedback sender	65	unrestricted	lib/src/feedbacksender.c:31, 413
Input sampling	96	unrestricted	vita/include/host_input.h:11, vita/src/host_input.c:261-262

The original bug (PR #242, merged 2026-08-11). chiaki_thread_create() creates every library thread at the default Vita priority 0x10000100, about 160 (lib/src/thread.c:71-72), and nothing promoted the feedback sender. The thread that puts every button press and stick movement on the wire was therefore the lowest-priority thread in the entire real-time input path, squeezed hardest exactly when decode and render load peak. It fit every symptom: input felt delayed but everything registered, sticks and buttons were affected equally (one shared thread), and video and audio were fine.

Why 65 and not 64. Receive, decode and audio are each pinned to 64 on a different dedicated user core, so they never compete with each other. The Vita has only three usable user cores, and all three are already claimed, one each. A fourth thread also at 64 with no core of its own would timeslice against whichever of those three owns the core it lands on. The feedback sender wakes 80-170 times a second to do GKCrypt keystream generation plus a sendto() that can block inside the Vita network stack, which is a real risk for decode and audio scheduling. Priority 65 still achieves the goal (far above input sampling's 96, vastly above the default ~160 that caused the bug) while guaranteeing it never ties with the three AV threads. For the same reason it gets no CPU affinity mask: there is no spare user core to dedicate, so it stays unrestricted, as input sampling already does.

Ranked ideas and their status

Sizes: S under a day, M 2-5 days, L 1-2 weeks.

Tier 1: high return, low risk

1. Split CPU affinity across the three user cores (S). Shipped, with a different layout. The plan proposed receive on USER_1 and audio on USER_2; what exists is receive on USER_0, decode on USER_1, audio on USER_2, each at priority 64 (table above). The receive thread's priority 64 and USER_0 pin came with PR feat(video): decouple sceAvcdecDecode from Takion recv thread (GH #188) #199 (commit 135e8fb); an earlier receive-thread affinity fix was in PR fix(takion): break reorder-queue head-of-line deadlock on overflow #195.
2. Symmetric SO_SNDBUF and a read-back log (S). Shipped, PR fix(net): SO_SNDBUF symmetric tuning + Senkusha 1ms RTT fallback fix #148 (lib/src/takion.c:384-402, 505-518).
3. Lower the default bitrate on the PSN path (S). Shipped, PR fix(psn): lower default bitrate to 3.5 Mbps on PSN/internet path #149: start bitrate is capped at 3500 kbps on the internet path (PSN_REMOTE_BITRATE_CAP_KBPS, vita/include/host_constants.h:5, applied at vita/src/host.c:201-208). The plan's opt-out config option was not built; it is a fixed constant.
4. Fix the 1 ms Senkusha RTT fallback (S). Shipped, PR fix(net): SO_SNDBUF symmetric tuning + Senkusha 1ms RTT fallback fix #148: the fallback is now 5 ms (lib/src/session.c:723). The proposed 30 ms value for PSN paths is still a TODO in the code comment.
5. Throttle the PSN WebSocket keepalive while streaming (S). Partly shipped, PR fix(psn): throttle WebSocket ping interval + reduce NAT socket burst #150: the ping interval went from 5 s to 10 s (lib/src/remote/holepunch.c:74), but it is a fixed value, not the proposed 30 s after the data connection is established.
6. Fewer sockets in the PSN NAT burst (S). Shipped, PR fix(psn): throttle WebSocket ping interval + reduce NAT socket burst #150: 24 on the Vita (holepunch.c:95), original 250 elsewhere.

Tier 2: high return, moderate effort

7. Port chiaki-ng's AV reorder queue with timeout flush (M). Not done as proposed. There are no per-stream AV reorder queues; video and audio bypass the reorder queue, which carries control messages only (256 entries, takion.c:69). Head-of-line blocking was instead addressed by the adaptive jitter buffer and PR fix(takion): break reorder-queue head-of-line deadlock on overflow #195 (reorder-queue deadlock fix).
8. Audio jitter buffer and packet-loss concealment (M). Not done. lib/src/audioreceiver.c has no jitter buffer.
9. Diagnostics overlay with NAT type, jitter, RSSI, PS5 target bitrate (M). Not done as an overlay. The on-screen panel still shows only "Latency" and "FPS" (vita/src/video_overlay.c:162-190). Treat its "Latency" value with caution: it is the handshake RTT plus the control-channel EWMA, not a live round trip (see the #251 IDR-wait freeze thread). The metrics went to the log instead (PIPE/FPS carries rssi=, vita/src/host_metrics.c:759).
10. Slab allocator for packet buffers (M). Shipped in a simpler form, PR fix(takion): eliminate per-packet malloc churn and raise drain cap to cut recv-thread lag (#192) #192: the receive thread reuses one 1500-byte buffer instead of allocating per packet; only packets that must be retained are copied (takion.c:1332-1336).
11. Move hardware decode to its own thread (M). Shipped, PR feat(video): decouple sceAvcdecDecode from Takion recv thread (GH #188) #199 (GH feat(video): add client-side loss/bitrate adaptation (no mid-session renegotiation) #188). A single-producer/single-consumer queue of 4 slots hands compressed frames to the VitaDecode thread; the decode-to-UI handoff via frame_ready_for_display is unchanged. It stops decode from blocking packet draining on the receive thread.
12. Port chiaki-ng's decoder flush plus IDR on data loss (M). Not done, and superseded. There is no decoder-flush call in vita/src. The project went the opposite way for the same freeze: submit damaged frames through the loss instead of flushing (PR fix(video): submit-through-loss — replace the ~1s IDR-wait freeze with motion (#251) #257, see the Video delivery is bursty: frame arrival gaps up to 706ms while the console renders on time — felt as input lag #251 thread).
13. Test IP_TOS = 0xB8 (DSCP EF) (S code, M to validate). Not done. No IP_TOS use in lib/ or vita/. It still needs an over-the-air capture to learn whether the Wi-Fi firmware maps it to a WMM access category.

Tier 3: speculative

14. NEON-accelerated FEC (L). Not done. Jerasure stays pure C.
15. Two-slot decoded-frame ring to remove silent overwrites (M). Not done as proposed. frame_overwrite_count still exists (vita/src/video.c:975). A twin-texture swap was added for a different purpose (the last-good-frame freeze, GH Display: snapshot_last_good_frame() memcpy costs 30.5ms of every 33ms frame #245, video.c:122-132).
16. Configurable STUN port guessing (S). Not done.
17. Replace the force_psn_holepunch global with a stream-scoped argument (S). Not done. It is still a one-shot flag consumed and cleared in host_stream() (vita/src/host.c:219-225).

Explicitly rejected (still true)

• Adaptive client-to-PS5 bitrate negotiation: the protocol does not support it.
• TURN relay fallback for double-symmetric NAT: Sony runs no relay.
• Mid-session path MTU re-probe: chiaki-ng does not either; low return on stable Wi-Fi.
• Async H.264 decoder API: none exists; the decode thread is the real lever.
• Multiple hardware decoder instances: single decode engine, calls would serialise.
• Tear-free and stutter-free vsync together: a hardware constraint.
• CPU above 444 MHz or a fourth core: both need a kernel plugin.
• 5 GHz Wi-Fi or 2x2 802.11n: the Vita radio is 1x1 2.4 GHz only.
• SO_PRIORITY: not exposed by sceNet. IP_TOS is the closest substitute (item 13).
• Pushing past 540p: the panel is 960x544.

Bottlenecks inventory, in short

The plan's bottleneck list (A-P) mapped to the items above: shared CPU 0 (A, item 1), decode on the receive thread (B, item 11), per-packet malloc (C, item 10), single decoded-frame slot (D, item 15), no send-buffer tuning (E, item 2), no QoS marking (F, item 13), no SIMD FEC (G, item 14), PSN using the LAN bitrate (H, item 3), keepalive CPU (I, item 5), the 1 ms Senkusha fallback (J, item 4), no diagnostics surfaced (K, item 9), the fragile global flag (L, item 17), no per-stream AV reorder timeout (M, item 7), no audio jitter buffer (N, item 8), the render-wait stall before swap (O, never measured), and the 48-socket NAT burst (P, item 6; now 24).

Verifying any of these

Build with ./tools/build.sh --env testing, stream for ten minutes on LAN and then over PSN, and compare frame_overwrite_count, decode timing and the PIPE/* log lines before and after. Stress-test at the edge of Wi-Fi range. Use grep -a on the logs.

Source: docs/ai/REMOTE_PLAY_SMOOTHNESS_PLAN.md (deleted from the repo in #281; recoverable from git history at commit 0e9dd13)