Tried and reverted: lessons from reverted work (dead-stream watchdog, PR #196)

[mauricio-gg]

Summary. In June 2026 we tried to detect a dead stream by watching decoded-frame timing in the app, and tear the session down when frames stopped. It misfired on a healthy stream, caused a ~14 second self-inflicted outage, and was reverted (PR #196, closed without merge on 2026-06-26). The "forward direction" it named, detecting socket death in the network layer, is what we built instead. It shipped as PR #209, PR #244 and PR #274, and the app-level watchdog was never revived.

Epic: Dead-Stream Watchdog (PR #196), reverted, 2026-06-26

Original goal. Detect that a streaming session has died on the Vita through socket-level monitoring, and trigger a graceful teardown before the PS5 releases the old session.

Why it was reverted. A critical regression showed up during hardware A/B testing (log 20639381559, build commit 6989b93).

• Root cause 1 (critical): a torn 64-bit read on ARMv7. last_decoded_frame_us (a uint64_t) was written by the decode thread and read by the UI thread with no lock or atomic on arm-vita-eabi (32-bit ARM). The UI thread hit a torn read: the high and low 32-bit halves came from different writes, so the timestamp looked like it was in the future. The unsigned subtraction underflowed and computed a stall of 18446744073709550 ms while the stream was healthy at 30 fps.
• Root cause 2 (secondary): RP_IN_USE reconnect rejection. The spurious teardown triggered an immediate reconnect. The PS5 rejected it (Remote is already in use, code=4) because it had not released the old session yet. Result: a self-inflicted ~14 second outage on a healthy stream.

Why this approach failed.

• App-level teardown based on frame timing is inherently racy when the threads are asymmetric (decode writes, UI reads).
• A 32-bit ARM platform cannot safely do 64-bit non-atomic reads across threads.
• The PS5 releases a session asynchronously. A reconnect attempted before the release fails with RP_IN_USE.

Alternative approach named at the time ("forward direction"). Detect suspend/resume and socket death in the library's transport layer: escalate ENOBUFS/EBADF in the Takion send/receive path, and handle the DISCONNECT-during-streaming path in streamconnection.c, then drive a clean teardown from there. This moves detection to where socket state is already observable, without comparing timestamps across threads.

Planned tasks that were never completed. App-level frame-stall watchdog, PS5 session reconnect logic, UI messaging for detected freezes.

Why PR #196 existed at all. The hardware log behind it showed the real user problem: press the PS button mid-stream and the stream froze for good. The PS-button suspend invalidated the Vita UDP socket (ENOBUFS, then EBADF), and every existing recovery trigger needed incoming_fps > 0, so a fully frozen stream was invisible to recovery. That problem is real. The watchdog was the wrong detector for it.

What happened to the forward direction

Checked against main (commit 0e9dd13).

• PR fix(takion): survive transient ENOBUFS and propagate mid-stream transport death (GH #208) #209 (merged 2026-08-10, fixes Takion socket death on transient ENOBUFS: recv thread exits, mid-stream DISCONNECT ignored, session freezes until force-close #208). A transient ENOBUFS on recv() no longer kills the Takion receive thread. takion_recv retries with a short backoff and gives up only after many consecutive failures (lib/src/takion.c:140-142, TAKION_RECV_TRANSIENT_*). A mid-stream transport DISCONNECT is no longer silently ignored: it is now propagated as transport_failed (lib/src/streamconnection.c:541-556, log line "Takion transport disconnected mid-stream"). The session then ends with CHIAKI_QUIT_REASON_STREAM_CONNECTION_TRANSPORT_FAILED (lib/src/session.c:169).
• PR fix(takion): make the ENOBUFS backoff actually elapse #244 (merged). The retry budget in fix(takion): survive transient ENOBUFS and propagate mid-stream transport death (GH #208) #209 was meant to be about 2 seconds but was burned in 30-40 ms. The backoff was a select() on the stop pipe, and on the Vita the stop pipe is itself a UDP socket, so during an ENOBUFS storm the "sleep" failed immediately. The fix guarantees the delay actually elapses (takion_recv_transient_backoff, lib/src/takion.c:1606) and adds a minimum persistence time (TAKION_RECV_TRANSIENT_MIN_PERSIST_MS = 1500, lib/src/takion.c:142). The trigger was reproducible: press the physical PS button and resume. PR fix(takion): make the ENOBUFS backoff actually elapse #244 identified this as the cause of the mid-stream session kicks seen in hardware log 55384582356.
• PR fix(session): hands-off recovery after transport death (#272) #274 (merged, fixes bug(session): hard-fallback recovery after transport death drops the user to the main menu instead of reconnecting #272). After the transport dies, the app now reconnects by itself behind the Reconnecting overlay. The library no longer runs its own soft-restart ladder on transport death; it ends the session with TRANSPORT_FAILED (lib/src/session.c:795-801). The decision is made before teardown in vita/src/host_quit.c:102-108. The fallback reconnect then runs from the UI thread, only after the old session has been joined and finalized (vita/src/ui.c:545-549). PR fix(session): poll for console release instead of a fixed 9 s wait during recovery (#277) #279 later replaced a fixed 9 second wait before that reconnect with polling for the console to release the old session, which addresses root cause 2 above (RP_IN_USE).

The app-level frame-stall watchdog is gone: no last_decoded_frame_us or watchdog code exists in lib/ or vita/ on main.

Lessons worth keeping

1. Never share a 64-bit value between threads on the Vita without an atomic or a lock. A 32-bit read can tear.
2. Do not tear a session down from a timing heuristic. Tear it down from a fact the transport reports (socket error, DISCONNECT).
3. After a session ends, the PS5 needs time to release it. Reconnecting too early gets RP_IN_USE and turns a small glitch into a long outage.

Source: DEPRECATED.md (deleted from the repo in #281; recoverable from git history at commit 0e9dd13)