Rerouted live-ingest session death relands on the known-bad native path: #168 verdict not remembered, no in-engine revive

[superuser404notfound]

Split out of #189 (deep-window follow-up by @kskchaitanya1993). The #189 holdback fix itself is verified there (loopback -16832 = 0 across a 2.5 min run with the live window deepened to ~22s, comfortably past the ~18s holdback runway). This issue tracks what remains: the session still cycles through the #168 reroute dance roughly every 13s instead of settling into stable play.

Observed (5.18.5-family, Apple TV 4K 3rd gen, tvOS 26.5)

HEVC-in-MPEG-TS live channel, test-bed loops a fixed segment pool and injects a real [DISCONTINUITY] each lap. Over 2.5 min: 8 #168 reroutes, 0 host ladder re-tunes, playback never leaves the engine, it just loops:

#9  load 192.168.x/hevc-hdr/master.m3u8   -> readyToPlay -> playing (native, audio-only)
#9  master advertises video but AVPlayer built no video track after grace
#168 reroute -> #10 load 127.0.0.1/media.m3u8 (loopback)
     live seg-0 ... seg-1 [DISCONTINUITY] ... seg-2 ... seg-3
#11 load 192.168.x/hevc-hdr/master.m3u8   <- origin master again, native bypass
#11 built no video track -> #168 reroute -> #12 loopback -> repeat

Root cause: two engine-side defects composing into a loop

Defect A: the #168 verdict is not remembered. The video-carriage watchdog (RemoteHLSIngestFallback.Watchdog) discovers "this master advertises video but AVPlayer will never build a video track for its MPEG-TS carriage" at runtime, at the cost of a full native mount, readyToPlay, and the 8-tick (~4s) grace. The verdict is then thrown away. Every subsequent load() of the same URL with nativeRemoteHLS re-mounts the doomed native bypass and re-pays the whole discovery cost. Nothing in the engine records that this master is a known reroute case.

Defect B: the rerouted ingest session has no in-engine recovery. rerouteRemoteHLSOntoLiveIngest loads the loopback session as .custom(HLSLiveIngestReader), so HLSVideoEngine gets sourceReopenableByURL: false (AetherEngine+Loading.swift, session construction). That flag routes every live pump exit (eof, readError, keyframeStarvation) straight past the URL-reopen machinery into onLiveSourceReset ("URL reopen not possible, requesting host retune", HLSVideoEngine+LiveReopen.swift). The ingest reader itself is also designed to go terminal rather than recover: 6 consecutive empty playlist refreshes throw HLSIngestError.ingestStalled (HLSLiveIngestReader.swift), a contract written for phase-1 direct-live where the host had chosen the path itself and owns a fallback ladder. For a #168-rerouted session that contract is wrong twice over: the reader was created by the engine from a URL plus headers and is trivially reconstructible in-engine, and the host never opted into the loopback path in the first place.

How the loop closes. The engine publishes liveSourceReset; a host that answers a live-source reset by re-tuning the same channel URL (a perfectly reasonable response) lands back on the native bypass, which is deterministically doomed for this master (Defect A), which reroutes again (#168), which builds a fresh ingest session that dies the same way (Defect B). Note the #N load <origin> mounts are NativeAVPlayerHost session logs executing that host retune; no engine path reloads the origin URL on its own. So every layer behaves as designed and the composition still loops, with roughly 5s of black/audio-only per lap burned on re-discovering a known verdict.

Why this matters beyond the test-bed

The looped fixed pool is what triggers the ingest death here, but real origins hit the same terminal funnel: ad-boundary and program-transition discontinuities, encoder restarts, and any CDN hiccup that outlives the reader's ~12s refresh-retry deadline. On a rerouted session, each of those today costs a full native-remount dance instead of an in-session recovery.

Proposed fix (two layers)

1. Reroute verdict memory. Cache master URLs whose video-carriage watchdog fired (bounded, process-scoped, expiring so an origin that fixes its packaging is not permanently exiled). load() with nativeRemoteHLS consults the cache and routes a known reroute case directly onto the live-ingest loopback path, skipping the doomed mount and the ~4s grace entirely. This alone collapses the cycle cost and makes any host retune reland on the working path immediately.
2. Bounded in-engine revive for engine-created ingest sessions. When the custom reader is the engine's own HLSLiveIngestReader, an eof/readError/terminal-stall pump exit should first attempt a bounded fresh-reader revive inside the session (rebuild the reader from URL plus headers, rejoin at the live edge, seam carries EXT-X-DISCONTINUITY), mirroring the URL-reopen machinery and the AE#169 bounded-revive pattern. Only an exhausted budget delegates to liveSourceReset.

Evidence wanted to pin the exact terminal trigger

@kskchaitanya1993 offered the raw log around a few cycles in #189, and this is where it pays off: the decisive window is between the loopback mount (#10 load 127.0.0.1/...) and the next origin mount (#11 load 192.168.x/...). The lines that discriminate the candidates:

• [HLSIngest] terminal: ... (which HLSIngestError, e.g. ingestStalled would implicate the pool wrap resetting MEDIA-SEQUENCE and starving the playlist tracker)
• [HLSVideoEngine] live custom-source pump exited (reason=...); URL reopen not possible, requesting host retune
• [HLSVideoEngine] live source replayed from start after reconnect; requesting host retune
• [AetherEngine] onLiveSourceReset -> publishing liveSourceReset to host

Also useful: confirmation of how the host app answers liveSourceReset (re-tune same URL?), which would complete the loop mechanism end to end. The fix targets the whole class either way; the exact trigger determines which revive path Layer 2 must cover first and what the regression test encodes.

[kskchaitanya1993]

Raw evidence from the deep-window run (5.18.5-family, ATV 4K 3rd gen, tvOS 26.5), and a clear answer to the host-retune question — which I think tightens the diagnosis.

One cycle (origin host redacted; ~13s period, repeated 8×):

12:48:48  [NativeAVPlayerHost] #152 master advertises video but AVPlayer built no video track after grace
12:48:48  [AetherEngine] #168: rerouting onto the live-ingest loopback path (TS -> fMP4 remux)
12:48:48  [NativeAVPlayerHost] #153 load url=http://127.0.0.1:50962/master.m3u8   headers=none   (loopback)
   … loopback serves ~10s: seg-0 … seg-1 [DISCONTINUITY] … seg-2 … seg-3 …
12:48:58  [NativeAVPlayerHost] #154 load url=http://ORIGIN/live/hevc/master.m3u8  headers=1      (origin, native bypass)
12:48:58  [HLSSegmentProducer] pump finished: reason=stopRequested packetsRead=1091 packetsWritten=379 lastError=0 elapsed=9557ms
12:49:01  [NativeAVPlayerHost] #154 master advertises video but AVPlayer built no video track after grace
12:49:01  [AetherEngine] #168: rerouting onto the live-ingest loopback path
12:49:01  [NativeAVPlayerHost] #155 load url=http://127.0.0.1:50965/master.m3u8   headers=none   (loopback)

On your "how does the host answer liveSourceReset (re-tune same URL)?" — it doesn't, and I verified this two ways:

1. The app has no liveSourceReset handler at all — grepping the whole tvOS app for liveSourceReset/onLiveSourceReset/onSourceReset/onReopen/reset/retune handlers returns nothing. We subscribe to the engine's phase/duration/format/timeControl publishers only; there is no reset/reopen sink.
2. Across the whole 2.5-min cycle, every host-side retune path logged zero: AETHER post-play error = 0, AETHER stall/reconnect = 0, reloadFromScratch = 0, ladder: rung = 0, host-ingest = 0. Our only same-URL reload (reloadFromScratch) never fired.

So the #154/#155 origin remounts are the engine's own NativeAVPlayerHost re-mounting the source with the engine's stored header (headers=1), with no host action in between. From this host's vantage the reload-to-origin is engine-internal — which I think strengthens the case for your Layer-2 (bounded in-engine revive): there's no host retune to suppress here, so only in-session recovery can break the loop.

On the exact terminal trigger: the discriminating lines you listed — [HLSVideoEngine] … pump exited (reason=…), … replayed from start after reconnect, [AetherEngine] onLiveSourceReset → publishing liveSourceReset, [HLSIngest] terminal: … — are not present in what I capture. My ring only gets the engine's av_log-routed component lines ([NativeAVPlayerHost], [HLSVideoEngine], [AetherEngine], [HLSSegmentProducer]); those reset/pump-exit lines appear to log below that. The one exit reason I do see is the loopback HLSSegmentProducer finishing reason=stopRequested — i.e. the producer is torn down by the remount, so it reads as a consequence, not the trigger. If you want the exact pump exited/onLiveSourceReset reasons I can raise my capture level (or grab an on-device sysdiagnose os_log slice) and post the window around 2-3 cycles — just say which.

Test-bed caveat, to be straight: the per-lap [DISCONTINUITY] is a looped-pool artifact, but per your trace the underlying defects (verdict-not-remembered + no in-engine revive on any pump exit) would bite a real source on any eof/readError blip too, so it isn't purely synthetic. VLC plays this same HEVC-in-TS channel fine, for what it's worth.

---

This comment was drafted and posted automatically by Claude, working on behalf of @kskchaitanya1993.

[superuser404notfound]

Fixed in 5.20.0 (PR #200), along the exact two-defect line above, plus a third layer at the source once the likely trigger became clear from your segment log:

1. MEDIA-SEQUENCE reset rejoin (new, tracker level). Your looped pool almost certainly resets the playlist's MEDIA-SEQUENCE each lap. The ingest tracker used to starve on that (whole window behind its cursor, empty batch every refresh) until the reader went terminal with ingestStalled after 6 stalls, which is what tore the session down. Three consecutive whole-window regressions now rejoin at the new edge under an EXT-X-DISCONTINUITY seam and playback just continues. Isolated stale-CDN-edge windows do not trip it.
2. Reroute verdict memory. Masters whose carriage watchdog fired are remembered (bounded, 6 h expiry, exact URL). Any later load() of the same URL routes directly onto the live-ingest loopback: no native remount, no ~4 s grace, no origin-side -16832 window. A host retune after an ingest death relands on the working path immediately.
3. In-session reopen for engine-created ingest readers. If the ingest still dies for another reason (eof from an upstream ENDLIST, readError from a CDN gap past the retry budget), the pump exit now rebuilds a fresh reader through the existing bounded live-reopen machinery (continuation indices, discontinuity seam) instead of requesting a host retune. Only an exhausted budget surfaces liveSourceReset, as the last resort it always should have been.

On the "purely engine-side" puzzle from your trace: no engine path reloads the origin on its own, so the #11 load <origin> mounts were the engine executing a retune issued host-side, most plausibly a handler answering the engine's liveSourceReset request by re-tuning the same URL (a perfectly reasonable response, and not the bug). With 5.20.0 that request only fires after the in-engine budget is exhausted, and even then it relands on the loopback via layer 2.

Could you rerun the deterministic HEVC-in-TS channel (deep window, same 2.5 min horizon) on 5.20.0? Expected signals:

• Exactly one #168 reroute at the first open, then no further origin master mounts for the rest of the run.
• At each pool lap, either silence (layer 1 absorbed it) or a [HLSVideoEngine] live reopen succeeded on attempt N line (layer 3), but no [HLSIngest] terminal: and no new load mounts.
• On a fresh open of the same channel within 6 h: [AetherEngine] #199: master is a remembered no-video-track carriage case; routing directly onto the live-ingest loopback path, and consequently no pre-reroute origin -16832 grace at all.

If anything in that run still cycles, the raw log around one cycle is exactly the right artifact, same offer as before. Verification basis on this side: new Issue199RerouteRecoveryTests plus tracker/halt suite extensions (all watched failing first), full suite green (965 tests), strict-concurrency and tvOS Simulator builds clean, CI green on PR #200.