Make AVIOReader.connStallTimeout configurable — 20 s is a long time to hold a frozen frame when the origin is on the same LAN

[rrgomes]

AVIOReader.connStallTimeout is a private static let fixed at 20 s. The comment
calls it a CDN stall threshold, and for a CDN that seems right: a distant origin
can legitimately go quiet for a while, and tearing down a connection that was
about to deliver is worse than waiting.

On a LAN origin the trade runs the other way, and I would like to be able to say
so per-load rather than take the 20 s.

What it looks like from here

A 95-minute session against a Plex server on the same wired LAN produced nine
reads that each sat almost exactly at the timeout:

slow read: 20075ms  stallWaits=10(20072ms,9signaled)  reconnects=1
slow read: 20040ms  stallWaits=9(20037ms,9signaled)   reconnects=1
slow read: 20133ms  stallWaits=10(20130ms,9signaled)  reconnects=1
slow read: 20092ms  stallWaits=7(20089ms,6signaled)   reconnects=1
...

Every one is the same shape: no bytes at all, the full timeout elapses, one
reconnect, and then data immediately:

22:29:04.965  pump stall at offset 65673024, reconnecting
22:29:04.969  pump conn start gen=9 offset=65673024 len=26MB
22:29:05.037  pump gen=9 first data after 70ms
22:29:05.053  slow read: 20092ms ... reconnects=1 connect=2ms

The reconnect costs 70 ms and resumes at full rate. So the connection was dead
for essentially the whole 20 s and the only thing that established that was the
timeout expiring. A shorter threshold would have recovered in a fraction of the
time, and the evidence that it would have succeeded is that it did succeed, at
full rate, the moment it was tried.

To be clear about whose problem the underlying fault is: mine. Something on
my LAN is dropping TCP connections without an RST — the same minutes also kill an
unrelated WebSocket to a different service on the same host, so it is not
AetherEngine and I am not asking you to fix it. What I am asking is that the
recovery time not be fixed at a value chosen for a very different network.

I have seen the same shape in an earlier session on a different box (nine
reconnect-and-stall pairs interleaved the same way), so this is not a one-off
capture.

Why 20 s specifically hurts here

Two things make the wait more expensive than a plain 20 s gap in delivery would
suggest.

The content is AV1 and decodes in software (codec=225 → software,
libdav1d, no Apple TV has an AV1 hardware decoder). After the reconnect the
decoder picks up mid-temporal-unit:

[error] Missing reference frame needed for show_existing_frame (frame_to_show_map_idx = 3)
[error] Failed to read unit 0 (type 3): Invalid data found when processing input
[error] Failed to parse temporal unit

so the recovery is not instant either — it costs a further resync on top of the
20 s.

This is a synchronised-playback client. Two people watch the same title in
step over a relay. A 20 s hold is not just a long buffer on one screen; it is 20
seconds of one viewer sitting on a frozen frame while the room's clock keeps
moving, which then has to be reconciled. Something in the 3–5 s range would fit
that context far better, and I would happily take a few more reconnects for it.

The ask

Expose it on LoadOptions, alongside the existing probesize /
maxAnalyzeDuration optionals, defaulting to today's 20 s so nothing changes for
anyone who does not set it:

/// No bytes for this long triggers a reconnect. nil keeps the default,
/// chosen for CDN origins that can legitimately go quiet.
public var connStallTimeout: TimeInterval?

A host that knows its origin is one hop away can then choose a threshold that
matches. I have no view on whether the default should move — you have far more
visibility into the CDN case than I do, and the argument for 20 s there looks
sound to me.

Happy to test whatever you land on; I can reproduce the stall class on demand at
the moment, which is not a sentence I expected to be glad about.

[rrgomes]

Withdrawing the diagnosis in this report. The stalls were not the network — they
were my own client saturating the CPU, and I should not have filed this before
I had a control.

I got a control the next evening by accident. The same box, the same wired LAN,
one session: two hours of 4K HEVC at 24,334 and 24,928 kbps, hardware-decoded,
with zero reader stalls and zero socket drops — then 33 minutes of 791 kbps
AV1, and 8 dropped WebSocket connections, 6 The network connection was lost,
and 3 more of the 20 s reads I described above. The stream that "broke the
network" was running at 3% of the bitrate of the two that did not.

The one thing that changed was dispatch: codec=225 → software. Everything else
in that session was → native.

What was actually happening: my router was sending 1080p60 AV1 to software decode
on a budget I had only ever validated at 1080p24 — it compared heights and had no
frame-rate term at all. dav1d then held exactly real time, 60.0–60.4 fps
sustained with no headroom, so the decode looked fine and the damage landed
somewhere else entirely. A saturated CPU stops servicing URLSession, so bytes stop
being delivered to the reader and connStallTimeout fires against a socket that
was never dead. The tell I missed at the time is in my own numbers: sixteen
WebSocket handshakes that timed out at 10 s having never opened. That is not
something a healthy LAN does to a healthy app.

Within the AV1 material the fault rate tracks frame rate — 0.077/min at 24p
against 0.191/min at 60p, a 2.5× ratio matching the pixel-rate ratio exactly.
Fixed on my side by budgeting on width × height × fps.

This also inverts the request, so please don't act on it as written. Under CPU
starvation a shorter connStallTimeout is actively worse: it tears down and
redials connections more often precisely when the app has no cycles to spare, and
each redial is work the decoder then has to compete with. The 20 s default was
doing the right thing here — it was absorbing my problem, not causing it.

There may still be a case for making it configurable for a genuine LAN origin, but
I no longer have evidence for it, and I would rather withdraw the ask than have you
build something on a measurement I have just retracted. Happy for you to close
this. If I hit a real LAN-origin case with the CPU headroom to prove it, I will
open a fresh one with the control in it from the start.

Sorry for the noise.

[superuser404notfound]

Thank you for the retraction, and more so for going back for a control before letting the
report stand. A withdrawn issue with a measurement in it is worth more to me than most
confirmed ones, and the shape of the control is the useful part: same box, same wired LAN,
one session, 4K HEVC at 24 Mbit/s hardware-decoded with zero faults, then 791 kbit/s AV1
in software with eight dropped sockets. The stream that "broke the network" running at 3%
of the bitrate of the two that did not is about as clean a disconfirmation as a field
capture gets.

Taking the ask off the table as requested. No LoadOptions.connStallTimeout, and the
default is not moving either.

I did chase one thread before closing, because your retraction pointed at something that
would have been ours if it held. The reasoning was: the software decoder takes
activeProcessorCount threads (SoftwareVideoDecoder.swift:124, FF_THREAD_FRAME | FF_THREAD_SLICE), the software demux and feed queues sit at .userInitiated, and every
byte the reader receives arrives on a URLSession delegate queue we never configure
(delegateQueue: nil at all four AVIOReader sites). If that implicit queue ran below the
decode workers, a saturated box would starve byte delivery, connStallTimeout would fire
against a live socket, and the reconnect would add work at the worst moment. That would
have made the engine an amplifier of your CPU problem rather than a bystander.

It does not hold, and it fails in both places I looked.

Delegate queue QoS, paired A/B. Four URLSession arms downloading concurrently from one
paced local origin, so a starved origin hits all four equally and only the delegate queue
differs. Load was four concurrent dav1d decodes of 1080p60 AV1, box pinned at 765% of 800%,
load average 16.9.

delegate queue	p95 gap	max gap	gaps > 100 ms
implicit (nil)	7.1 ms	27 ms	0
.userInitiated	7.1 ms	27 ms	0
.userInteractive	7.1 ms	27 ms	0
.utility	16.2 ms	78 ms	0

Throughput was identical in all four arms, 60.1 MB per 5 s window, saturated or not. The
.utility row is the positive control: the harness does see QoS, that arm loses latency and
callbacks exactly as it should. Which is what makes the null result in the other three
meaningful rather than merely absent. URLSession's implicit queue is already above the
decode workers, so there is no headroom to buy.

The whole reader, end to end. aetherctl play --sw against a range origin on loopback
shaped to 40 MB/s, 1080p60 AV1 at 133 Mbit/s so dav1d is genuinely working, three 100 s
sessions:

arm	CPU	stalls	reconnects	slow reads	clock after 100 s
idle box	61%	0	0	0	102.51 s
4 external decoders	780-790%	0	0	0	102.50 s
8 external decoders	787%	0	0	0	103.00 s

Not one stall in 300 seconds of playback with the machine pinned, including at 2x
oversubscription.

What starvation does cost is visible in the same runs, and it is the decoder, not the link.
The idle session passed the 5000-frame enqueue milestone; neither loaded session got past
1000 in the same wall time, while the presentation clock advanced normally in all three. The
reader followed the decoder down rather than fighting it: 83 range requests on the idle box
against 47 under load, same 100 s. So when the box runs out of cycles the engine drops
frames and asks the link for less. It does not press a connection it cannot drain and then
mistake the result for a dead socket.

Two limits worth stating, since both cut toward your case rather than mine. This is an M1
with 8 cores, not a 6-core Apple TV, and tvOS schedules under memory and thermal pressure
that macOS does not apply the same way. And the origin is on loopback, so a connection that
dies without an RST is out of scope by construction. That last one matters, because it is
the part of your original report that CPU load does not explain: an unrelated WebSocket to a
different service on the same host died in the same minutes. Sixteen handshakes that timed
out at 10 s having never opened is consistent with your starvation reading, but connections
being dropped silently is a separate fault, and it was there before the AV1 sessions were.

Closing as withdrawn. If you do hit a LAN-origin case with the CPU headroom to prove it, open
a fresh one and I will look again. The budget fix on your side (width x height x fps rather
than height alone) is the right shape.