fix: dedicate a non-blocking fd for terminal input to stop TTY-reader event-loop freezes

[bkrabach]

Summary

The CLI could freeze completely — no output, no timers, for hours — until the user pressed a key. Root cause: prompt_toolkit's PosixStdinReader.read() does a non-atomic select-then-read on a blocking fd 0:

if not select.select([self.stdin_fd], [], [], 0)[0]:   # readiness check
    return ""
...
data = os.read(self.stdin_fd, count)                   # BLOCKS

fd 0 has no O_NONBLOCK and the tty is raw with VMIN=1, so if any other process consumes the pending byte between those two lines, os.read parks forever. It is registered via loop.add_reader (input/vt100.py:169), so it runs on the asyncio event-loop thread — run_forever cannot advance and no timer fires.

This gives prompt_toolkit its own dedicated /dev/tty fd opened O_RDONLY|O_NONBLOCK. A fresh os.open() creates a new open file description, so the flag is private to us and does not leak to fd 0, the parent shell, or inherited children. PosixStdinReader already swallows OSError (BlockingIOError is a subclass), so a would-block read degrades to an empty read instead of hanging.

Evidence

Production forensics across 19,028 real session logs: 21 distinct freeze events between 2026-07-06 and 2026-08-02. Every one shows zero events across every session in the project during the gap — total process death. Worst case: a bash call with timeout: 35 returned "Command timed out after 35 seconds" 4h 04m later.

Loop-thread stack captured mid-stall:

prompt_toolkit/input/posix_utils.py:87 in read      <-- os.read BLOCKED
prompt_toolkit/input/vt100.py:98 in read_keys
prompt_toolkit/application/application.py:686 in read_from_input
asyncio/events.py:89 in _run
asyncio/base_events.py:2050 in _run_once
asyncio/base_events.py:683 in run_forever

Competing readers observed in real sessions: ssh without -n, amplifier-digital-twin exec (incus exec), and nested amplifier run / amplifier tool invoke. 16 of the 21 events contain one.

Controlled A/B, n=20 per arm, real pty + live PromptSession + heartbeat watchdog:

Arm	max stall	runs stalling >2s
baseline	40.73s	7 / 20
with this fix	2.00s	0 / 20

Baseline stalls included 14.2s, 24.7s, 24.7s, 29.2s, 29.2s, 33.2s, 40.7s. With the fix the worst observed stall is 2.00s.

Note this is distinct from #231 (stdout_offload.py), which fixed the analogous freeze on the write side. Freezes continued after #231 landed on 2026-07-12. Same family — an unbounded syscall on the loop thread — opposite direction.

Test Plan

• tests/test_dedicated_tty_input.py — real pty attached to fd 0; asserts the dedicated fd is distinct and non-blocking while fd 0's OFD is not (the regression guard against the flag-leak trap), and that the race degrades to an empty read. The race is forced deterministically via a fake select, so the test does not depend on winning a ~38% probabilistic race.
• tests/test_interactive_chat_tty_teardown.py — fd closed on interactive_chat teardown; close is idempotent and cannot mask an original error from a finally block.
• Full suite: 1252 passed, 14 failed. The 14 were verified pre-existing by checking out origin/main and re-running the same 5 files: 14 failed / 42 passed on both. Zero regressions.
• Graceful fallback verified for Windows, non-tty stdin, and unopenable /dev/tty (CI/containers) — returns None and prior behavior applies.
• python_check parity with baseline; new files are fully clean.

Reviews

Spec review found the fd teardown was wired into execute_single() (which never opens the fd) but missing from interactive_chat() (which does) — fixed in 531973c. Code-quality review: APPROVED, zero critical or important issues.

In-situ verification (real CLI, real project directory)

Verified against the real amplifier CLI in the reporter's actual project directory using py-spy dump at 10 Hz. A sample counts as frozen only when the loop thread's TOP frame is the blocking read:

read                       (prompt_toolkit/input/posix_utils.py:87)   <-- BLOCKED in os.read
read_keys                  (prompt_toolkit/input/vt100.py:98)
read_from_input            (application.py:686)
callback_wrapper           (vt100.py:166)                             <-- add_reader callback
_run_once                  (asyncio/base_events.py:2050)
run_forever                (asyncio/base_events.py:683)

Idle-in-epoll is scored separately and never counted as a freeze.

Arm	Trials	Samples	Frozen samples	Trials that froze	Frozen wall-time
baseline (no fix)	12	3600	395	12 / 12	39.5s
with this fix	12	3600	0	0 / 12	0s

Fisher exact on 12/12 vs 0/12 ~= 3.7e-7. Arms interleaved (a,b,a,b...).

Instrument validity. Two earlier attempts were discarded as invalid and are not counted above: a pyte-based PTY that never answers CPR (CPR replies to ESC[6n are the tty input the race is fought over -- a terminal that does not answer them cannot reproduce the bug), and a scripted-Enter approach that produced false "freezes" from unsubmitted prompts. The instrument used here answers CPR (39-41 replies per trial in both arms) and submits nothing -- no LLM turn runs and no keystroke timing is interpreted; the freeze is read directly off the stack.

Controls.

• No competitor -> 0 frozen samples in 24/24 trials, both arms. The competing reader is the cause.
• Contention was real in the fixed arm too: the competitor won 99.6% of tty bytes with the fix vs 90.1% without. Bytes were stolen in both arms; only the unfixed arm wedged.
• Arm identity proven from inside each process: baseline raises ModuleNotFoundError for dedicated_tty_input; fixed arm resolves it and holds an extra /dev/tty fd with O_NONBLOCK set. 12/12 probes each.

Caveats. py-spy pauses the target ~21% of wall time, identically in both arms. Parks under ~100ms can be missed, so the baseline rate is a lower bound and the fixed arm's zero means "no park >=100ms in 3600 samples," not proof of impossibility. The baseline's 1.6s max streak is capped by the harness, not by reality -- production freezes run for hours because a frozen loop never redraws, so it never emits ESC[6n, so no CPR reply ever arrives to release the read.

Process note. This in-situ verification completed AFTER this PR was opened; the PR was opened on controlled-harness evidence alone. Nothing has been merged -- this evidence is on the record before the merge gate.

🤖 Generated with Amplifier

[bkrabach]

Post-merge follow-up, recorded here for anyone who lands on this PR from a bisect.

This change was correct on Linux and totally broke macOS. The mechanism it fixes is real — the non-atomic select-then-read in prompt_toolkit's PosixStdinReader is a genuine event-loop freeze, and the production forensics in the description hold up. But os.open("/dev/tty", O_RDONLY|O_NONBLOCK) is not pollable by kqueue. On macOS the open succeeds and loop.add_reader() then raises OSError(EINVAL) at attach time — inside prompt_toolkit, well past this module's fallback. Every interactive session on macOS died at the first prompt: silent exit right after the banner on prompt_toolkit >= 3.0.53 (which converts that OSError to EOFError), an error loop on <= 3.0.52.

Reproduced on real hardware (macOS 26.6, arm64, Python 3.12.12, prompt_toolkit 3.0.52), driving this module's actual code path under pty.fork():

branch	fd opened	result
this PR (e793151)	/dev/tty	FAIL — OSError 22, REPL error loop every iteration
#250 (7063fde)	/dev/ttys008	OK
#251	/dev/ttys008	OK

Fixed by #250 (thanks @Joi) and generalized in #251.

Three things worth carrying forward, since the code was right and the process is what failed:

1. The fallback validated the wrong property. It checked "can I open this fd?" and never "can the event loop poll it?" This module's own docstring says a broken state should fail loudly rather than proceed degraded — instead it proceeded silently dead. fix: probe tty fd pollability instead of platform-checking; add CI (Linux + macOS) #251 adds a pollability probe so an openable-but-unpollable fd falls back to a working prompt and logs a warning.
2. ## Known gap: The fix has not yet been exercised in a live interactive session was in this PR's body at merge time. That gap is exactly where the bug was. When the body says the thing isn't proven, that's the merge blocker, not a footnote.
3. There was no CI in this repo at all. 875 lines through the interactive input path, admin-merged 39 minutes after opening, with no automated gate on any platform. fix: probe tty fd pollability instead of platform-checking; add CI (Linux + macOS) #251 adds ubuntu + macOS runners.

The diagnosis in this PR was excellent work. The gap was between "proven on the platform I was on" and "shipped to every platform."