feat(progress): surface long-running Bash/ScheduleWakeup waits in chat — silent 5–10 min holds look hung

[nathanschram]

Context

When a Claude session executes a Bash tool call that internally polls or
sleeps for several minutes (e.g. until gh run list … grep -q completed; do sleep 15; done waiting for a CloudFlare Pages deploy, or a sleep 600
waiting for a job, or any npm run build on a slow repo), the Untether
chat shows a single static ▸ Bash line for the entire duration.
From the user's phone there is no visual difference between "still
polling" and "frozen". The same pattern applies to ScheduleWakeup-driven
/loop-style waits and to long-running MCP tool calls.

Concrete reproduction (live, 2026-05-05 staging)

Session 9905b5fe-7e21-4042-8715-941c9206fa52 on @hetz_lba1_bot,
project lba-web, chat -5193338937. The user reported "session frozen
~40 min". Investigation found two real waits:

1. 10 m 04 s Bash polling loop:

   until gh run list --branch main --limit 2 --json status,workflowName \
     --jq '.[] | select(.workflowName == "Deploy Production") | .status' \
     2>/dev/null | grep -q "completed"; do
     sleep 15
     echo "  $(date +%H:%M:%S) Deploy Production: …"
   done
   echo "Deploy Production done."
   

   The loop emitted a useful "still alive" line every 15 s on subprocess
   stdout. Untether saw zero of those — the tool result is buffered until
   the Bash call returns, so the ▸ Bash progress line was static for
   the full 10 minutes.

2. 14 m 00 s waiting on a control_request for an inline approval
   keyboard the user didn't notice (separate UX gap, see follow-up
   comment in this issue thread). Outside scope of this issue.

Plus three CI-poll waits in the same run (3 m + 1.5 m + 2.5 m).

Net effect: the run's Telegram surface looked dead for ~17 minutes of
external-system polling that was actually proceeding normally.

Related issues

• stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470 — stall-message backoff once last_event_type=result. That
  issue solves the post-result-idle silence-then-warn-spam problem;
  this issue is its mid-tool sibling. Together they cover the two
  ways a Claude session can look hung in Telegram. See section
  "Coordination with stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470" below — the stall-suppression work is
  shared.
• feat: full /loop support — agent self-pacing via ScheduleWakeup interception #289 — full /loop support via ScheduleWakeup interception.
  Surfacing the wait time per loop iteration is functionally adjacent;
  if both ship together a 4 m 30 s /loop delay should render as a
  live countdown rather than a static action line.
• enhancement: detect & handle long-running Claude Code tools (Monitor/Bash-bg/ScheduleWakeup) that hold the subprocess past result — UX gap blocks operator confidence #346 / enhancement: surface Claude Code background-task state (Monitor / Bash-bg / Agent-bg / ScheduleWakeup / RemoteTrigger) #347 (closed in v0.35.2) — first round of background-task
  surfacing (Monitor / Bash-bg / ScheduleWakeup state). This issue is
  the next layer: not just that a long tool is running, but that it
  is making progress.

Proposed work — three coordinated changes

1. Long-tool elapsed-time progress timer

When a Bash/ScheduleWakeup/long-MCP tool call exceeds a threshold
(default 60 s), the progress message should update its label to show the
elapsed time and (where available) the tool input snippet:

▸ Bash · 7m 12s · still running
  $ until gh run list --branch main … grep -q "completed"; do sleep 15; …

Implementation:

• New per-action timer in MarkdownFormatter.format_action_line() keyed
  on action_id.
• Driven by an anyio task spawned from the Bash/Schedule action handler
  in runner_bridge.py that re-renders the action line every
  [progress] long_tool_render_interval seconds (default 30 s).
• Cap at one timer per active action; cancel on tool_result.
• Respect [progress] verbosity and /verbose toggle — the elapsed
  string is always shown in verbose=on, hidden in verbose=off unless
  elapsed exceeds a higher threshold (long_tool_visible_threshold,
  default 120 s).

Critical files:

• src/untether/markdown.py — format_action_line, action timer hook
• src/untether/runner_bridge.py — spawn/cancel timer in
  the ActionEvent action_started/action_completed branches
• src/untether/settings.py — [progress] long_tool_render_interval,
  [progress] long_tool_visible_threshold
• tests/test_meta_line.py — fixture that drives a 5-minute synthetic
  Bash tool, asserts ≥ 8 elapsed-string updates and that the timer is
  cancelled on the tool_result event.

2. Surface Bash subprocess stdout as a "last line" on the progress action

Claude Code already streams Bash stdout in its tool_use_id partial
events when run in --verbose --output-format stream-json (which
Untether uses by default). The line above —
$(date +%H:%M:%S) Deploy Production: … — would have been a strong
"still alive" signal. Untether currently discards these partials.

Implementation:

• Extend claude_schema (src/untether/schemas/claude.py) and
  translate_claude_event (src/untether/runners/claude.py) to
  preserve the most recent stdout line per active Bash tool call.

• Surface it on the action line as a sub-line:

  ▸ Bash · 7m 12s · still running
    $ until gh run list … (10 lines suppressed)
    └─ 11:34:12 Deploy Production: in_progress
  

• Throttle: at most one update per 5 s per action (configurable via
  [progress] bash_stdout_render_interval, default 5).

• Bound: trim the rendered line to ≤ 120 chars.

• Strip ANSI colour codes via the existing strip_ansi helper used by
  _sanitise_stderr siblings.

Critical files:

• src/untether/schemas/claude.py — preserve bash_output_chunk partials
• src/untether/runners/claude.py:translate_claude_event — feed
  state.bash_action_stdout[action_id] = line
• src/untether/markdown.py:format_action_line — render the sub-line
• tests/test_claude_schema.py and tests/test_meta_line.py —
  fixture & rendering coverage

3. Coordination with #470 — suppress stall warnings during long-tool waits

Today the stall watchdog
(runners/claude.py:_subprocess_liveness_stall) fires every ~3 min
based on idle_seconds >= stall_threshold AND
tree_active/cpu_active. Once changes 1+2 land, a long Bash poll
or ScheduleWakeup wait is no longer a "stall" — the run is
actively progressing.

Both this issue AND #470 should suppress Telegram-side stall warnings
when the subprocess is in one of these expected-idle states:

State	Suppression rationale
last_event_type == "result" (post-turn idle)	#470's original case — turn is done, stdin is just held open
Active Bash tool with last stdout line within stall_threshold/2	the Bash command is producing output, run is alive
Active ScheduleWakeup with timer fire-time still in the future	the agent explicitly asked to wait — not a stall
Active long-MCP-tool call (already partially handled by mcp_tool_timeout per #154)	already covered, mention here for completeness

The structlog WARNING event should still fire on each stall threshold
crossing for observability (the untether-issue-watcher consumes
those). The change is purely the Telegram surfacing.

New events:

• progress_edits.stall_long_bash_suppressed — when active Bash tool
  has fresh stdout
• progress_edits.stall_schedule_wakeup_suppressed — when active
  ScheduleWakeup timer is still in the future

This overlaps directly with #470's progress_edits.stall_post_result_suppressed
proposal — the three suppression rules should land in the same
progress_edits.stall_detected decision branch in runner_bridge.py
so the logic stays in one place.

Action item for #470: I'll comment on that issue to widen its scope
to include long-tool-wait suppression (this issue) so the changes ship
together rather than as overlapping patches.

Acceptance criteria

• A Bash tool call that runs for ≥ 60 s renders an elapsed-time
  string on its action line, refreshing at least every 30 s.
• If the Bash call produces stdout, the most recent ≤120-char line
  is surfaced as a sub-action under the action line, refreshing
  at least every 5 s while changing.
• A ScheduleWakeup action shows time-until-fire in the action
  line, refreshing at least every 30 s (synergy with feat: full /loop support — agent self-pacing via ScheduleWakeup interception #289 if that
  ships first).
• During an active long-tool wait (Bash with fresh stdout, or
  ScheduleWakeup with future fire time, or post-result stdin idle
  per stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470), Telegram stall warnings are suppressed.
• structlog subprocess.liveness_stall warnings still fire on
  each threshold crossing (observability preserved).
• New progress_edits.stall_*_suppressed info logs for each
  suppression branch.
• No regression on existing fast tools — sub-second tool calls
  render identically (no flicker, no elapsed string).

Test plan

Unit — extend tests/test_exec_bridge.py and tests/test_meta_line.py:

1. Synthetic 5-minute Bash with periodic stdout — assert elapsed
   string updates ≥ 10 times and stdout sub-line refreshes.
2. Synthetic ScheduleWakeup with delaySeconds=180 — assert
   countdown renders.
3. Synthetic long Bash + stall threshold cross — assert structlog
   WARNING fires but Telegram surfacing is suppressed.
4. Synthetic post-result + stall threshold cross — assert stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470's
   suppression still works (regression).
5. Sub-1-s Bash — assert no elapsed string rendered.

Integration — against @untether_dev_bot, claude-test chat:

• Send run a bash command: for i in 1 2 3 4 5 6 7 8 9 10; do echo "tick $i $(date +%H:%M:%S)"; sleep 12; done → observe action line
  count up over 2 min with rolling stdout sub-line; no stall warning.
• Send /loop … (whatever the syntax is by the time this ships) and
  let it self-pace → observe countdown per iteration.

Out of scope

• The 14-min approval-keyboard-wait surfaced in the same lba-web
  session — separate UX issue: chat-side reminder when a
  control_request is pending past the stall threshold. Will file
  separately if there's appetite.
• The CloudFlare Pages production-deploy time itself — that's CDN-side,
  Untether can only render the wait better, not shorten it.

Source

Filed after the v0.35.3 staging dogfood session
9905b5fe-7e21-4042-8715-941c9206fa52 on 2026-05-05, where a 45-min
run felt frozen but was actually 17 min of healthy CI/CDN polling +
9 min of agent work + a separate 14-min approval-keyboard wait.

[nathanschram]

Hard requirements (added per user direction)

These are now non-negotiable for v0.35.3 promote:

Requirement A — visibility MUST be obvious

For every long-running Claude tool that holds the session past a few seconds (ScheduleWakeup, Bash including its Monitor/background variants, mcp__* long calls, and any future tool with the same shape), Telegram MUST make it VERY apparent that work is in progress and what specifically is happening. The static ▸ Bash line is the failure mode we are removing.

Concrete acceptance bar:

• A user glancing at the chat for ≤ 2 seconds can answer all three: (1) is it alive? (2) what is it doing? (3) how long has it been doing it?
• The progress message updates at minimum every 30 seconds for any tool call lasting > 60 s — even if the tool itself produces no output (i.e. the elapsed-time string alone is enough to satisfy "alive").
• For Bash specifically: when stdout is being produced, the most recent output line is rendered as a sub-line under the action, refreshing ≤ every 5 s. The 10-min Deploy Production: in_progress polling loop in the lba-web reproduction would have rendered as a live, updating sub-line — that is the target.
• For ScheduleWakeup: a live countdown to the fire time is shown on the action line (▸ ScheduleWakeup · fires in 4m 12s · "build still running, check in 4.5 min").
• Tool name + key argument are always rendered (truncated to a sensible width, full content available in /verbose). The user must never have to ask "what's it doing right now" of a chat surface that has been silent.
• The contract holds regardless of /verbose toggle — /verbose off may show a more compact form, but it must never collapse to a state where the chat looks frozen.

Requirement B — zero stall warnings during expected waits

Telegram-side stall warnings (subprocess.liveness_stall surfacing ops) MUST fire only when the system is genuinely frozen. They must NOT fire while:

• A ScheduleWakeup timer is pending and the fire-time is still in the future
• A Bash tool is active AND has produced any stdout within the last stall_threshold / 2 seconds (default → 90 s with the standard 180 s stall threshold)
• A Bash tool was started < bash_grace_seconds ago (default 60 s) — gives the command time to start producing output
• A Monitor / background-task handle is live and the tracked process is reporting heartbeats (per enhancement: surface Claude Code background-task state (Monitor / Bash-bg / Agent-bg / ScheduleWakeup / RemoteTrigger) #347's existing tracking)
• A long-MCP-tool call is active within mcp_tool_timeout (already covered by Stall watchdog: longer threshold and contextual messaging for MCP tool calls #154)
• last_event_type == "result" and stdin is just being held open (covered by stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470)

Definition of "genuinely frozen" — all of these together:

1. No JSONL event from Claude Code in ≥ stall_threshold seconds, AND
2. No stdout produced by any active tool in ≥ stall_threshold / 2 seconds, AND
3. No active ScheduleWakeup timer with a future fire time, AND
4. No active long-MCP-tool within its own per-tool timeout, AND
5. The subprocess CPU usage is < threshold over the last 30 s (existing cpu_active check)

Only when ALL of those hold simultaneously may a Telegram stall warning be enqueued.

structlog WARNING events still fire on each threshold crossing for untether-issue-watcher and ops dashboards — that observability is preserved. The change is purely the chat-side surfacing decision.

Required new info logs (so suppressions are auditable)

Each suppression branch emits an info log so we can verify in journalctl that the right rule fired:

• progress_edits.stall_long_bash_suppressed — Bash with fresh stdout
• progress_edits.stall_bash_grace_suppressed — Bash within grace window
• progress_edits.stall_schedule_wakeup_suppressed — ScheduleWakeup pending
• progress_edits.stall_monitor_active_suppressed — Monitor/bg-task heartbeat fresh
• progress_edits.stall_post_result_suppressed — already proposed in stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470
• (existing progress_edits.stall_children_active_suppressed covers the cpu_active branch)

Updated acceptance criteria

• Every long-running Bash/ScheduleWakeup/MCP call > 60 s emits a visible progress update at minimum every 30 s.
• Bash stdout is surfaced as a rolling sub-line, ≤ 120 chars, refreshed ≤ every 5 s.
• ScheduleWakeup renders a live countdown to fire-time.
• Zero Telegram stall warnings during any of the suppression states above.
• structlog WARNINGs continue to fire on every threshold crossing (regression test).
• All five new progress_edits.stall_*_suppressed info logs land.
• Genuinely frozen sessions (all 5 conditions hold simultaneously) DO produce a Telegram stall warning — ie suppression is precisely scoped, not a blanket disable.
• Integration test: a 5-minute synthetic Bash polling loop produces an active progress surface for the full 5 min and zero stall warnings.
• Integration test: a 30-minute fully-frozen subprocess (kill -STOP the tree) produces exactly one Telegram stall warning at the threshold crossing, then escalates per existing Claude Code stuck after MCP tool_result via Untether (upstream claude-code#39700) #322 logic.

This issue and #470 must ship together in v0.35.3 — the suppression matrix is shared.

[nathanschram]

rc8 implemented locally (commit 1334b3e)

Implementation landed on feature/470-481-stall-suppression-visibility, version bumped to 0.35.3rc8. Not yet PR'd to dev — pending Nathan's review and integration-test pass. Coordinated with #470 in the same commit (shared progress_edits.stall_detected decision branch in runner_bridge.py).

What shipped

• 30 s heartbeat tick folded into existing _stall_monitor (new [progress] heartbeat_interval, default 30, range 5–120)
• ActionState.started_at + last_update_at wall-clock fields populated from new ProgressTracker.clock callable
• format_action_line long-running tail: ▸ Bash · 3m 47s · npm run build for non-completed actions > 60 s old, regardless of /verbose
• format_verbose_detail branches for BashOutput (last line of result_preview), KillShell, ScheduleWakeup (countdown + reason), Monitor (countdown)
• format_duration / format_countdown helpers
• now: float | None threaded through Presenter.render_progress Protocol, MarkdownFormatter.render_progress_parts, MarkdownPresenter, TelegramPresenter
• 5 new suppression branches in _stall_monitor, gated by not frozen_escalate master gate so genuinely-frozen sessions still warn:
  • progress_edits.stall_post_result_suppressed (stall-message backoff once last_event_type=result (post-result idle is benign noise to the user) #470)
  • progress_edits.stall_schedule_wakeup_suppressed (engine_state.live_wakeups deadline future)
  • progress_edits.stall_monitor_active_suppressed (engine_state.live_monitors deadline future)
  • progress_edits.stall_bash_grace_suppressed (new [watchdog] bash_grace_seconds, default 60 s)
  • progress_edits.stall_long_bash_suppressed (BashOutput within stall_threshold/2)
• Same 5 booleans gate the auto-cancel _STALL_MAX_WARNINGS arm via new progress_edits.stall_auto_cancel_suppressed_expected_wait log
• Bonus fix: delaySeconds field-name fix in _register_background_handle (runners/claude.py:418) — production wakeup deadlines were always 0.0 due to the only-delay_ms read

Tests

• 32 new unit tests across test_exec_bridge.py (11), test_claude_runner.py (3), test_verbose_progress.py (18)
• Full suite: 2548 passed, 82.26% coverage
• Lint clean, lockfile synced (rc7 → rc8)
• Basic integration test U3 via @untether_dev_bot: 33 s Claude Code run, zero stall warnings, ✓ turn complete footer renders cleanly

Known limitations / follow-ups

• #482 (NEW, milestone v0.35.3) — strict 5-second rolling Bash stdout sub-line is not achievable without upstream Claude Code interim tool_result deltas. rc8 ships the BashOutput-polling proxy (refreshes at each polling cycle, ~15-30 s). Foreground Bash without polling is still silent until completion.
• #289 (moved to v0.35.3) — ScheduleWakeup countdown rendering is accurate after the delaySeconds fix, but the timer doesn't actually fire follow-up runs without feat: full /loop support — agent self-pacing via ScheduleWakeup interception #289 implementation. Suppression of stall warnings while a wakeup is pending DOES work today.
• Long-running integration tests deferred to staging dogfood: 5-min synthetic Bash polling, 30-min kill -STOP for genuinely-frozen path, 12-min post-result idle to verify closing message. Unit tests cover the logic exhaustively but live runs need to happen on @untether_dev_bot before promoting to staging.

Coordination notes for reviewers

• Frozen-ring escalation (existing _FROZEN_ESCALATION_THRESHOLD = 3) overrides all 5 new suppressions — locked by 2 regression tests (test_stall_post_result_overridden_by_frozen_ring, test_stall_schedule_wakeup_overridden_by_frozen_ring)
• structlog WARN events at runner.py:1002 (subprocess.liveness_stall) and runner_bridge.py (progress_edits.stall_detected) are unchanged — untether-issue-watcher and ops dashboards continue receiving them. Only chat-side surfacing changed.
• Engine-agnostic by construction: Bash/BashOutput suppression uses tracker._actions; ScheduleWakeup/Monitor/post-result use getattr(stream, "engine_state", None) duck-typing (Claude only today, no-ops cleanly for Codex/OpenCode/Pi/Gemini/AMP)

[nathanschram]

Status update — 2026-05-06

Shipped in v0.35.3rc8 (commit 1334b3e). Awaiting v0.35.3 stable promotion before close.

The rc8 scaffolding (heartbeat tick, suppression matrix, format_action_line elapsed-time tail, format_verbose_detail per-tool branches) is the foundation for both the /loop work in #289 and the Monitor multi-line / Bash streaming work in #482 / #374.

See:

• docs/plans/2026-05-06-289-loop-and-cron-interception.md — full /loop plan (uses rc8 visibility)
• docs/plans/2026-05-06-482-bash-streaming-and-monitor-completion.md — Plan B (Monitor lines / Bash streaming / lifecycle hardening)

Co-shipped with #470.