Skip to content

Fix unbounded stderr_buffer leak; make Daemon stderr reliable#8

Merged
nyo16 merged 2 commits into
masterfrom
fix/stderr-buffer-leak
Jun 28, 2026
Merged

Fix unbounded stderr_buffer leak; make Daemon stderr reliable#8
nyo16 merged 2 commits into
masterfrom
fix/stderr-buffer-leak

Conversation

@nyo16

@nyo16 nyo16 commented Jun 28, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes two findings from a performance review of the OTP I/O paths.

1. Unbounded stderr_buffer memory leak

In the default :consume stderr mode, NetRunner.Process drained the child's
stderr pipe (correct — keeps the child from blocking on a full pipe) but
prepended every chunk to state.stderr_buffer, a list that was never read
anywhere in lib/ (verified write-only). For long-running processes —
especially NetRunner.Daemon — this grew the GenServer heap without bound,
retaining data nothing could use.

NetRunner.Stream.AbnormalExit already defined a :stderr field and a
message/1 that formats it, but was never raised with stderr populated — the
orphaned consumer half of a half-built diagnostic feature.

Fix: retain only a bounded tail of stderr.

  • Keep the most-recent :stderr_tail_bytes bytes (default 8192) as a single
    binary; drain-and-drop the rest. Stats still count every byte — only
    retention is capped.
  • New accessor NetRunner.Process.stderr_tail/1 surfaces it (useful for
    diagnosing why a command failed).
  • New :stderr_tail_bytes option, validated as a non-negative integer (0
    disables retention while still draining).

2. Daemon stderr-pipe race

NetRunner.Daemon ran its own stderr drain task and left the default
:consume internal consumer on, so the two raced the same FD and on_output
received an arbitrary subset of stderr. Daemon.init/1 now forces
stderr: :disabled so its drain task is the sole reader and on_output gets
the full stream, in order.

Deliberately out of scope

  • stream!/2 / run/2 halt semantics are unchanged (no new raise on
    nonzero exit). AbnormalExit is kept defined-but-unraised as a future opt-in,
    to avoid a breaking change.
  • The select-handler micro-optimization (retry only the FD that fired) was
    deferred — low impact at current usage (0–2 parked reads).

Notes

  • No C / shepherd / protocol changes — entirely BEAM-side. PTY mode unaffected.
  • The stderr tail is raw bytes and may begin mid-character; treat it as
    diagnostic text, not guaranteed-valid UTF-8 (documented).

Test plan

  • mix format --check-formatted, mix compile --warnings-as-errors,
    mix credo --strict, mix dialyzer — all clean.
  • mix test — 150 passed (added test/stderr_tail_test.exs: tail bound,
    custom cap, cap=0, leak guard (O(1) retention vs O(n) throughput), option
    validation, :disabled read path, and Daemon on_output full-stream).

nyo16 added 2 commits June 28, 2026 12:54
In the default :consume stderr mode, every stderr chunk was prepended to
state.stderr_buffer — a list that was never read anywhere in lib/. For
long-running processes (especially NetRunner.Daemon) this grew the GenServer
heap without bound, retaining data nothing could use.

Replace it with a bounded tail:
- Retain only the most-recent :stderr_tail_bytes bytes (default 8192) as a
  single binary; drain-and-drop the rest. Stats still count every byte.
- Expose it via NetRunner.Process.stderr_tail/1 (the long-orphaned
  AbnormalExit.stderr feature; kept defined-but-unraised for a future opt-in).
- Add and validate the :stderr_tail_bytes option (non-negative integer; 0
  disables retention while still draining).

Also fix a stderr-pipe race in NetRunner.Daemon: it ran its own stderr drain
task while leaving the default :consume internal consumer on, so the two raced
the same FD and on_output saw an arbitrary subset. Daemon now forces
stderr: :disabled so its drain task is the sole reader and on_output receives
the full stream in order.

No C/shepherd/protocol changes. PTY mode unaffected.
The macOS test job ran OTP 27.2 / Elixir 1.17.3, which setup-beam does not
reliably provide for the now-arm64 macos-latest runner — bump it to
OTP 28.3.3 / Elixir 1.18.4 to match a supported build.

Also bump actions/checkout from the aging v4 to v7 across all jobs.
@nyo16 nyo16 force-pushed the fix/stderr-buffer-leak branch from dec3266 to e2a6f4a Compare June 28, 2026 17:22
@nyo16 nyo16 merged commit 150a84c into master Jun 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant