Skip to content

log: capture sandbox stderr and keep a durable on-disk copy#144

Merged
Meirtz merged 2 commits into
masterfrom
log/sandbox-output-observability
Jul 3, 2026
Merged

log: capture sandbox stderr and keep a durable on-disk copy#144
Meirtz merged 2 commits into
masterfrom
log/sandbox-output-observability

Conversation

@Meirtz

@Meirtz Meirtz commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Delivers the two observability gaps inherited from #122's /log intent, additively — the structured LogRecord bridge with ack/replay stays; this only adds what that bridge cannot see.

  • fd-2 capture (log: sandbox fd-2 (stderr) output never reaches the host /log namespace #138): fd 2 becomes a capture pipe like fd 1. Child-process stderr, C-extension writes, and direct sys.stderr prints replay on the host /log namespace under agentix.sandbox.stderr (same chunked, deadlock-free drain as stdout). The root console handler is repointed at the saved REAL stderr, so stdlib records are not delivered twice and a worker diagnostic emitted against a broken outbound pipe cannot self-amplify.
  • Durable in-sandbox log (log: no durable in-sandbox log artifact — output that outlives the connection is unrecoverable #139): every bridged record and captured line is also appended to a size-bounded (64 MiB × 1 rotation) $AGENTIX_LOG_DIR/sandbox-<worker-id>.log (default /tmp/agentix; empty disables), so output that outlives the connection — or the host — has a post-mortem artifact. Best-effort: a write failure detaches the handler rather than reporting into its own capture pipe, and announces the detach on /log so a truncated file is distinguishable from a quiet sandbox.

Review

Three-lens adversarial review (fd lifecycle / log-routing topology / durability promises) with refuters per finding — 5 distinct defects confirmed with empirical repros, all fixed in the second commit:

  1. Unlocked handler.emit() from the drain path racing a threaded rollover permanently detached the file → handler.handle() (locked, still bypasses the logger tree).
  2. Worker crash tracebacks vanished into the reader-less capture pipe (zero bytes anywhere, exit misreported as 120) → main() restores fd 2 from the saved real stderr on the way out; verified by spawning a worker with an injected read_frame crash — full traceback now reaches the container stderr, exit 1.
  3. The shared default path let two workers race one file's rotation (rename fails → permanent detach) → per-worker filename; every spawn has a fresh id.
  4. Outbound-write failures logged a traceback per frame, self-sustaining against a broken pipe and flooding the durable file during the death window → first failure at exception level, repeats at debug.
  5. Silent permanent detach was indistinguishable from a quiet sandbox → announced on /log (the wire is not the broken resource).

Known standing constraint (pre-existing in kind for the fd-1 capture, now shared by fd 2, documented in-code): a remote async def that blocks the event loop while its child spews ≥64 KiB into a captured fd wedges both sides — async callables must not block the loop; sync callables run in threads and are unaffected.

Testing

  • 4 new behavior tests (TDD, red→green): sys.stderr print → host record; child-process stderr → host record; stdlib records NOT re-captured as raw stderr; durable file contains records + both stdio streams.
  • pytest -q tests plugins: 607 passed, 6 deselected; ruff clean; pyright 0 errors at 3.11 and 3.13.

Closes #138. Closes #139.

🤖 Generated with Claude Code

Meirtz and others added 2 commits July 3, 2026 18:15
fd 2 becomes a capture pipe like fd 1: child-process stderr, C-extension
writes, and direct sys.stderr prints replay on the host /log namespace
under agentix.sandbox.stderr (#138). The structured record bridge stays —
capture is additive, and the root console handler is repointed at the
REAL stderr so stdlib records are not delivered twice and a worker
diagnostic on a broken outbound pipe cannot self-amplify.

Every bridged record and captured line is also appended to a
size-bounded $AGENTIX_LOG_DIR/sandbox.log inside the sandbox (#139), so
output that outlives the connection — or the host — has a post-mortem
artifact. Best-effort: a write failure detaches the file handler for
good rather than reporting into its own capture pipe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial review confirmed five defects, all fixed with empirical
repros: captured-line file writes now go through handler.handle() (an
unlocked emit racing a threaded rollover closed the stream and
permanently detached the file); the durable log is per-worker
(sandbox-<id>.log — two processes rotating one file race doRollover's
rename, same permanent detach, and the default dir is machine-shared);
main() restores fd 2 from the saved real stderr on the way out (a
crash traceback printed after the loop closed went to a reader-less
pipe — zero bytes anywhere, exit misreported as 120); outbound-write
failures log a traceback once, not per frame (each logged traceback
became a /log frame on the same broken queue, flooding the durable
file during the death window); and detaching the file handler now
announces itself on /log so a truncated post-mortem file is
distinguishable from a quiet sandbox.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Meirtz Meirtz merged commit 500bb56 into master Jul 3, 2026
5 checks passed
@Meirtz Meirtz deleted the log/sandbox-output-observability branch July 3, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant