Skip to content

Preserve raw worker stdin at the server boundary#72

Merged
t-kalinowski merged 12 commits into
mainfrom
zod-crlf-stdin-coverage
May 19, 2026
Merged

Preserve raw worker stdin at the server boundary#72
t-kalinowski merged 12 commits into
mainfrom
zod-crlf-stdin-coverage

Conversation

@t-kalinowski
Copy link
Copy Markdown
Member

@t-kalinowski t-kalinowski commented May 19, 2026

Summary

This PR tightens the worker stdin contract so the server no longer canonicalizes client-supplied line endings before handing input to the worker. Non-empty input still gets one trailing \n when it is not already newline-terminated, but supplied \r\n and bare \r bytes are preserved.

It also updates control-prefixed input handling so Ctrl-C/Ctrl-D consume only the control byte. Any immediate newline remains part of the follow-up input, so Ctrl-D is a bare restart, while Ctrl-D\n is a restart followed by a blank input and Ctrl-D\nfoo is a restart followed by the leading blank line plus foo.

Protocol-worker shutdown handling is hardened by validating session_end.reason and optional message_b64, and reset handling now snapshots output before graceful shutdown so teardown-only output does not leak into the new session started reply.

Public Facing Changes

  • repl stdin preserves supplied CRLF and bare-CR bytes at the server boundary, with only the final-newline append rule.
  • Ctrl-C/Ctrl-D followed by newline is treated as a control action with follow-up input, not as a bare control.
  • Invalid protocol-worker session_end metadata now fails as a protocol error instead of being accepted as a clean shutdown.

Internal Changes

  • Expanded the Zod fixture with deterministic hooks for raw stdin reporting, interrupt facts, buffered-stdin discard, active stdin reads, delayed/hanging shutdown, and malformed session_end frames.
  • Added Zod protocol coverage for stdin preservation, control-tail newlines, interrupt delivery, reset EOF behavior, shutdown hooks, and protocol validation.
  • Updated tests and snapshots to use unterminated raw control bytes where they mean bare Ctrl-C/Ctrl-D.
  • Updated protocol docs and test README examples to match the raw stdin contract.

Diff Composition

Measured against main, this PR is 783 insertions and 100 deletions across 22 files.

  • runtime src/: +93/-26 (13.5% of churn)
  • inline tests inside src/: +2/-2 (0.5% of churn)
  • tests in tests/: +669/-61 (82.7% of churn)
  • docs: +11/-3 (1.6% of churn)
  • snapshots: +8/-8 (1.8% of churn)

Largest files:

  • tests/zod_protocol.rs: +343/-0
  • tests/fixtures/zod-worker.rs: +186/-16
  • src/worker_process.rs: +54/-19
  • tests/sandbox_state_updates.rs: +41/-11
  • tests/python_backend.rs: +32/-10

t-kalinowski added 12 commits May 19, 2026 09:52
@t-kalinowski
Add Zod readline discard recovery test
ed313df 
Teach the standalone Zod worker to emit readline_discard for buffered stdin after an interrupt so stale active-turn input is accounted for before a follow-up tail runs.

Validation: cargo check; cargo build; python3 tests/run_integration_tests.py --binary target/debug/mcp-repl; cargo clippy --all-targets --all-features -- -D warnings; cargo test --quiet; cargo +nightly fmt.
@t-kalinowski
Validate protocol session_end reasons
95d9bd9 
Reject worker sideband session_end frames that carry unrecognized reasons, and validate optional session_end message_b64 payloads before accepting the frame as final.

Add a Zod test hook and MCP-level regression coverage so malformed protocol-worker session_end data fails fast instead of being treated as a clean session shutdown.

Validation: cargo test --quiet --test zod_protocol zod_worker_invalid_session_end_reason_is_protocol_error; cargo test --quiet --test zod_protocol; cargo check; cargo build; python3 tests/run_integration_tests.py --binary target/debug/mcp-repl; cargo clippy --all-targets --all-features -- -D warnings; cargo test --quiet; cargo +nightly fmt
@t-kalinowski
Add Zod shutdown conformance hooks
c188689 
Add slow-shutdown and hang-shutdown commands to the standalone Zod worker so public protocol tests can exercise delayed graceful shutdown and pending shutdown states. Cover both through MCP-level tests and update the active protocol plan command list.

Validation:
- cargo test --quiet --test zod_protocol shutdown (red before fixture change)
- cargo test --quiet --test zod_protocol
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Split Zod interrupt observations
269d681 
Add an interrupt-report fixture command that records sideband interrupt
notifications separately from OS interrupt delivery. Cover it through the
public Zod protocol test and document the command in the active protocol plan.

Validation:
- cargo +nightly fmt
- cargo check
- cargo build
- cargo test --test zod_protocol zod_worker_reports_sideband_and_os_interrupt_facts -- --nocapture
- cargo test --test zod_protocol --quiet
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
@t-kalinowski
Preserve control-tail newline bytes
caf3519 
Consume only the leading Ctrl-C/Ctrl-D byte when splitting control-prefixed input so an immediate newline remains part of the tail payload. Add a public Zod MCP regression that proves the worker observes that newline before the tail command.

Update existing tests that intend bare controls to send unterminated control bytes, and update transcript snapshots to show those bare inputs explicitly.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Preserve protocol worker stdin bytes
c671370 
Move CR/CRLF normalization out of the shared request path and keep it in the built-in R/Python drivers, so custom protocol workers receive the exact client stdin bytes plus the existing single appended LF rule.

Add public Zod MCP coverage for supplied CRLF bytes and a trailing bare carriage return. Update the protocol plan to state the byte-preservation contract.

Validation:
- cargo test --test zod_protocol zod_worker_preserves_crlf_stdin_and_appended_newline -- --exact
- cargo test --test write_stdin_edge_cases write_stdin_accepts_crlf_input -- --exact
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Fix reset teardown output and Ctrl-C bundle reuse
8301046 
Snapshot pending output before reset shutdown so teardown-only worker output
does not leak into the reset reply. Also classify Ctrl-C plus newline/CRLF
as follow-up input for timeout bundle reuse; only bare Ctrl-C reuses the
full timeout reply.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Cover reset EOF for active Zod stdin
ca758d7 
Add a Zod command that blocks on a second stdin read and verify repl_reset
closes the active stdin stream without sending interpreter shutdown text.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Fix Unix Python CRLF stdin normalization
576d9d7 
Normalize CRLF and bare CR input before the built-in Unix Python backend
writes request bytes to its PTY. Keep custom protocol worker payloads
byte-preserving by applying the normalization only when the protocol driver is
serving the built-in Python bridge.

Add a Unix public regression test that sends a CRLF-formatted multiline Python
block through the MCP repl tool and asserts it executes without an injected
blank line or IndentationError.

Review finding:
- [P2] Normalize CRLF before Unix Python PTY writes — /Users/tomasz/github/posit-dev/mcp-repl/src/worker_process.rs:2624-2624
  On Unix, the built-in Python backend uses `ProtocolBackendDriver`, whose `prepare_input_payload` now preserves stdin bytes. Since the previous normalization was removed before this call, CRLF input is written to the PTY as `\r\n`; the terminal maps `\r` to another newline, so a block like `if True:\r\n    print("A")` reaches Python with a blank line and raises `IndentationError`. Custom protocol workers can preserve bytes, but built-in Python still needs newline normalization before writing to the PTY.

Response:
- Added `python_crlf_multiline_block_executes` to cover CRLF multiline Python input through the public MCP tool path.
- Updated `ProtocolBackendDriver::prepare_input_payload` so only the built-in Unix Python bridge normalizes CRLF/CR before PTY writes; custom protocol workers still preserve stdin bytes.

Validation:
- cargo test --test python_backend python_crlf_multiline_block_executes -- --nocapture (failed before fix, passed after fix)
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Fix Zod command fixture argument count
f6fa2ca 
Move the shutdown-log path into CommandState so run_command stays within the clippy argument limit after the stdin-close reset coverage was adapted.

Validation:
- cargo test --test zod_protocol --quiet
- cargo test --test python_backend --quiet crlf
- cargo test --quiet timeout_bundle_reuse_treats_newline_ctrl_c_as_follow_up_input
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Preserve worker stdin bytes at server boundary
91b7e43 
Keep server-side input payload preparation limited to the MCP UTF-8 string
boundary and the explicit final-newline append rule. Backend/runtime-specific
CRLF interpretation belongs to the worker instead of the server.

This also removes the Unix Python CRLF multiline expectation, keeps CRLF
acceptance scoped to Windows, and rewords the Zod protocol regression around
raw client byte preservation.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski
Fix CI stdin edge cases
a0d2472 
Normalize CRLF input in the built-in R backend before writing bytes to R so
Windows requests do not leave carriage returns in R source. Keep protocol
worker byte preservation intact by leaving the protocol driver unchanged.

Make the files-mode timeout poll regression deterministic by replacing a
short sleep with an explicit file gate before releasing the request.

Validation:
- cargo check
- cargo build
- python3 tests/run_integration_tests.py --binary target/debug/mcp-repl
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --quiet
- cargo +nightly fmt
@t-kalinowski t-kalinowski merged commit 1c960f3 into main May 19, 2026
5 checks passed
@t-kalinowski t-kalinowski deleted the zod-crlf-stdin-coverage branch May 19, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@t-kalinowski