Skip to content

Add Tcp.sendBytes — a byte-clean one-shot TCP call - #764

Merged
jasisz merged 2 commits into
jasisz:mainfrom
n1bor:tcp-send-bytes
Aug 1, 2026
Merged

Add Tcp.sendBytes — a byte-clean one-shot TCP call#764
jasisz merged 2 commits into
jasisz:mainfrom
n1bor:tcp-send-bytes

Conversation

@n1bor

@n1bor n1bor commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Add Tcp.sendBytes — a byte-clean one-shot TCP call

Fixes #763. Broader context, including the other three binary-protocol
blockers this does not address: #753.

The problem

aver-rt/src/tcp.rs ends send with:

Ok(String::from_utf8_lossy(&buf).into_owned())

Every non-UTF-8 sequence in the response becomes U+FFFD. The substitution is
many-to-one — 128 of the 256 possible single bytes collapse to the same
replacement character — so it cannot be undone. It is also silent: the call
returns Result.Ok with mangled data, and String.len reports the same length
an uncorrupted read would, so neither the return shape nor a length check
reveals it.

That makes every binary protocol unreachable from Aver. Demonstrated against a
server that always replies with the four bytes F9 BE B4 D9 (the Bitcoin
mainnet magic — chosen because each byte is invalid UTF-8 for a different
reason):

Tcp.send      -> Result.Ok("����")
Tcp.sendBytes -> Result.Ok([249, 190, 180, 217])

The change

Adds Tcp.sendBytes : (String, Int, List<Int>) -> Result<List<Int>, String>.

Identical socket behaviour to Tcp.send — open, write, shutdown(Write), read
to EOF, same 10 MB cap, same timeouts, same port validation — but the payload
and the response stay bytes end to end, with no encoding or decoding in either
direction.

Design decisions

Additive, not a fix in place. Changing Tcp.send's return type would break
every existing caller, and making it error on invalid UTF-8 would regress
callers who currently tolerate lossy text. A sibling method leaves existing code
untouched and lets the docs steer new code to the right one.

List<Int> as the byte carrier. Follows the existing Byte namespace
convention, which already operates on Int rather than introducing a byte
primitive. Worth a maintainer opinion: Vector<Int> would give O(1) indexed
access, which matters more when parsing a length-prefixed header than when
sending one. I went with List<Int> for consistency with Args.get and
Disk.listDir, and because Vector.fromList makes the conversion cheap at the
one place it matters.

Out-of-range bytes are catchable, not fatal. A value outside 0..=255
returns Result.Err("Tcp.sendBytes: byte 256 at index 1 is out of range (0–255)") rather than trapping — the same treatment the port range gets after
Phase 4.7+ fix #13. The index is included so a long payload is debuggable. A
non-List<Int> argument is still a type error, caught statically.

What's wired

  • aver-rt::tcp::send_bytes — the runtime call
  • VM path: service dispatch, register/register_nv, DECLARED_EFFECTS,
    effects, and the VmBuiltin enum
  • Type checker: signature and effect registration
  • Effect classification: GenerativeOutput, with two new RuntimeType
    variants (ListInt, ResultListIntStr). Oracle verify <fn> trace works
    with a given tcp: Tcp.sendBytes = [stub] binding — confirmed against
    .result, .trace.length() and .trace.contains()

What's not wired — deliberately, and I'd like direction

Codegen backends. Tcp.sendBytes is VM-only. I stopped there rather than
guess at three backends in a first PR:

  • Rust codegen currently reports Compiled ... [Rust], then the generated
    project fails at cargo build with "MIR walker could not render the main fn
    body". Not a silent miscompile, but the error arrives one stage too late. The
    binding itself looks like a short arm next to the existing Tcp.send one; the
    Vec<u8>AverList conversion is the only real work.
  • wasm-gc rejects it — but so does any program with a List<Int> literal,
    with or without this change (List literal: cannot resolve list instantiation (got List<Int>)). I verified that against the released 0.27.1 binary too, so
    it's pre-existing and orthogonal.
  • wasip2 already rejects all Tcp.* as host territory, so nothing to do.

Happy to extend to any of these if you'd point me at the shape you'd want.

CHANGELOG. Left alone on purpose — AGENTS.md is explicit that
tools/release.py owns version headers, and adding an entry means guessing the
next version number.

cargo fmt --all -- --check passes clean (rustfmt 1.9.0) with no diff.

Tests

Two added to tests/eval_spec.rs, alongside the existing tcp_tests:

  • tcp_send_bytes_round_trips_non_utf8 — echo server, asserts
    [249, 190, 180, 217] survives intact. Marked #[ignore] like its
    tcp_send_echo_server neighbour since it binds a socket.
  • tcp_send_bytes_rejects_out_of_range_byte — asserts the error names both the
    offending value and its index.

Manually verified: empty payload → Result.Ok([]); connection refused,
out-of-range port, and a String passed where List<Int> is expected all
produce the right diagnostics.

`Tcp.send` decodes its response with `String::from_utf8_lossy`, so every
non-UTF-8 sequence becomes U+FFFD. The substitution is many-to-one — 128
of the 256 single-byte values collapse to the same replacement character
— so it cannot be undone, and it is silent: the call still returns
`Result.Ok`, and `String.len` reports what an uncorrupted read would.
That puts every binary protocol out of reach.

Add `Tcp.sendBytes : (String, Int, List<Int>) -> Result<List<Int>, String>`
alongside it. Same socket behaviour as `Tcp.send` — open, write,
shutdown(Write), read to EOF, same cap, timeouts and port validation —
but payload and response stay bytes end to end.

Additive rather than a change in place: reshaping `Tcp.send` would break
every existing caller, and erroring on invalid UTF-8 would regress the
ones already living with lossy text.

Byte values outside 0..=255 return a catchable `Result.Err` naming the
value and its index, matching how the port range is handled rather than
trapping. A non-`List<Int>` payload stays a static type error.

Wired through the VM path, the type checker, and the effect
classification table, so Oracle `verify <fn> trace` can stub it. The
codegen backends are not wired yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@jasisz jasisz left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed, and ran it locally — both new tests pass (including the integration one with --include-ignored), fmt is clean. This is a well-made first PR: the wiring is complete across the VM path, the additive sibling is the right call, and the docs/llms.txt steering toward sendBytes for binary work is appreciated.

Answers to your open questions:

  • List<Int> is right for now. Consistency with the Byte namespace wins at this stage; if parsing pressure shows Vector<Int> should be the primary shape, that's a pre-1.0 revisit, not something to solve here.
  • VM-only is fine for this PR. I'd take the Rust codegen arm as a follow-up rather than grow this one. (Edited: I originally echoed the "wasip2 is host territory" line from the PR description here — that's stale, and the mistake is on me since it's my own code: Tcp.* graduated on --target wasip2 in 0.20 with an in-module wasi-sockets implementation, and plain --target wasm-gc routes Tcp.send through an aver.tcp_send host import. So both wasm targets can and will carry sendBytes — I'm taking that as follow-up PRs alongside the Rust codegen arm.)

One fixup before I merge: in bytes_arg, an Int outside i64 falls into the to_i64() failure and reports "payload must be a List". Aver's Int is arbitrary-precision, so this is reachable from ordinary source, and it lands as a runtime trap with a wrong message:

[65, 256]                        → Result.Err("… byte 256 at index 1 is out of range (0–255)")   — catchable, as designed
[65, 1208925819614629174706176]  → Runtime error: Tcp.sendBytes: payload must be a List<Int>     — trap, and the message is wrong

By your own (correct) design both belong in the catchable out-of-range Result.Err, index included — treating to_i64()'s None the same as the u8::try_from failure should do it.

The two things you hit while testing deserve their own issues — the Rust codegen reporting Compiled [Rust] and then failing at cargo build, and the wasm-gc List<Int> literal rejection. Would you file them with your repros? You found them, you should get the credit.

CI and Proof are already green here; once the fixup is in and everything stays green, I'll merge.

@jasisz

jasisz commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Update on the two side-findings: the wasm-gc List<Int> literal rejection turned out to be a discovery gap in the backend's type registry and is fixed in #766 — no need to file that one. The Rust codegen reporting success and then failing at cargo build is still worth an issue if you're up for it.

@jasisz

jasisz commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Didn't want your first PR hanging on a 3-liner, so I pushed the fixup myself (346f3f7) — an Int outside i64 now takes the same catchable out-of-range Result.Err as 256, index included, with a regression test alongside yours. Hope you don't mind. Merging once CI is green — thanks again, this makes binary responses actually reachable from Aver.

@jasisz
jasisz merged commit 1898c98 into jasisz:main Aug 1, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tcp.send silently corrupts non-UTF-8 responses, making binary protocols unreachable

2 participants