AFTP v2: fountain-coded UDP data plane (--fec) + BBR-tuned TCP for lossy links - #1
Merged
Conversation
Adds a split-plane AFTP v2: the TCP control plane negotiates FEC and
carries feedback, while file data rides a new UDP data plane as RaptorQ
symbols. Packet loss becomes repair bandwidth instead of TCP round-trip
stalls, so transfers survive links where every TCP tool collapses.
Core (src/aftp/fec/):
- codec.rs RaptorQ block encode/decode, systematic symbols first
- envelope.rs HMAC-SHA256-authenticated symbol envelope (session-scoped)
- udp.rs UDP data plane, 8 MiB socket buffers, DoS-quiet reject
- pacing.rs BBR-style pacer (BtlBw/RTprop filters, gain cycle, token
bucket with sub-ms debt accumulation)
- transfer.rs block scheduler, per-block adaptive source/repair, bounded
sender memory (encoder dropped on BlockOk), feedback loop
Control plane: FEC_OFFER/ACCEPT/NEEDMORE/BLOCK_OK frames + CAP_FEC,
fail-safe negotiation (falls back to the reliable TCP path against a
peer without FEC). Wired through client get/put and server get/put.
Congestion control tuning that made the lossy regimes complete:
- receiver patience adapts from measured NeedMore round-trips (cap 1 s)
- pacer fed real delivery samples from feedback (were dead code)
- startup guard caps unsampled bytes so block 0 can't flood a slow queue
- delivery samples are per-interval deltas, not lifetime averages
- loss-hint decays when repair proves unnecessary
Also:
- Cargo.toml: crc32fast/rcgen/aws-lc-rs were scoped under cfg(windows),
silently making the crate Windows-only; moved to [dependencies]. Add
raptorq = "2".
- docs/BENCHMARKS.md, README.md: replace fabricated third-party
throughput comparison with a measured netem head-to-head; correct
turbo-engine claims (no mmap uploads; socket tuning is real).
- bench/: netem harness + raw result data, so every published number
is reproducible and traceable.
Measured (50 MB, netem, median): good 3.2 s, bad 13.0 s (6/6, only
completer), broken 103 s (3/3). Every TCP contender times out on
bad/broken. Cost: ~150 MB sender RSS (bounded, file-size independent).
Tests: 76 lib + 306 release integration + 7 FEC e2e pass; clippy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The plain TCP path timed out on the lossy netem regimes because it inherited the kernel default, CUBIC. CUBIC is loss-based: it reads the random 2%/10% netem drop as congestion and cuts its window, so throughput collapses to ~MSS/(RTT*sqrt(p)) — a few hundred KB/s at 2% loss — regardless of free bandwidth. Not a bug in our code; CUBIC doing what CUBIC does. tune_tcp_socket now requests BBR per-socket via setsockopt(TCP_CONGESTION, "bbr") on Linux, best-effort (falls back to the kernel default if the tcp_bbr module is absent). BBR models bottleneck bandwidth and RTprop and ignores loss as a congestion signal — the same principle as the FEC data plane's pacer, applied to the reliable path. Measured (50 MB, netem): - bad (2% loss / 80 ms): timeout -> 14.8 s median (3/3), ~47 MB RSS. Now competitive with --fec (13.0 s) at a third of the memory. CUBIC tools (atp-tcp, rsync) still time out. - broken (10% loss + reorder + dup): still times out. Congestion control was never the bottleneck here; TCP's reliability layer is — every lost byte costs a retransmit RTT and reordering triggers spurious fast-retransmits. That regime needs --fec (completes 3/3). Docs updated: head-to-head table now shows aft (TCP + BBR) completing bad; the "only completer" claim moves from bad to broken; caveat added that the bad result requires tcp_bbr to be loaded on the host. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surface the netem head-to-head (aft --fec, aft TCP+BBR, atp, rsync) directly in the README Performance section, with the good/bad/broken results and the "which path for which link" guidance. Points to bench/ and docs/BENCHMARKS.md for raw data and methodology. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the BBR fix. The head-to-head showed the TCP path trailing atp on the clean `good` regime (3.9 s vs 2.9 s), the only cell where AFT was slower. That gap was a measurement artifact: `good` on netem has high run-to-run variance, and a 3-run median caught BBR on an unlucky sample and atp on a lucky one. Re-measured at 9 runs per tool: - aft (TCP, BBR): 2.8 s median, range 2.74-2.92 (tightest) - atp (TCP): 3.0 s median, one 14.6 s outlier - atp (RaptorQ): 3.0 s - rsync (ssh): 3.3 s - aft --fec: 3.4 s AFT's TCP path is now the fastest tool measured on a clean link, and the most consistent — BBR's pacing keeps the tail tight, while the CUBIC- based tools go bimodal when the 0.1% random loss trips a window cut. So BBR is the right default across the board: it wins `good` outright and carries `bad` (14.8 s) where CUBIC collapses. Made the algorithm selectable via AFT_TCP_CC (default "bbr", empty = kernel default) so operators can pin CUBIC or any installed algorithm per site. Docs/README/bench updated with the 9-run `good` row and the corrected narrative; raw data in bench/results_good_9run.jsonl. Verified: 306 release integration tests pass; clippy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… bound receiver memory A pre-release security audit flagged the FEC data plane as NO-GO: it moved file contents in cleartext over UDP, silently voiding the transmission confidentiality the tool advertises the moment `--fec` engaged — even on a TLS (`aftps://`) connection. This closes that and the compounding findings. C1 (Critical) — confidentiality. Rewrote the symbol envelope to seal the payload with AES-256-GCM (wire version 2): `prefix(18) || nonce(12) || ciphertext || tag(16)`, the 18-byte header bound in as GCM AAD, a fresh 96-bit OsRng nonce per datagram. FLAG_ENCRYPTED is now implemented, not refused; the HMAC-over-plaintext path is gone. New tests assert the plaintext never appears on the wire, the nonce varies per datagram, and nonce/AAD tampering is rejected. H2 (High) — replay/predictability. The upload session id was `DefaultHasher(host, port, file_size)` — fixed-key SipHash, attacker- computable and identical across same-size uploads, so the derived symbol key was predictable and reused. Both paths now use `fec::random_session_id()` (OsRng u64), exchanged in the offer on the control plane. Key label bumped to v2. M4 (Medium) — memory DoS. Downloads streamed the whole peer-declared file into RAM. `download_fec` now streams to a `.aftp-part` temp via FileBlockWriter, SHA-256-verifies by streaming, then renames into place (temp removed on any failure — still fail-closed at the destination). `recv_object_into` caps concurrent RaptorQ decoders to the window so a sender cannot force unbounded ~8 MiB allocations. H3 / M5 — documented (not code-changed, consistent with the existing crypto pipeline): docs/SECURITY.md gains a "FEC Data Plane Security" section stating the FEC crypto uses the aes-gcm/hmac crates (not the aws-lc-rs FIPS provider, which covers only TLS) and that unauthenticated `--fec` is CRC32-only, lab-only, never for CUI. README carries a confidentiality caveat. Also: clippy `-D warnings` fixes so the branch is CI-clean on Linux (apply_feedback/turbo allow-attributes, sort_by_key, clamp, is_some_and). Re-audited: GO. 64 lib + 306 release integration + 7 FEC e2e tests pass; clippy -D warnings clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
admercs
added a commit
that referenced
this pull request
Jul 26, 2026
…sent gate, perf, FIPS, dep bumps)
Implements all of HANDOFF.md §6, verified on Linux (cargo test --lib: 104,
integration_tests: 312, clippy --all-targets -D warnings clean under both the
default and `fips` feature sets; release build links).
#5 QUIC-datagram data plane + whole-tree packing
- Negotiate a QUIC-datagram FEC plane end-to-end via CAP_FEC_QUIC (0x80);
fall back transparently when unsupported (src/aftp/fec/quic_dgram.rs).
- Fold a directory into one fountain-coded object via a Merkle manifest
(src/aftp/fec/manifest.rs). Both gated by the same auth/consent rules.
#4 Client-side unauthenticated-FEC consent gate
- The client refuses to advertise CAP_FEC on an unauthenticated connection
unless it opts in with --fec-insecure; the gate sits before capability
advertisement so the server is never offered a plane the client would drop.
#3 Broken-regime FEC performance
- Root cause: the receiver patience timer was seeded from a hardcoded 50 ms
RTT (100 ms patience) vs a ~400 ms real RTT, firing NeedMore before symbols
could arrive and provoking premature repair / timeouts.
- Fix: seed patience from a measured control-plane RTT (client GET->HEAD_RESP,
server HELLO_ACK->first-command), clamped to [50 ms, 500 ms].
- A/B on the broken regime (n=5): 2/5 timeouts -> 0/5, median ~137 s -> ~79 s.
- Adds AFT_FEC_TRACE per-block sender instrumentation (off by default).
#2 FIPS-route the FEC crypto
- Centralize the data-plane AEAD + KDF in src/aftp/fec/symcrypto.rs, routing
to aws-lc-rs (FIPS 140-3) under --features fips and RustCrypto otherwise.
Wire-identical across backends (KDF known-answer test).
#1 Dependency major-bump migration
- russh 0.46 -> 0.62 (RUSTSEC-2026-0154/0153): russh-keys folded into
russh::keys, native async Handler, AuthResult, PrivateKeyWithHashAlg.
- rust-s3 0.35 -> 0.37.2: clears the quick-xml 0.32 and rustls-webpki 0.101.7
pins (RUSTSEC-2026-0194/0195/0098/0099/0104); S3 handler unchanged.
- pqc_kyber -> ml-kem 0.3.2 (KyberSlash): FIPS-203 ML-KEM-1024, a deliberate
breaking key-format change (v1->v2 header; legacy files rejected with a
regenerate hint). rsa Marvin is now transitive-only via ssh-key (monitor).
Docs: refresh HANDOFF.md §6 and SECURITY.md (FIPS status, algorithm table,
advisory triage).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
admercs
pushed a commit
that referenced
this pull request
Jul 26, 2026
…sent gate, perf, FIPS, dep bumps)
Implements all of HANDOFF.md §6, verified on Linux (cargo test --lib: 104,
integration_tests: 312, clippy --all-targets -D warnings clean under both the
default and `fips` feature sets; release build links).
#5 QUIC-datagram data plane + whole-tree packing
- Negotiate a QUIC-datagram FEC plane end-to-end via CAP_FEC_QUIC (0x80);
fall back transparently when unsupported (src/aftp/fec/quic_dgram.rs).
- Fold a directory into one fountain-coded object via a Merkle manifest
(src/aftp/fec/manifest.rs). Both gated by the same auth/consent rules.
#4 Client-side unauthenticated-FEC consent gate
- The client refuses to advertise CAP_FEC on an unauthenticated connection
unless it opts in with --fec-insecure; the gate sits before capability
advertisement so the server is never offered a plane the client would drop.
#3 Broken-regime FEC performance
- Root cause: the receiver patience timer was seeded from a hardcoded 50 ms
RTT (100 ms patience) vs a ~400 ms real RTT, firing NeedMore before symbols
could arrive and provoking premature repair / timeouts.
- Fix: seed patience from a measured control-plane RTT (client GET->HEAD_RESP,
server HELLO_ACK->first-command), clamped to [50 ms, 500 ms].
- A/B on the broken regime (n=5): 2/5 timeouts -> 0/5, median ~137 s -> ~79 s.
- Adds AFT_FEC_TRACE per-block sender instrumentation (off by default).
#2 FIPS-route the FEC crypto
- Centralize the data-plane AEAD + KDF in src/aftp/fec/symcrypto.rs, routing
to aws-lc-rs (FIPS 140-3) under --features fips and RustCrypto otherwise.
Wire-identical across backends (KDF known-answer test).
#1 Dependency major-bump migration
- russh 0.46 -> 0.62 (RUSTSEC-2026-0154/0153): russh-keys folded into
russh::keys, native async Handler, AuthResult, PrivateKeyWithHashAlg.
- rust-s3 0.35 -> 0.37.2: clears the quick-xml 0.32 and rustls-webpki 0.101.7
pins (RUSTSEC-2026-0194/0195/0098/0099/0104); S3 handler unchanged.
- pqc_kyber -> ml-kem 0.3.2 (KyberSlash): FIPS-203 ML-KEM-1024, a deliberate
breaking key-format change (v1->v2 header; legacy files rejected with a
regenerate hint). rsa Marvin is now transitive-only via ssh-key (monitor).
Docs: refresh HANDOFF.md §6 and SECURITY.md (FIPS status, algorithm table,
advisory triage).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Makes AFT competitive with ATP on the lossy/latent links where TCP-only transfer structurally collapses. Two complementary paths, each measured head-to-head under
tcnetem:setsockopt(TCP_CONGESTION, "bbr"), Linux, best-effort). BBR ignores random loss as a congestion signal, so the TCP path finishes where CUBIC collapses.--fecfountain-coded UDP data plane — for links so lossy that reliable transport cannot drain at all (10% loss + reordering), file data rides RaptorQ symbols over UDP with a TCP control plane. Loss becomes a bit of extra repair bandwidth instead of a retransmit round trip.Architecture
Split-plane AFTP v2: existing TCP control plane (negotiation + feedback via new
FEC_OFFER/ACCEPT/NEEDMORE/BLOCK_OKframes,CAP_FEC) alongside a new UDP data plane (src/aftp/fec/): RaptorQ codec, HMAC-authenticated symbol envelope, BBR-style pacer, block scheduler with per-block adaptive source/repair and bounded sender memory. Negotiation is fail-safe — against a peer without FEC the client falls back to the reliable TCP path.Measured head-to-head (50 MB, netem, median)
bad, BBR-TCP finishes at ~47 MB RSS — competitive with--fecat a third of the memory; CUBIC-based tools time out.broken, only--feccompletes; even BBR-TCP cannot, because the bottleneck there is TCP's reliability layer, not congestion control.Full methodology, raw data, and honest caveats in
docs/BENCHMARKS.mdandbench/(netem harness + result JSONL, every number reproducible).Also in this PR
crc32fast/rcgen/aws-lc-rswere TOML-scoped undercfg(windows), silently making the crate Windows-only. Moved to[dependencies].--fecCLI flag (global, negotiated).Caveats (also in docs)
aft --fecpeak RSS ~150 MB (bounded, file-size independent) — ~12–15× rsync/ATP.badTCP result requires thetcp_bbrmodule loaded on the host (modprobe tcp_bbr); BBR is a Linux per-socket opt-in, Windows/macOS use the OS default.--rq-allow-unauthenticated-lab/ netem interaction), not necessarily algorithmic.Verification
76 lib + 306 release integration + 7 FEC e2e tests pass; clippy clean.
🤖 Generated with Claude Code