rpc: pin QUIC InitialPacketSize at 1200 for 1280-MTU tunnels#870
Merged
phinze merged 1 commit intoJun 29, 2026
Merged
Conversation
…U tunnels quic-go defaults InitialPacketSize to 1280 bytes (a hardcoded const in v0.57.1). As a UDP payload that's a 1308-byte IPv4 datagram sent with DF set, so it can't traverse a 1280-MTU path. Tailscale and WireGuard tunnels use a 1280 MTU (the IPv6 minimum), so the first handshake packet to a cluster reached over one of them is undeliverable: the kernel rejects it with EMSGSIZE and nothing reaches the peer, surfacing as "timeout: no recent network activity". It breaks both directions, since the server's cert-bearing handshake flight hits the same wall. Pinning the initial at 1200 (the RFC 9000 spec minimum) lets the handshake fit through. Path MTU discovery raises the packet size again once the handshake completes, so 1500-MTU paths are unaffected. quic-go declined to lower their default (quic-go#5634, closed not-planned), so the fix belongs here in the application config. Fixes MIR-1270. Refs: quic-go#5573, quic-go#5634, tailscale#2633
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthrough
Comment |
phinze
deleted the
phinze/mir-1270-miren-rpc-quic-handshake-fails-over-1280-mtu-tunnels
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's going on
Remote
mirencalls to any cluster you reach over Tailscale hang on the first real RPC withtimeout: no recent network activity, while on-box calls and calls to public-IP cloud clusters work fine. The culprit turned out to be packet size, not anything in our RPC layer.quic-go bumped its default
InitialPacketSizeto 1280 bytes a few releases back (we're on v0.57.1, where it's a hardcodedconst InitialPacketSize = 1280). As a UDP payload that's a 1308-byte IP packet, and QUIC sets the don't-fragment bit. Tailscale'stailscale0has a 1280 MTU (the IPv6 minimum, standard for WireGuard-based tunnels), so the handshake's opening packet is too big to send: the kernel rejects it with EMSGSIZE and it never hits the wire. A packet capture on the server during a failing call showed zero inbound packets.It's symmetric: even once the client's Initial is small enough to get through, the server's handshake flight carries the TLS cert, gets capped at the same 1280, and drops on its own egress, so the handshake stalls there instead. Both ends need the smaller initial. This also explains why it never looked fleet-wide — every cloud cluster is addressed over a public 1500-MTU path where 1308 fits fine. Only Tailscale-addressed clusters hit the 1280 wall.
Since when
Not a recent regression. quic-go unified its initial size to 1280 at v0.44.0 (before that it used 1252 for IPv4, which just fit a 1280-MTU tunnel). miren has carried a quic-go past that line since the QUIC/cbor RPC stack first landed (v0.48.2, Dec 2024), so every tagged release has shipped this. The recent v0.49.0 to v0.57.1 bump is unrelated; both carry the 1280 default.
The fix
Pin
InitialPacketSizeto 1200, the RFC 9000 minimum (§14.1: an Initial datagram may only exceed 1200 bytes "if the sender believes that the network path and peer both support" it, which we can't assume across arbitrary tunnels). That's a 1228-byte IPv4 datagram and 1248-byte IPv6, both comfortably under a 1280 MTU. Path MTU discovery still ramps packets back up after the handshake, so nothing changes for normal 1500-MTU paths.This is a known quic-go-over-Tailscale gotcha (tailscale#2633, open since 2021; quic-go#5573 is our exact symptom). quic-go declined to lower their own default (quic-go#5634, closed not-planned), so the right place to fix it is our application config.
How it was verified
Built a patched client and server and ran against a homelab cluster reached over Tailscale. Before:
app listtimes out, zero packets at the server. Client-only patch: the Initial reaches the server and the handshake starts (error changes to "handshake did not complete in time"). Both ends patched:app listcompletes and returns the app table, with every datagram staying at or under the 1280 MTU.Fixes MIR-1270.