deflake several tests under flakestress (oversubscription) - #71
Merged
Conversation
bradfitz
commented
Sep 3, 2026
bradfitz
commented
Member
- tailcat: resend meow pings instead of betting the timeout on one packet
- cmd/tailcat: give SOCKS dials a longer budget than one WireGuard handshake
- cmd/tailcat: let forward pick a free local port with 0
- cmd/tailcat: improve serve test diagnostics, widen client watchdog
A client ping sent a single meow packet over DERP and then waited out its whole 10 second timeout for the ack. DERP delivery is best effort: the relay drops packets addressed to a key that isn't connected yet, both sides drop sends when a relay write queue is full, and the ack can be lost the same ways. Any one lost packet meant a guaranteed 10 second failure, seen as "tailcat Ping: context deadline exceeded" in nixpkgs' package build when their loaded test infrastructure ran TestServeExitNode. Resend the ping every second until the ack arrives or the timeout expires, and treat send failures as retryable too, since a dropped send is just packet loss happening early. The server acks every ping, so duplicates are harmless. Under a 32-way flakestress run this took TestServeExitNode's failure rate from about 1% to zero for this failure mode. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
…shake The socks5 package caps each dial at 5 seconds, which is also WireGuard's handshake retransmit interval, so a single lost handshake packet pushed the dial past its budget and failed the CONNECT with a general-failure reply. Detach from the package's deadline in our dialer and allow 15 seconds, enough for a few handshake attempts. Found by flakestress in TestServeExitNode/socks5, but it bit real clients the same way. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Accept a local port of 0 in forward mappings, meaning the operating system chooses a free port, and print each listener's bound address unconditionally (not only with --verbose) since that line is the only way to learn the picked port. TestForwardEndToEnd used to grab an ephemeral port, close it, and hope to rebind it moments later, but under a parallel flakestress run another process could take the port first, so the test dialed a stranger and failed with EOF at about a 1% rate. It now forwards from port 0 and reads the printed address instead, which also matches how ssh -L 0:host:port behaves. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Run the serve tests' servers and clients with --verbose and include both processes' stderr when a client fails or hangs, so a flake captured under stress comes with enough context to diagnose. The output is only printed on failure. Also raise runClient's watchdog from 30 to 60 seconds: it only exists to catch true hangs, and a loaded machine can drop relayed packets and leave a transfer legitimately waiting out TCP retransmit backoff. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This was referenced Sep 3, 2026
Closed
bradfitz
added a commit
that referenced
this pull request
Sep 4, 2026
ProxyConns propagates EOF in each direction with CloseWrite, then used to call Close on both connections as soon as its copy loops returned. For a gonet.TCPConn, that changes the gVisor endpoint to application-closed immediately after its FIN was queued. If the FIN is lost around that transition, the peer can wait forever for EOF. When the peer FIN has already arrived, subscribe to gVisor’s hangup event and give the local FIN up to five seconds to be acknowledged before the final Close. The wait happens entirely inside ProxyConns, is bounded if the peer disappears, and preserves its simple ownership contract: it closes both connections before returning. Together with the outstanding tailscale.com netcheck race fix (tailscale/tailscale#21086), this completed 12,203 TestServeExitNode runs over 30 minutes without a failure. The netcheck fix prevents spurious rebinds; this change independently prevents a lost teardown FIN from wedging the client. Updates #71 Updates #73 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.