fix(personhog): move harness ports out of the ephemeral range - #71099
Merged
Conversation
The harness's fixed listen ports (51xxx) sat inside Linux's ephemeral range (32768-60999), where the kernel allocates the local port of every outbound connection. During bring-up all eight services open Kafka, PG, and etcd connections in the same instant the listeners bind, so a listener occasionally lost its port to someone's outbound socket and died with EADDRINUSE — observed in CI on the very first gate step, passing on retry. The whole range moves to 24xxx, below the ephemeral floor, where the collision is structurally impossible.
A service crash produced an unreadable dump: the harness forced RUST_BACKTRACE=1 into every spawned service, so a panic printed fifty frames of runtime internals that pushed the panic message itself out of the 30-line log tail the failure report embeds — the one line naming the cause was the one line missing. Spawned services now run without backtraces (the panic message is the signal) and with NO_COLOR, and log_tail strips any remaining ANSI escapes, which previously rendered as literal escape-code soup when re-emitted through the harness's own logging. The gate CI steps also drop RUST_BACKTRACE so the harness's own anyhow errors, which already carry full context, stop appending 35-frame tokio backtraces.
Contributor
|
Reviews (1): Last reviewed commit: "fix(personhog): make harness failure rep..." | Re-trigger Greptile |
jose-sequeira
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The e2e harness's fixed listen ports (51051-51160) sit inside Linux's ephemeral port range (default 32768-60999), where the kernel allocates the local port of every outbound connection. During stack bring-up, eight services open Kafka, Postgres, and etcd connections in the same instant the listeners bind, so a listener occasionally loses its port to someone's outbound socket and dies with EADDRINUSE. This is a pure lottery that has now failed the gate on unrelated PRs (observed twice in CI: a leader gRPC port and a leader metrics port), always passing on retry.
The failure reports for those crashes were also nearly useless. The harness forced
RUST_BACKTRACE=1into every spawned service, so the panic printed ~50 frames of runtime internals that pushed the panic message itself out of the 30-line log tail the report embeds. The one line naming the cause was the one line missing, and what did survive was wrapped in raw ANSI escape codes rendered as literal\x1b[2msoup.Changes
RUST_BACKTRACE(a panic prints just its message, which fits the log tail) and withNO_COLOR=1.log_tailadditionally strips ANSI escape sequences as a guarantee for anything that ignoresNO_COLOR.RUST_BACKTRACE: 1env. The harness's own errors are context-rich anyhow chains, and the 35-frame tokio backtrace appended to each one carried no signal. The workspacecargo teststep keeps its backtrace setting, where it is actually useful.How did you test this code?
All automated, no manual testing beyond running the harness locally:
cargo test -p personhog-test-harnesspasses (10 tests). One new unit test covers the ANSI stripper, since a broken stripper would silently degrade every future failure report rather than fail any run.NO_COLORat the source.-D warningsand fmt are clean; the workflow YAML parses.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Built with Claude Code. The port collision was diagnosed from two CI failures with the same shape (EADDRINUSE at bring-up on the very first gate step, so no leftover-process explanation was possible, and green on retry). The readability problem was found while diagnosing the second failure: the embedded log tail showed backtrace frames 44 through 53 but not the panic message, which turned out to be the harness's own doing via the forced
RUST_BACKTRACE=1. An alternative considered for the ports was reserving them viaip_local_reserved_portsin CI, rejected as runner-specific plumbing when moving the range fixes it everywhere, including local runs.