telemetry/geoprobe: retry transient bind errors in publisher AddProbe#3818
Merged
Conversation
nikw9944
force-pushed
the
nikw9944/doublezero-3765
branch
2 times, most recently
from
fb0ef8e to
9203fb1
Compare
ben-dz
approved these changes
Jun 1, 2026
nikw9944
force-pushed
the
nikw9944/doublezero-3765
branch
from
9203fb1 to
eb2419e
Compare
nikw9944
force-pushed
the
nikw9944/doublezero-3765
branch
from
eb2419e to
632798e
Compare
Lift the existing retry-on-bind-error helper from pinger.go into a package-shared retry.go and use it to wrap the per-probe UDP socket allocation in Publisher.AddProbe. The kernel can return EINVAL from bind(2) when many goroutines concurrently allocate ephemeral sockets, which intermittently fails geoprobe publisher tests in CI. Refs #3765
nikw9944
force-pushed
the
nikw9944/doublezero-3765
branch
from
632798e to
cba3ae8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
Publisher.AddProbewith a retry-on-bind-error helper, mirroring the existing mitigation inPinger. The kernel can returnEINVALfrombind(2)when many goroutines concurrently allocate ephemeral sockets, which intermittently failsTestPublisher_RemoveProbe/TestPublisher_AddProbein CI withbind: invalid argument.isBindErrorand the retry loop out ofpinger.gointo a small package-sharedretry.go, exposing a genericretryOnBindError[T]helper. The pinger path now calls the same helper, removing duplicated logic.netns.RunInNamespaceclosure so all attempts and backoffs stay on the same OS-thread-locked netns whenManagementNamespaceis set.Testing Verification
retry_test.gocovers the four important branches of the helper: success after a transient bind failure, fail-fast on a non-bind error, give-up aftersenderRetriesattempts, and prompt return on a cancelled context.go test -race -count=5 -run 'TestRetryOnBindError|TestIsBindError|TestPublisher_AddProbe|TestPublisher_RemoveProbe' ./controlplane/telemetry/internal/geoprobe/...passes locally.go test ./controlplane/telemetry/internal/geoprobe/...passes.golangci-lint run ./controlplane/telemetry/internal/geoprobe/...reports0 issues.