telemetry: log and count samples dropped at buffer capacity - #4144
Open
elitegreg wants to merge 2 commits into
Open
telemetry: log and count samples dropped at buffer capacity#4144elitegreg wants to merge 2 commits into
elitegreg wants to merge 2 commits into
Conversation
elitegreg
marked this pull request as ready for review
August 3, 2026 15:23
elitegreg
enabled auto-merge (squash)
August 3, 2026 15:23
elitegreg
force-pushed
the
gm/telemetry-drop-samples-metric
branch
from
August 3, 2026 15:34
9f60fe6 to
03ff35e
Compare
This was referenced Aug 3, 2026
elitegreg
added a commit
that referenced
this pull request
Aug 3, 2026
Resolves: #4128 Independent of #4144 and #4145 (different file), so this one branches from `main`. ## Summary of Changes - `ledgerPeerDiscovery.refresh` no longer empties the peer cache before doing work that can fail. It cleared `p.peers` under the lock and then called `LocalNet.Interfaces()`, so a transient failure there returned with zero peers and `Pinger.Tick` iterated an empty slice, probing nothing until a later refresh succeeded. - The cache is now replaced only once the new list is built, and the lock covers just that assignment rather than the whole build. The clear was redundant with the existing assignment at the end of the happy path. - Success path is unchanged. ## Diff Breakdown | Category | Files | Lines (+/-) | Net | |------------|-------|-------------|------| | Tests | 1 | +79 / -0 | +79 | | Core logic | 1 | +6 / -5 | +1 | | Docs | 1 | +3 / -0 | +3 | | **Total** | 3 | +88 / -5 | +83 | A one-line behavioral fix plus the regression test that pins it. <details> <summary>Key files (click to expand)</summary> - [`controlplane/telemetry/internal/telemetry/peers.go`](https://github.com/malbeclabs/doublezero/pull/4146/files#diff-9c369dff3cb79259b8bc34d8d952b923103baeef1515402614c23a997a06c286) — drops the `p.peers = make(...)` clear, moves the mutex to wrap only `p.peers = peers`, and leaves a comment that nothing in the build may clear the cache </details> ## Testing Verification - New test lets the first refresh discover a peer, then fails every subsequent `LocalNet.Interfaces()` call, and asserts `GetPeers()` still returns the fully populated peer (link, device, tunnel, TWAMP port) after at least three failed refreshes. It fails against the pre-fix code, which returns an empty list. - Existing peer discovery tests pass unchanged, covering the success path and the skip cases. - Package passes under `-race`, since the change moves what the mutex covers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #4126
Summary of Changes
submitter_retries_exhausted, which fires whether the batch was requeued or destroyed.doublezero_device_telemetry_agent_samples_dropped_totalwith areasonlabel (buffer_full), so the account-full drop path (device/telemetry: account-full path reports success and logs the wrong count #4127) can report on the same series with a different reason.submitter_buffer_fullerror type on the existingdoublezero_device_telemetry_agent_errors_totalcounter, one increment per dropped batch.Diff Breakdown
Mostly tests around a nine-line observability fix; no behavior change to submission or requeueing.
Key files (click to expand)
controlplane/telemetry/internal/telemetry/submitter.go— the over-capacity branch inTicknow warns with the dropped count, buffer length, and capacity, and bumps both counters; the requeue branch is unchanged apart from being nested under the existing!successcheckcontrolplane/telemetry/internal/metrics/metrics.go— registers thesamples_dropped_totalcounter, thereasonlabel, and thesubmitter_buffer_fullerror typeTesting Verification
droppedSamples=2andcapacity=2, that the drop counter rises by the batch length, that the error counter rises once, and that nothing is requeued. Confirmed it fails against the pre-fix submitter.go test ./controlplane/telemetry/...passes exceptinternal/netns, which fails identically onmainin this environment (needs privileges).