telemetry: count and correctly report samples dropped when account is full - #4145
Open
elitegreg wants to merge 1 commit into
Open
telemetry: count and correctly report samples dropped when account is full#4145elitegreg wants to merge 1 commit into
elitegreg wants to merge 1 commit into
Conversation
elitegreg
force-pushed
the
gm/telemetry-account-full-metric
branch
from
August 3, 2026 15:33
5c5438a to
f6bded5
Compare
elitegreg
force-pushed
the
gm/telemetry-drop-samples-metric
branch
from
August 3, 2026 15:34
9f60fe6 to
03ff35e
Compare
elitegreg
force-pushed
the
gm/telemetry-account-full-metric
branch
from
August 3, 2026 15:35
f6bded5 to
282d4ad
Compare
elitegreg
marked this pull request as ready for review
August 3, 2026 15:39
This was referenced Aug 3, 2026
elitegreg
force-pushed
the
gm/telemetry-account-full-metric
branch
from
August 3, 2026 16:55
282d4ad to
a51bf2a
Compare
elitegreg
added a commit
that referenced
this pull request
Aug 3, 2026
Resolves: #4128 Independent of #4144 and #4145 (different file), so this one branches from `main`. ## Summary of Changes - `ledgerPeerDiscovery.refresh` no longer empties the peer cache before doing work that can fail. It cleared `p.peers` under the lock and then called `LocalNet.Interfaces()`, so a transient failure there returned with zero peers and `Pinger.Tick` iterated an empty slice, probing nothing until a later refresh succeeded. - The cache is now replaced only once the new list is built, and the lock covers just that assignment rather than the whole build. The clear was redundant with the existing assignment at the end of the happy path. - Success path is unchanged. ## Diff Breakdown | Category | Files | Lines (+/-) | Net | |------------|-------|-------------|------| | Tests | 1 | +79 / -0 | +79 | | Core logic | 1 | +6 / -5 | +1 | | Docs | 1 | +3 / -0 | +3 | | **Total** | 3 | +88 / -5 | +83 | A one-line behavioral fix plus the regression test that pins it. <details> <summary>Key files (click to expand)</summary> - [`controlplane/telemetry/internal/telemetry/peers.go`](https://github.com/malbeclabs/doublezero/pull/4146/files#diff-9c369dff3cb79259b8bc34d8d952b923103baeef1515402614c23a997a06c286) — drops the `p.peers = make(...)` clear, moves the mutex to wrap only `p.peers = peers`, and leaves a comment that nothing in the build may clear the cache </details> ## Testing Verification - New test lets the first refresh discover a peer, then fails every subsequent `LocalNet.Interfaces()` call, and asserts `GetPeers()` still returns the fully populated peer (link, device, tunnel, TWAMP port) after at least three failed refreshes. It fails against the pre-fix code, which returns an empty list. - Existing peer discovery tests pass unchanged, covering the success path and the skip cases. - Package passes under `-race`, since the change moves what the mutex covers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #4127
Stacked on #4144 (
gm/telemetry-drop-samples-metric), which adds the sharedsamples_dropped_totalcounter this reuses. Base is that branch, so the diff shown is just this change; merge #4144 first.Summary of Changes
SubmitSamplesreturnsnilonErrSamplesAccountFullsoTickcounts it as a successful submission, which means data loss previously left nothing but a warning. It now incrementsdoublezero_device_telemetry_agent_samples_dropped_total{reason="account_full"}andsubmitter_account_fullon the errors counter.len(samples), the whole flushed partition, rather than what was actually lost, overstating the loss whenever a partition spans more than one batch (a 300-sample partition that filled the account on its second batch loggeddroppedSamples=300when 61 samples were lost).handleAccountFull.Note on the dropped count
The issue proposed counting
len(batch). That undercounts for the same reasonlen(samples)overcounts: when the account fills mid-partition, the batches behind the failing one are never attempted and are discarded along with it, andBuffer.Removealso throws away whatever the collector buffered since the flush.handleAccountFulltherefore countslen(samples)-i(the failing batch plus everything after it) and adds the partition's buffered length, read before the removal. The log reports the two separately asunsubmittedSamplesandbufferedSamples.Diff Breakdown
Mostly tests; the fix itself is one extracted helper and two call sites.
Key files (click to expand)
controlplane/telemetry/internal/telemetry/submitter.go— newhandleAccountFullrecords the error and drop counters, logs the unsubmitted and buffered counts, and removes the partition; bothErrSamplesAccountFullbranches inSubmitSamplesnow call it withlen(samples)-icontrolplane/telemetry/internal/metrics/metrics.go— adds theaccount_fulldrop reason and thesubmitter_account_fullerror typeTesting Verification
Buffer.Remove) and asserts the drop counter rises by 3 for a 2-sample flush, the error counter by 1, and the log carriesunsubmittedSamples=2 bufferedSamples=1.unsubmittedSamples=300. Against the pre-fix submitter this test sees the olddroppedSamples=300, confirming the miscount.drops_samples_if_account_full,initializes_then_drops_samples_if_account_full) still pass unchanged.