fix(envd): include suppressed count in exporter error logs - #2680
Conversation
Joe's review on #2676: when the exporter rate-limits a repeated error kind, also report how many were swallowed since the last emitted log.
PR SummaryMedium Risk Overview Exporter error logging is now rate-limited via a shared helper that appends a suppressed-count, which can mask high-frequency failures for up to a minute (only reporting the aggregate count when logging resumes). Reviewed by Cursor Bugbot for commit 08ae062. Bugbot is set up for automated code reviews on this repo. Configure here. |
❌ 6 Tests Failed:
View the full list of 8 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Same rate-limited reporting pattern as the JSON/send errors.
There was a problem hiding this comment.
Code Review
The log exporter's start function contains a goroutine leak because the loop over the triggers channel does not check for context cancellation. Additionally, the sendInstanceLogs function fails to return errors for non-2xx HTTP status codes, which prevents server-side failures from being tracked by the new suppression counters.
Generalizes the JSON-err / send-err suppression and reuses it for the oversized-line drop in Write(). Lock-free via CAS; may slightly miscount under contention but that's acceptable for a noise cap.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d59fb9734
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Addresses gemini review: avoids the goroutine leak when ctx is cancelled and makes server-side collector failures (auth, rate limit, 5xx) flow through the send-error suppression instead of being silently dropped.
Switches the lock-free wall-clock impl to time.Since + a small mutex so NTP/VM-resume backward jumps don't extend the suppression window.
Use atomic.Pointer[time.Time] so time.Since keeps the monotonic reading and a wall-clock jump can't extend the suppression window.
🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-29) ### Features * **envd:** add --no-cgroups flag to disable cgroup management ([#2811](#2811)) ([e10814c](e10814c)) * **envd:** add optional EntryInfo to watch FilesystemEvent ([#2930](#2930)) ([bbbc7c8](bbbc7c8)) * **envd:** allow opting into watching network mounts ([#2982](#2982)) ([9799dd0](9799dd0)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([f8c7b5b](f8c7b5b)) ### Bug Fixes * added envd to artifact repository ([#3432](#3432)) ([6c4f0e2](6c4f0e2)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** avoid Start deadlock after request cancellation ([#3256](#3256)) ([04317f8](04317f8)) * **envd:** bound the in-memory logs queue ([#2676](#2676)) ([05c9939](05c9939)) * **envd:** discard output when no subscriber is connected ([#2639](#2639)) ([8cf1795](8cf1795)) * **envd:** fall back to lazy unmount when forced NFS umount fails ([#2683](#2683)) ([5346a0d](5346a0d)) * **envd:** ignore closed pty read errors ([#2769](#2769)) ([6118672](6118672)) * **envd:** include suppressed count in exporter error logs ([#2680](#2680)) ([35c1141](35c1141)) * **envd:** make /init lock ctx-aware to prevent retry pile-up ([#2702](#2702)) ([173afd4](173afd4)) * **envd:** make CA install lock ctx-aware ([#2690](#2690)) ([83ee89f](83ee89f)) * **envd:** replace env vars in /init instead of merging ([#2706](#2706)) ([1b52e9a](1b52e9a)) * **envd:** replace time.Sleep with ticker in ScanAndBroadcast for prompt shutdown ([#3374](#3374)) ([002fd9f](002fd9f)) * **envd:** self-heal MMDS routing on /init lookup failure ([#2701](#2701)) ([90944d5](90944d5)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **envd:** stop misleading CA install cancel errors on rapid /init ([#3206](#3206)) ([91d09e4](91d09e4)) * **envd:** suppress repeat MMDS poll failures ([#2678](#2678)) ([73d691a](73d691a)) * **envd:** tolerate busy tmpfs cleanup in tests ([#2938](#2938)) ([a485834](a485834)) * **envd:** use constant-time comparison for signature validation ([#3145](#3145)) ([fcf92fa](fcf92fa)) * **envd:** use WithoutCancel for CA cleanup goroutine ctx ([#3207](#3207)) ([ee7bf84](ee7bf84)) ### Performance Improvements * **envd:** stop logging streamed payload content ([#2755](#2755)) ([db3868c](db3868c)) * **sandbox:** keep envd logging out of journald ([#2675](#2675)) ([f6943ca](f6943ca)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-29) ### Features * **envd:** add --no-cgroups flag to disable cgroup management ([#2811](#2811)) ([e10814c](e10814c)) * **envd:** add optional EntryInfo to watch FilesystemEvent ([#2930](#2930)) ([bbbc7c8](bbbc7c8)) * **envd:** allow opting into watching network mounts ([#2982](#2982)) ([9799dd0](9799dd0)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([1abece1](1abece1)) ### Bug Fixes * added envd to artifact repository ([#3432](#3432)) ([b7024ba](b7024ba)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** avoid Start deadlock after request cancellation ([#3256](#3256)) ([04317f8](04317f8)) * **envd:** bound the in-memory logs queue ([#2676](#2676)) ([05c9939](05c9939)) * **envd:** discard output when no subscriber is connected ([#2639](#2639)) ([8cf1795](8cf1795)) * **envd:** fall back to lazy unmount when forced NFS umount fails ([#2683](#2683)) ([5346a0d](5346a0d)) * **envd:** ignore closed pty read errors ([#2769](#2769)) ([6118672](6118672)) * **envd:** include suppressed count in exporter error logs ([#2680](#2680)) ([35c1141](35c1141)) * **envd:** make /init lock ctx-aware to prevent retry pile-up ([#2702](#2702)) ([173afd4](173afd4)) * **envd:** make CA install lock ctx-aware ([#2690](#2690)) ([83ee89f](83ee89f)) * **envd:** replace env vars in /init instead of merging ([#2706](#2706)) ([1b52e9a](1b52e9a)) * **envd:** replace time.Sleep with ticker in ScanAndBroadcast for prompt shutdown ([#3374](#3374)) ([002fd9f](002fd9f)) * **envd:** self-heal MMDS routing on /init lookup failure ([#2701](#2701)) ([90944d5](90944d5)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **envd:** stop misleading CA install cancel errors on rapid /init ([#3206](#3206)) ([91d09e4](91d09e4)) * **envd:** suppress repeat MMDS poll failures ([#2678](#2678)) ([73d691a](73d691a)) * **envd:** tolerate busy tmpfs cleanup in tests ([#2938](#2938)) ([a485834](a485834)) * **envd:** use constant-time comparison for signature validation ([#3145](#3145)) ([fcf92fa](fcf92fa)) * **envd:** use WithoutCancel for CA cleanup goroutine ctx ([#3207](#3207)) ([ee7bf84](ee7bf84)) ### Performance Improvements * **envd:** stop logging streamed payload content ([#2755](#2755)) ([db3868c](db3868c)) * **sandbox:** keep envd logging out of journald ([#2675](#2675)) ([f6943ca](f6943ca)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Follow-up to #2676 addressing @djeebus' review: when the exporter rate-limits a repeated error kind, also report how many similar errors were swallowed since the last emitted log line.
Bumps envd to 0.5.23.