Skip to content

fix(envd): include suppressed count in exporter error logs - #2680

Merged
ValentaTomas merged 7 commits into
mainfrom
fix/envd-suppressed-count
May 17, 2026
Merged

fix(envd): include suppressed count in exporter error logs#2680
ValentaTomas merged 7 commits into
mainfrom
fix/envd-suppressed-count

Conversation

@ValentaTomas

Copy link
Copy Markdown
Member

Follow-up to #2676 addressing @djeebus' review: when the exporter rate-limits a repeated error kind, also report how many similar errors were swallowed since the last emitted log line.

Bumps envd to 0.5.23.

Joe's review on #2676: when the exporter rate-limits a repeated error
kind, also report how many were swallowed since the last emitted log.
@cla-bot cla-bot Bot added the cla-signed label May 17, 2026
@cursor

cursor Bot commented May 17, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes error-handling behavior in the log exporter by rate-limiting logs and by failing on non-2xx HTTP responses, which could affect observability and log delivery if the collector returns unexpected statuses.

Overview
Non-2xx responses from the logs collector are now treated as send failures, which may surface as new errors and could cause repeated retries without successfully draining the buffer if the collector returns redirects/4xx/5xx.

Exporter error logging is now rate-limited via a shared helper that appends a suppressed-count, which can mask high-frequency failures for up to a minute (only reporting the aggregate count when logging resumes).

Reviewed by Cursor Bugbot for commit 08ae062. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented May 17, 2026

Copy link
Copy Markdown

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
2622 6 2616 5
View the full list of 8 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/metrics::TestTeamMetrics

Flake rate in main: 71.24% (Passed 218 times, Failed 540 times)

Stack Traces | 0.87s run time
=== RUN   TestTeamMetrics
=== PAUSE TestTeamMetrics
=== CONT  TestTeamMetrics
    team_metrics_test.go:61: 
        	Error Trace:	.../api/metrics/team_metrics_test.go:61
        	Error:      	Should be true
        	Test:       	TestTeamMetrics
        	Messages:   	MaxConcurrentSandboxes should be >= 0
--- FAIL: TestTeamMetrics (0.87s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestUpdateNetworkConfig

Flake rate in main: 76.59% (Passed 228 times, Failed 746 times)

Stack Traces | 217s run time
=== RUN   TestUpdateNetworkConfig
=== PAUSE TestUpdateNetworkConfig
=== CONT  TestUpdateNetworkConfig
--- FAIL: TestUpdateNetworkConfig (216.75s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false

Flake rate in main: 77.14% (Passed 219 times, Failed 739 times)

Stack Traces | 7.35s run time
=== RUN   TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false
Executing command curl in sandbox imfbk56uz8mbrjtv06k4y
    sandbox_network_update_test.go:372: Command [curl] output: event:{start:{pid:1367}}
    sandbox_network_update_test.go:372: Command [curl] output: event:{end:{exit_code:35 exited:true status:"exit status 35" error:"exit status 35"}}
Executing command curl in sandbox imfbk56uz8mbrjtv06k4y
    sandbox_network_update_test.go:372: Command [curl] output: event:{start:{pid:1368}}
    sandbox_network_update_test.go:372: Command [curl] output: event:{end:{exit_code:35 exited:true status:"exit status 35" error:"exit status 35"}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{start:{pid:1369}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{data:{stdout:"HTTP/2 302 \r\nx-content-type-options: nosniff\r\nlocation: https://dns.google/\r\ndate: Sun, 17 May 2026 03:39:10 GMT\r\ncontent-type: text/html; charset=UTF-8\r\nserver: HTTP server (unknown)\r\ncontent-length: 216\r\nx-xss-protection: 0\r\nx-frame-options: SAMEORIGIN\r\nalt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000\r\n\r\n"}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_network_update_test.go:391: Command [curl] completed successfully in sandbox imfbk56uz8mbrjtv06k4y
    sandbox_network_update_test.go:391: 
        	Error Trace:	.../api/sandboxes/sandbox_network_out_test.go:74
        	            				.../api/sandboxes/sandbox_network_update_test.go:60
        	            				.../api/sandboxes/sandbox_network_update_test.go:391
        	Error:      	An error is expected but got nil.
        	Test:       	TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false
        	Messages:   	https://8.8.8.8 should be blocked
--- FAIL: TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false (7.35s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost

Flake rate in main: 56.05% (Passed 385 times, Failed 491 times)

Stack Traces | 0s run time
=== RUN   TestBindLocalhost
=== PAUSE TestBindLocalhost
=== CONT  TestBindLocalhost
--- FAIL: TestBindLocalhost (0.00s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_::1

Flake rate in main: 64.12% (Passed 216 times, Failed 386 times)

Stack Traces | 7.3s run time
=== RUN   TestBindLocalhost/bind_::1
=== PAUSE TestBindLocalhost/bind_::1
=== CONT  TestBindLocalhost/bind_::1
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1264}}
Executing command python in sandbox il7qibgqr0aqm7h85jins
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_::1
        	Messages:   	Unexpected status code 502 for bind address ::1
--- FAIL: TestBindLocalhost/bind_::1 (7.30s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_localhost

Flake rate in main: 64.00% (Passed 216 times, Failed 384 times)

Stack Traces | 7.58s run time
=== RUN   TestBindLocalhost/bind_localhost
=== PAUSE TestBindLocalhost/bind_localhost
=== CONT  TestBindLocalhost/bind_localhost
Executing command python in sandbox iteu7jckx39mzji4vl1b9
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1255}}
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_localhost
        	Messages:   	Unexpected status code 502 for bind address localhost
--- FAIL: TestBindLocalhost/bind_localhost (7.58s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 65.96% (Passed 226 times, Failed 438 times)

Stack Traces | 75.7s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:26: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (75.72s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 66.67% (Passed 216 times, Failed 432 times)

Stack Traces | 41.6s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1257}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 185 MB\nFree memory before tmpfs mount: 799 MB\nMemory to use in integrity test (80% of free, min 64MB): 639 MB\n"}}
Executing command bash in sandbox ip0maiycxdv8pkdd2fmr6 (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"639+0 records in\n639+0 records out\n670040064 bytes (670 MB, 639 MiB) copied, 10.9122 s, 61.4 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" if=/dev/urandom of=/mnt/testfile bs=1M count=639\"\n\tUser time (seconds): 0.00\n\tSystem time (seconds): 10.66\n\tPercent of CPU this job got: 97%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:10.92\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2700\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 3\n\tMinor (reclaiming a frame) page faults: 346\n\tVoluntary context switches: 4\n\tInvoluntary context switches: 127\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 832 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox iutd6q7ojghtl2hathh7y
Executing command bash in sandbox iutd6q7ojghtl2hathh7y (user: root)
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{start:{pid:1274}}
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{data:{stdout:"c110aa29f7b8f10d893679a2917c05cc9be4d0497154a4dd4af415ee1538eb84\n"}}
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:74: Command [bash] completed successfully in sandbox iutd6q7ojghtl2hathh7y
Executing command bash in sandbox iutd6q7ojghtl2hathh7y (user: root)
    sandbox_memory_integrity_test.go:99: Command [bash] output: event:{start:{pid:1277}}
    sandbox_memory_integrity_test.go:100: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:100
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox iutd6q7ojghtl2hathh7y: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (41.58s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Same rate-limited reporting pattern as the JSON/send errors.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The log exporter's start function contains a goroutine leak because the loop over the triggers channel does not check for context cancellation. Additionally, the sendInstanceLogs function fails to return errors for non-2xx HTTP status codes, which prevents server-side failures from being tracked by the new suppression counters.

Comment thread packages/envd/internal/logs/exporter/exporter.go Outdated
Comment thread packages/envd/internal/logs/exporter/exporter.go Outdated
Generalizes the JSON-err / send-err suppression and reuses it for the
oversized-line drop in Write(). Lock-free via CAS; may slightly miscount
under contention but that's acceptable for a noise cap.
@ValentaTomas
ValentaTomas marked this pull request as ready for review May 17, 2026 03:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d59fb9734

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/envd/internal/logs/exporter/rate_limited_logger.go Outdated
Addresses gemini review: avoids the goroutine leak when ctx is cancelled
and makes server-side collector failures (auth, rate limit, 5xx) flow
through the send-error suppression instead of being silently dropped.
Switches the lock-free wall-clock impl to time.Since + a small mutex so
NTP/VM-resume backward jumps don't extend the suppression window.
Use atomic.Pointer[time.Time] so time.Since keeps the monotonic reading
and a wall-clock jump can't extend the suppression window.
@ValentaTomas
ValentaTomas enabled auto-merge (squash) May 17, 2026 03:27
@ValentaTomas
ValentaTomas merged commit 35c1141 into main May 17, 2026
53 checks passed
@ValentaTomas
ValentaTomas deleted the fix/envd-suppressed-count branch May 17, 2026 03:48
charlie-e2b pushed a commit that referenced this pull request Jul 29, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-29)


### Features

* **envd:** add --no-cgroups flag to disable cgroup management
([#2811](#2811))
([e10814c](e10814c))
* **envd:** add optional EntryInfo to watch FilesystemEvent
([#2930](#2930))
([bbbc7c8](bbbc7c8))
* **envd:** allow opting into watching network mounts
([#2982](#2982))
([9799dd0](9799dd0))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([f8c7b5b](f8c7b5b))


### Bug Fixes

* added envd to artifact repository
([#3432](#3432))
([6c4f0e2](6c4f0e2))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** avoid Start deadlock after request cancellation
([#3256](#3256))
([04317f8](04317f8))
* **envd:** bound the in-memory logs queue
([#2676](#2676))
([05c9939](05c9939))
* **envd:** discard output when no subscriber is connected
([#2639](#2639))
([8cf1795](8cf1795))
* **envd:** fall back to lazy unmount when forced NFS umount fails
([#2683](#2683))
([5346a0d](5346a0d))
* **envd:** ignore closed pty read errors
([#2769](#2769))
([6118672](6118672))
* **envd:** include suppressed count in exporter error logs
([#2680](#2680))
([35c1141](35c1141))
* **envd:** make /init lock ctx-aware to prevent retry pile-up
([#2702](#2702))
([173afd4](173afd4))
* **envd:** make CA install lock ctx-aware
([#2690](#2690))
([83ee89f](83ee89f))
* **envd:** replace env vars in /init instead of merging
([#2706](#2706))
([1b52e9a](1b52e9a))
* **envd:** replace time.Sleep with ticker in ScanAndBroadcast for
prompt shutdown ([#3374](#3374))
([002fd9f](002fd9f))
* **envd:** self-heal MMDS routing on /init lookup failure
([#2701](#2701))
([90944d5](90944d5))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **envd:** stop misleading CA install cancel errors on rapid /init
([#3206](#3206))
([91d09e4](91d09e4))
* **envd:** suppress repeat MMDS poll failures
([#2678](#2678))
([73d691a](73d691a))
* **envd:** tolerate busy tmpfs cleanup in tests
([#2938](#2938))
([a485834](a485834))
* **envd:** use constant-time comparison for signature validation
([#3145](#3145))
([fcf92fa](fcf92fa))
* **envd:** use WithoutCancel for CA cleanup goroutine ctx
([#3207](#3207))
([ee7bf84](ee7bf84))


### Performance Improvements

* **envd:** stop logging streamed payload content
([#2755](#2755))
([db3868c](db3868c))
* **sandbox:** keep envd logging out of journald
([#2675](#2675))
([f6943ca](f6943ca))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-29)


### Features

* **envd:** add --no-cgroups flag to disable cgroup management
([#2811](#2811))
([e10814c](e10814c))
* **envd:** add optional EntryInfo to watch FilesystemEvent
([#2930](#2930))
([bbbc7c8](bbbc7c8))
* **envd:** allow opting into watching network mounts
([#2982](#2982))
([9799dd0](9799dd0))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([1abece1](1abece1))


### Bug Fixes

* added envd to artifact repository
([#3432](#3432))
([b7024ba](b7024ba))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** avoid Start deadlock after request cancellation
([#3256](#3256))
([04317f8](04317f8))
* **envd:** bound the in-memory logs queue
([#2676](#2676))
([05c9939](05c9939))
* **envd:** discard output when no subscriber is connected
([#2639](#2639))
([8cf1795](8cf1795))
* **envd:** fall back to lazy unmount when forced NFS umount fails
([#2683](#2683))
([5346a0d](5346a0d))
* **envd:** ignore closed pty read errors
([#2769](#2769))
([6118672](6118672))
* **envd:** include suppressed count in exporter error logs
([#2680](#2680))
([35c1141](35c1141))
* **envd:** make /init lock ctx-aware to prevent retry pile-up
([#2702](#2702))
([173afd4](173afd4))
* **envd:** make CA install lock ctx-aware
([#2690](#2690))
([83ee89f](83ee89f))
* **envd:** replace env vars in /init instead of merging
([#2706](#2706))
([1b52e9a](1b52e9a))
* **envd:** replace time.Sleep with ticker in ScanAndBroadcast for
prompt shutdown ([#3374](#3374))
([002fd9f](002fd9f))
* **envd:** self-heal MMDS routing on /init lookup failure
([#2701](#2701))
([90944d5](90944d5))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **envd:** stop misleading CA install cancel errors on rapid /init
([#3206](#3206))
([91d09e4](91d09e4))
* **envd:** suppress repeat MMDS poll failures
([#2678](#2678))
([73d691a](73d691a))
* **envd:** tolerate busy tmpfs cleanup in tests
([#2938](#2938))
([a485834](a485834))
* **envd:** use constant-time comparison for signature validation
([#3145](#3145))
([fcf92fa](fcf92fa))
* **envd:** use WithoutCancel for CA cleanup goroutine ctx
([#3207](#3207))
([ee7bf84](ee7bf84))


### Performance Improvements

* **envd:** stop logging streamed payload content
([#2755](#2755))
([db3868c](db3868c))
* **sandbox:** keep envd logging out of journald
([#2675](#2675))
([f6943ca](f6943ca))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants