Skip to content

fix(nfsproxy): deflake TestRoundTrip EADDRINUSE - #2987

Merged
ValentaTomas merged 1 commit into
mainfrom
fix-nfsproxy-roundtrip-flake
Jun 11, 2026
Merged

fix(nfsproxy): deflake TestRoundTrip EADDRINUSE#2987
ValentaTomas merged 1 commit into
mainfrom
fix-nfsproxy-roundtrip-flake

Conversation

@ValentaTomas

Copy link
Copy Markdown
Member

Summary

TestRoundTrip in pkg/nfsproxy fails intermittently with bind: address already in use (e.g. on PR #2985). The go-nfs-client library binds a random ephemeral local port when dialing and only retries EADDRINUSE collisions for privileged dials, so non-privileged dials flake on port collisions. Retry the dials in the test on EADDRINUSE.

@cla-bot cla-bot Bot added the cla-signed label Jun 11, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a dialRetry helper function in proxy_test.go to retry dialing when encountering syscall.EADDRINUSE errors, resolving potential ephemeral port collisions during NFS client tests. No issues or critical findings were identified, and there is no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
2793 3 2790 7
View the full list of 3 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSandboxListPaginationRunningLargerLimit

Flake rate in main: 37.67% (Passed 1034 times, Failed 625 times)

Stack Traces | 97.7s run time
=== RUN   TestSandboxListPaginationRunningLargerLimit
    sandbox_list_test.go:327: Created sandbox 1/12: ioai2fm17e18w8whx1axm
    sandbox_list_test.go:327: Created sandbox 2/12: igpvku6qp3x1am309elbr
    sandbox_list_test.go:327: Created sandbox 3/12: ix3ap8sacuyrbgvypen1w
    sandbox_list_test.go:327: Created sandbox 4/12: igg39lxhgc7s8x8tlfcun
    sandbox_list_test.go:327: Created sandbox 5/12: ib6k0hwi9f09yv8qs6jfq
    sandbox_list_test.go:327: Created sandbox 6/12: io0omwlbe9obnx3gx37rm
    sandbox_list_test.go:327: Created sandbox 7/12: i0vyu7b9n3tm3w7wskblb
    sandbox_list_test.go:327: Created sandbox 8/12: il76gdx1x60a8jq59dyen
    sandbox_list_test.go:327: Created sandbox 9/12: i61v4mtf5x0k1j7lzx819
    sandbox_list_test.go:327: Created sandbox 10/12: izvqglsnk582c88i7xpkp
    sandbox_list_test.go:327: Created sandbox 11/12: ijnh1g9ly508imzwvizpw
    sandbox_list_test.go:327: Created sandbox 12/12: i40cn7k18yg357ld2ax3b
    sandbox_list_test.go:330: 
        	Error Trace:	.../api/sandboxes/sandbox_list_test.go:340
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	"[]" should have 12 item(s), but has 0
    sandbox_list_test.go:330: 
        	Error Trace:	.../api/sandboxes/sandbox_list_test.go:330
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxListPaginationRunningLargerLimit
--- FAIL: TestSandboxListPaginationRunningLargerLimit (97.69s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 52.45% (Passed 1027 times, Failed 1133 times)

Stack Traces | 63.4s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (63.35s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 52.45% (Passed 1017 times, Failed 1122 times)

Stack Traces | 194s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1259}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 188 MB\nFree memory before tmpfs mount: 796 MB\nMemory to use in integrity test (60% of free, min 64MB): 477 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"477+0 records in\n477+0 records out\n500170752 bytes (500 MB, 477 MiB) copied, 2.20782 s, 227 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=477\"\n\tUser time (seconds): 0.00\n\tSystem time (seconds): 2.19\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:02.21\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2652\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor (reclaiming a frame) page faults: 341\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 8\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 668 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox iqtyhpgutkgtrmjqwj5j4
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1275}}
Executing command bash in sandbox ikj0hxq2ty5nofm103z0e (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"14f6b201f78eb330b9a4d773f41b0269d38c8e413f377eccd94f381aae515898\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox iqtyhpgutkgtrmjqwj5j4
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1278}}
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
Executing command bash in sandbox iqtyhpgutkgtrmjqwj5j4 (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox iqtyhpgutkgtrmjqwj5j4: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (194.29s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@ValentaTomas
ValentaTomas marked this pull request as ready for review June 11, 2026 20:12
@cursor

cursor Bot commented Jun 11, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Test-only change with no production behavior impact.

Overview
TestRoundTrip was failing intermittently when the NFS client library picked an ephemeral local port that was already in use, because non-privileged dials are not retried on that error. The test now retries portmap and NFS TCP dials up to five times when the error is EADDRINUSE, so parallel CI runs are less likely to flake on bind collisions.

Reviewed by Cursor Bugbot for commit 6dd1f2a. Bugbot is set up for automated code reviews on this repo. Configure here.

@ValentaTomas
ValentaTomas enabled auto-merge (squash) June 11, 2026 20:12
@ValentaTomas
ValentaTomas merged commit 55f4d18 into main Jun 11, 2026
53 checks passed
@ValentaTomas
ValentaTomas deleted the fix-nfsproxy-roundtrip-flake branch June 11, 2026 20:33
Piwriw pushed a commit to Piwriw/infra that referenced this pull request Jun 12, 2026
## Summary

`TestRoundTrip` in `pkg/nfsproxy` fails intermittently with `bind:
address already in use` (e.g. on [PR
e2b-dev#2985](https://github.com/e2b-dev/infra/actions/runs/27373170636/job/80890489083)).
The `go-nfs-client` library binds a random ephemeral local port when
dialing and only retries `EADDRINUSE` collisions for privileged dials,
so non-privileged dials flake on port collisions. Retry the dials in the
test on `EADDRINUSE`.
charlie-e2b added a commit that referenced this pull request Jul 30, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([657559e](657559e))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([411b63e](411b63e))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([f8c7b5b](f8c7b5b))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([b22e820](b22e820))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([c684bd2](c684bd2))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([776ba39](776ba39))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([fda5e45](fda5e45))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([e58af28](e58af28))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P->compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([728bba3](728bba3))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([7167818](7167818))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([586ad74](586ad74))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([bfdbb24](bfdbb24))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([f551118](f551118))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([4be33ba](4be33ba))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([1abece1](1abece1))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([ea94196](ea94196))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([5385594](5385594))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([4bd42d2](4bd42d2))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([d56e0a8](d56e0a8))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([dfa9764](dfa9764))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P-&gt;compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([8694d08](8694d08))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([73399b3](73399b3))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([9be382f](9be382f))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([f828d12](f828d12))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants