perf(clean-nfs-cache): restore dirfd-relative statx by levb · Pull Request #2766 · e2b-dev/runtime

levb · 2026-05-20T23:12:34Z

No description provided.

PR #2256 changed statInDir from fd-relative statx (using an open directory fd) to AT_FDCWD with the full absolute path. On NFS this inflates a single GETATTR into a LOOKUP chain across every path component — 5-10x more RPCs per stat in practice. In prod this showed up as a ~2.5 min first-batch warmup and roughly the first 10 minutes of every hourly run being scan-bound instead of delete-bound. The race #2256 was working around was real: on ctx cancellation, scanDir returned early via ctx.Err(), triggering the deferred df.Close() while Statter goroutines were still calling df.Fd() and unix.Statx on the fd. After Close, the freed fd number can be reused by another goroutine in the same process (a sibling Scanner opening a new directory), so an in-flight Statter can call statx on the wrong fd. With uniform NFS-cache filenames across directories the stat may silently succeed against the wrong file. The right fix is lifetime management, not removing the dependency. This change: - restores statInDir(df *os.File, filename) using fd-relative unix.Statx(int(df.Fd()), filename, ...) on Linux, and matches the signature on macOS (perf there doesn't matter). - changes statReq to carry *os.File again. - in scanDir, counts submitted requests and always drains exactly that many responses from responseCh before returning, even on ctx cancellation. responseCh is buffered to len(filenames) so Statter sends never block, and Statter always emits a response per request it pulls (it doesn't re-check ctx mid-processing). df.Close() can therefore only run after every Statter that holds df is done. Race detector run (go test -race ./cmd/clean-nfs-cache/cleaner/...) passes; existing tests cover the cancel path.

Two cosmetic issues both visible at the end of every successful run: 1. When preRun computes that current disk usage is already at or below --disk-usage-target-percent, TargetBytesToDelete is left at 0. The main loop's first check (DeletedBytes(0) >= TargetBytesToDelete(0)) then immediately logged "target bytes deleted reached, draining remaining candidates" — implying the cleaner had done work, when in fact there was nothing to do. Now short-circuited in main.go with an explicit "disk already at or below target, nothing to do". 2. Each Scanner goroutine in flight at drain time logged "error during scanning: context canceled" before exiting cleanly, making routine shutdown look like a failure. Now suppressed: a context.Canceled / context.DeadlineExceeded from FindCandidate is treated as a shutdown signal and the Scanner returns without logging or writing to errCh. Also fixed a pre-existing typo in the scanner error log field name ("continousCount" -> "continuousCount").

codecov · 2026-05-20T23:14:15Z

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
2655	3	2652	7

View the full list of 3 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSandboxListPaused_NoMetadata

Flake rate in main: 53.58% (Passed 389 times, Failed 449 times)

Stack Traces | 13.2s run time

=== RUN   TestSandboxListPaused_NoMetadata
=== PAUSE TestSandboxListPaused_NoMetadata
=== CONT  TestSandboxListPaused_NoMetadata
    sandbox_list_test.go:225: Sandbox creation failed status=500 body={"code":500,"message":"Failed to place sandbox"}
    sandbox_list_test.go:225: Sandbox creation={Body:[123 34 99 111 100 101 34 58 53 48 48 44 34 109 101 115 115 97 103 101 34 58 34 70 97 105 108 101 100 32 116 111 32 112 108 97 99 101 32 115 97 110 100 98 111 120 34 125] HTTPResponse:0x23a02dacab0 JSON201:<nil> JSON400:<nil> JSON401:<nil> JSON500:0x23a02dbc7f8}
    sandbox_list_test.go:225: 
        	Error Trace:	.../internal/utils/sandbox.go:163
        	            				.../api/sandboxes/sandbox_list_test.go:225
        	Error:      	Not equal: 
        	            	expected: 201
        	            	actual  : 500
        	Test:       	TestSandboxListPaused_NoMetadata
--- FAIL: TestSandboxListPaused_NoMetadata (13.25s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 65.50% (Passed 394 times, Failed 748 times)

Stack Traces | 85.9s run time

=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (85.87s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 65.84% (Passed 384 times, Failed 740 times)

Stack Traces | 116s run time

=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1263}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 187 MB\nFree memory before tmpfs mount: 797 MB\nMemory to use in integrity test (80% of free, min 64MB): 637 MB\n"}}
Executing command bash in sandbox i40ssmy88n2gzmddwxmsy (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"637+0 records in\n637+0 records out\n667942912 bytes (668 MB, 637 MiB) copied, 8.89581 s, 75.1 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=637\"\n\tUser time (seconds): 0.00\n\tSystem time (seconds): 8.75\n\tPercent of CPU this job got: 98%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:08.90\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2696\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor (reclaiming a frame) page faults: 344\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 115\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 831 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox i9y2o2dh1q95ojhiiibyr
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1280}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"db2e772af500585ecfe64cf7048a7747467b5cb7e3df971adf75bacc555ca98d\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox i9y2o2dh1q95ojhiiibyr
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1283}}
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
Executing command bash in sandbox i9y2o2dh1q95ojhiiibyr (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox i9y2o2dh1q95ojhiiibyr: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (115.54s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 690e3e4073

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-05-20T23:15:03Z

+	// below disk-usage-target-percent. Short-circuit here so we don't spin
+	// up workers just to immediately drain (which also produced a misleading
+	// "target bytes deleted reached" log).
+	if opts.TargetBytesToDelete == 0 && opts.TargetFilesToDelete == 0 {


Guard early exit with disk-usage target check

This early return now bypasses Cleaner.validateOptions() for configurations where both byte/file targets are zero but disk-usage-target-percent is also zero (for example --disk-usage-target-percent=0 with no explicit delete targets), so the command exits successfully with “nothing to do” instead of surfacing the existing usage error. That turns an invalid configuration into a silent no-op and can hide misconfigured automation.

Useful? React with 👍 / 👎.

gemini-code-assist

Code Review

I have no feedback to provide.

djeebus

All the important stuff is covered by tests, and the rest seems reasonable

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **cfg:** add DISABLE_STARTUP_RECLAIM flag ([#3081](#3081)) ([7677ca6](7677ca6)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * **featureflags:** support per-service context providers ([#3100](#3100)) ([65297c1](65297c1)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **metrics:** break down pause-snapshot latency by step ([#3426](#3426)) ([657559e](657559e)) * **metrics:** label pause telemetry by fs_only ([#3425](#3425)) ([411b63e](411b63e)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orch:** add envd-version to LaunchDarkly sandbox context ([#3051](#3051)) ([37d3b92](37d3b92)) * **orch:** add less, nftables, iputils-ping, and jq to base provisioning ([#2736](#2736)) ([a1e010e](a1e010e)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** debug a sandbox guest kernel with resume-build -gdb ([#3040](#3040)) ([37bb0dc](37bb0dc)) * **orch:** decouple warm resume from memfile dedup ([#3166](#3166)) ([77f25a0](77f25a0)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([f8c7b5b](f8c7b5b)) * **orchestrator/cgroup:** list and destroy leaked sandbox cgroups ([#3086](#3086)) ([bce1d84](bce1d84)) * **orchestrator/nbd:** inspect and disconnect connected devices ([#3087](#3087)) ([4d47148](4d47148)) * **orchestrator/network:** list slot namespaces ([#3089](#3089)) ([c23dbc7](c23dbc7)) * **orchestrator/network:** list slot namespaces ([#3090](#3090)) ([fbfce25](fbfce25)) * **orchestrator:** add -force-reboot to resume-build to cold-boot memory-snaphsot builds ([#3208](#3208)) ([cf8f15b](cf8f15b)) * **orchestrator:** add allocated resource metrics for sandboxes ([#2943](#2943)) ([95cb6d3](95cb6d3)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** add NetworkAssignHook for sandbox lifecycle extensions ([#3290](#3290)) ([3261963](3261963)) * **orchestrator:** add soft-delete marker label to the check metric ([#3144](#3144)) ([1ce64f8](1ce64f8)) * **orchestrator:** add v4HeaderForUncompressed FF bit ([#2669](#2669)) ([1f459ee](1f459ee)) * **orchestrator:** always include execution metrics in sandbox webhook events ([#2852](#2852)) ([440edfe](440edfe)) * **orchestrator:** classify envd-init by exit type ([#3139](#3139)) ([1e39a4f](1e39a4f)) * **orchestrator:** graceful sandbox drain on shutdown ([#3069](#3069)) ([6ce68e3](6ce68e3)) * **orchestrator:** graceful template-build drain on shutdown ([#3079](#3079)) ([1b3001c](1b3001c)) * **orchestrator:** improved read-path telemetry ([#3063](#3063)) ([bc3fe84](bc3fe84)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orchestrator:** make build-reserved-disk-space-mb default 256MB ([#3065](#3065)) ([d473f98](d473f98)) * **orchestrator:** record upload compression metrics ([#2761](#2761)) ([9092e35](9092e35)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** run startup reclaim on boot ([#3123](#3123)) ([79b838e](79b838e)) * **orchestrator:** single-instance flock on startup ([#3143](#3143)) ([1320d6e](1320d6e)) * **orchestrator:** soft-delete consumer enforcement for storage index ([#3034](#3034)) ([fbfc918](fbfc918)) * **orchestrator:** tag envd-init meters with start_type ([#3125](#3125)) ([4466b48](4466b48)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **orchestrator:** track sandbox lifecycles ([#2998](#2998)) ([057f20c](057f20c)) * **orchestrator:** write layer sizes (logical/mapped/diff) to object metadata ([#3122](#3122)) ([11869c0](11869c0)) * **orch:** harvest resume-prefetch trace on pause ([#3067](#3067)) ([97bd4a5](97bd4a5)) * **orch:** last-cycle memory prefetch on resume ([#3258](#3258)) ([b22e820](b22e820)) * **orch:** make resume-build -gdb work on real nodes + add copy-build -gdb ([#3108](#3108)) ([c684bd2](c684bd2)) * **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP) ([#3039](#3039)) ([a98cf2c](a98cf2c)) * **orch:** per-start UFFD startup working-set metric ([#2960](#2960)) ([dc386b2](dc386b2)) * **orch:** premade NixOS base-image support ([#3412](#3412)) ([776ba39](776ba39)) * **orch:** record envd init duration histogram on failure with success attribute ([#2749](#2749)) ([afa7458](afa7458)) * **orch:** snapshot fragmentation metrics ([#2931](#2931)) ([842b007](842b007)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **shared:** add OTEL instrumentation to AWS S3 storage client ([#3172](#3172)) ([25b0fd1](25b0fd1)) * **storage:** per-role storage URLs, env-free storage library ([#3246](#3246)) ([fcbe909](fcbe909)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) * **storage:** write-through compressed templates to NFS on upload ([#2827](#2827)) ([57503c1](57503c1)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([fda5e45](fda5e45)) * **block:** rephrase misleading error message in pwritevAll ([#2816](#2816)) ([1555f1b](1555f1b)) * **cache:** use 512-byte units for stat.Blocks in FileSize ([#2949](#2949)) ([0f632a9](0f632a9)) * **clean-nfs-cache:** exclude zombies from delete_age ([#3191](#3191)) ([3fa2aeb](3fa2aeb)) * **compression:** correctness findings from compression audit ([#2803](#2803)) ([d21a6a9](d21a6a9)) * **copy-build:** resolve compression suffix for build data files ([#2859](#2859)) ([8966f7e](8966f7e)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **inspect-build:** adapt validate to new Chunker upstream API ([#2989](#2989)) ([2e0d3da](2e0d3da)) * **nbd:** adjust status poll sleep from 100ns to 100µs ([02bf51b](02bf51b)) * **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid useless busy spinning ([#2884](#2884)) ([02bf51b](02bf51b)) * **nfsproxy:** deflake TestRoundTrip EADDRINUSE ([#2987](#2987)) ([55f4d18](55f4d18)) * **orch:** denormalize upload metric file type ([#2865](#2865)) ([b1646ca](b1646ca)) * **orch:** disable the chronyd seccomp filter on Alpine when using PHC ([#3453](#3453)) ([e58af28](e58af28)) * **orchestrator:** anchor rsync CWD to root in template file copy ([#2835](#2835)) ([7160db9](7160db9)) * **orchestrator:** atomically replace metadata ([#3321](#3321)) ([0c4ad6b](0c4ad6b)) * **orchestrator:** avoid serializing upload headers twice ([#2762](#2762)) ([9b7b149](9b7b149)) * **orchestrator:** chunk readiness bug in P2P->compressed ([#3185](#3185)) ([74a6e5b](74a6e5b)) * **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto… ([#3173](#3173)) ([88ff17c](88ff17c)) * **orchestrator:** discard poisoned nftables conn on firewall errors ([#3008](#3008)) ([03f10e0](03f10e0)) * **orchestrator:** drop stale pre-init logs ([#3297](#3297)) ([8ec4be5](8ec4be5)) * **orchestrator:** emit compression ratios as fractions, not BP ([#2772](#2772)) ([866f4c1](866f4c1)) * **orchestrator:** export dirty-page stall counter from process start ([#2992](#2992)) ([badc8ad](badc8ad)) * **orchestrator:** harden Firecracker process shutdown ([#2996](#2996)) ([df662e7](df662e7)) * **orchestrator:** harden shutdown network cleanup ([#3000](#3000)) ([de2f391](de2f391)) * **orchestrator:** implement Docker COPY merge semantics in template builds ([#3283](#3283)) ([9174104](9174104)) * **orchestrator:** keep dedup empty-pages telemetry scan-only ([#2991](#2991)) ([35d0832](35d0832)) * **orchestrator:** let build-cache threshold flag raise above its fal… ([#3175](#3175)) ([06393c3](06393c3)) * **orchestrator:** log missing egress proxy in startup reclaim instead of defaulting silently ([#3116](#3116)) ([6ca3163](6ca3163)) * **orchestrator:** make copy-build handle filesystem-only snapshots ([#3299](#3299)) ([62add04](62add04)) * **orchestrator:** measure ext4 free space from block groups ([#3282](#3282)) ([f18f05f](f18f05f)) * **orchestrator:** normalize upload metric file labels ([#2767](#2767)) ([6dec8b3](6dec8b3)) * **orchestrator:** order egress config/firewall updates to close BYOP enable race ([#3313](#3313)) ([7faa59e](7faa59e)) * **orchestrator:** order envd.service after local-fs.target ([#3043](#3043)) ([ea2663e](ea2663e)) * **orchestrator:** order envd.service after systemd-tmpfiles-setup ([#3130](#3130)) ([9481811](9481811)) * **orchestrator:** pause upload retain retry ([#2993](#2993)) ([4f81799](4f81799)) * **orchestrator:** pin tap device host-side MAC address ([#3271](#3271)) ([3c786ba](3c786ba)) * **orchestrator:** pin UFFD copy source buffers ([#2745](#2745)) ([837fa91](837fa91)) * **orchestrator:** preserve full ENV value across stdout chunks ([#2740](#2740)) ([4822e6d](4822e6d)) * **orchestrator:** read V3 ancestors as uncompressed instead of failing ([#2994](#2994)) ([c479dd3](c479dd3)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * **orchestrator:** report real V4 header compression ratio ([#2771](#2771)) ([ecd344e](ecd344e)) * **orchestrator:** resolve remaining P2P/compression/V5 issues ([#3015](#3015)) ([1e4379e](1e4379e)) * **orchestrator:** sanitize OCI pull errors ([#3096](#3096)) ([a3af6c0](a3af6c0)) * **orchestrator:** scope rootfs hash to provision default ([#3129](#3129)) ([475f955](475f955)) * **orchestrator:** stop Checks health-loop leaking ([#2739](#2739)) ([17e6e60](17e6e60)) * **orchestrator:** survive SIGBUS from failing disks under mmap'd caches ([#3385](#3385)) ([728bba3](728bba3)) * **orchestrator:** tolerate missing header for legacy templates ([#3026](#3026)) ([8a44bfe](8a44bfe)) * **orch:** fall back to ID_LIKE with a warning instead of rejecting ([#3459](#3459)) ([7167818](7167818)) * **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind flag) ([#3048](#3048)) ([efd3d4d](efd3d4d)) * **orch:** split scheduling base build id per artifact ([#2920](#2920)) ([3e35a2a](3e35a2a)) * **orch:** validate copy-build -gdb buckets before the snapshot copy ([#3446](#3446)) ([586ad74](586ad74)) * **shared:** never report a failed envd command stream as success ([#3281](#3281)) ([69c06b6](69c06b6)) * **storage:** compression upload & cache correctness fixes ([#3231](#3231)) ([980748f](980748f)) * **storage:** don't assume V4+ ancestor gaps are uncompressed ([#3447](#3447)) ([bfdbb24](bfdbb24)) * **uffd:** dedupe deferred page faults ([#2864](#2864)) ([9680a41](9680a41)) * WrapContextAsUserError should not misclassify internal timeouts as user cancellations ([#3155](#3155)) ([8f83959](8f83959)) ### Performance Improvements * **build:** cache resolved Diff per BuildId within File.ReadAt ([#2838](#2838)) ([53de07f](53de07f)) * **build:** parallelize fragmented backing reads ([#2872](#2872)) ([c7655a7](c7655a7)) * **clean-nfs-cache:** restore dirfd-relative statx ([#2766](#2766)) ([6bdbedb](6bdbedb)) * **header:** add V5 columnar varint header format ([#2847](#2847)) ([9dd931b](9dd931b)) * **header:** pack cached Header.Mapping into a compact form ([#2844](#2844)) ([7f0b13c](7f0b13c)) * **orchestrator:** add memfile dedup density threshold ([#2862](#2862)) ([7ccfa02](7ccfa02)) * **orchestrator:** avoid V3-ancestor header refresh ([#2999](#2999)) ([cb6aa0b](cb6aa0b)) * **orch:** metrics for dirty page throttling ([#2858](#2858)) ([d2aa554](d2aa554)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

🤖 I have created a release *beep* *boop* --- ## 0.0.1 (2026-07-30) ### Features * **api:** add sandbox IAM workload token configuration ([13ddb3d](13ddb3d)) * **api:** add sandbox workload identity permission ([#3319](#3319)) ([13ddb3d](13ddb3d)) * **api:** SOCKS5 egress proxy on sandbox network config (BYOP) ([#2642](#2642)) ([1fc3820](1fc3820)) * **cfg:** add DISABLE_STARTUP_RECLAIM flag ([#3081](#3081)) ([7677ca6](7677ca6)) * **clickhouse:** implement multi-cluster fan-out for events and stats ([#2925](#2925)) ([39594c6](39594c6)) * dynamic sandbox log routing and ClickHouse-backed log reads ([#3236](#3236)) ([1b19a3b](1b19a3b)) * **envd:** give envd realtime IO priority, reset for user processes ([#2681](#2681)) ([f4bd1b2](f4bd1b2)) * **envd:** split collapse stats into real migrations vs already-huge ([#3021](#3021)) ([0d77614](0d77614)) * **envd:** support user-defined file metadata via xattrs ([#2732](#2732)) ([da8fbe4](da8fbe4)) * **featureflags:** support per-service context providers ([#3100](#3100)) ([65297c1](65297c1)) * freeze user cgroup across pause/resume to keep envd /init responsive ([#2688](#2688)) ([eceb741](eceb741)) * **metrics:** break down pause-snapshot latency by step ([#3426](#3426)) ([f551118](f551118)) * **metrics:** label pause telemetry by fs_only ([#3425](#3425)) ([4be33ba](4be33ba)) * **observability:** add kill_reason to sandbox.lifecycle.killed ([#2833](#2833)) ([e45418f](e45418f)) * **observability:** include kill_reason in kill-path structured logs ([#2846](#2846)) ([33c49f7](33c49f7)) * **orch:** add envd-version to LaunchDarkly sandbox context ([#3051](#3051)) ([37d3b92](37d3b92)) * **orch:** add less, nftables, iputils-ping, and jq to base provisioning ([#2736](#2736)) ([a1e010e](a1e010e)) * **orch:** collapse envd's heap into 2 MiB hugepages before pause to cut cold-resume faults ([#2997](#2997)) ([6677f73](6677f73)) * **orch:** debug a sandbox guest kernel with resume-build -gdb ([#3040](#3040)) ([37bb0dc](37bb0dc)) * **orch:** decouple warm resume from memfile dedup ([#3166](#3166)) ([77f25a0](77f25a0)) * **orch:** distro-aware template base-image provisioning ([#3411](#3411)) ([1abece1](1abece1)) * **orchestrator/cgroup:** list and destroy leaked sandbox cgroups ([#3086](#3086)) ([bce1d84](bce1d84)) * **orchestrator/nbd:** inspect and disconnect connected devices ([#3087](#3087)) ([4d47148](4d47148)) * **orchestrator/network:** list slot namespaces ([#3089](#3089)) ([c23dbc7](c23dbc7)) * **orchestrator/network:** list slot namespaces ([#3090](#3090)) ([fbfce25](fbfce25)) * **orchestrator:** add -force-reboot to resume-build to cold-boot memory-snaphsot builds ([#3208](#3208)) ([cf8f15b](cf8f15b)) * **orchestrator:** add allocated resource metrics for sandboxes ([#2943](#2943)) ([95cb6d3](95cb6d3)) * **orchestrator:** add dummy orchestrator binary for local API dev ([#2744](#2744)) ([ab56e25](ab56e25)) * **orchestrator:** add NetworkAssignHook for sandbox lifecycle extensions ([#3290](#3290)) ([3261963](3261963)) * **orchestrator:** add soft-delete marker label to the check metric ([#3144](#3144)) ([1ce64f8](1ce64f8)) * **orchestrator:** add v4HeaderForUncompressed FF bit ([#2669](#2669)) ([1f459ee](1f459ee)) * **orchestrator:** always include execution metrics in sandbox webhook events ([#2852](#2852)) ([440edfe](440edfe)) * **orchestrator:** classify envd-init by exit type ([#3139](#3139)) ([1e39a4f](1e39a4f)) * **orchestrator:** graceful sandbox drain on shutdown ([#3069](#3069)) ([6ce68e3](6ce68e3)) * **orchestrator:** graceful template-build drain on shutdown ([#3079](#3079)) ([1b3001c](1b3001c)) * **orchestrator:** improved read-path telemetry ([#3063](#3063)) ([bc3fe84](bc3fe84)) * **orchestrator:** LD-gated ClickHouse write fan-out feature flag ([#3152](#3152)) ([f046fcf](f046fcf)) * **orchestrator:** make build-reserved-disk-space-mb default 256MB ([#3065](#3065)) ([d473f98](d473f98)) * **orchestrator:** record upload compression metrics ([#2761](#2761)) ([9092e35](9092e35)) * **orchestrator:** report hugepage metrics to API ([#3182](#3182)) ([7735bae](7735bae)) * **orchestrator:** run startup reclaim on boot ([#3123](#3123)) ([79b838e](79b838e)) * **orchestrator:** single-instance flock on startup ([#3143](#3143)) ([1320d6e](1320d6e)) * **orchestrator:** soft-delete consumer enforcement for storage index ([#3034](#3034)) ([fbfc918](fbfc918)) * **orchestrator:** tag envd-init meters with start_type ([#3125](#3125)) ([4466b48](4466b48)) * **orchestrator:** track and report last status change timestamp ([#2980](#2980)) ([f79be77](f79be77)) * **orchestrator:** track sandbox lifecycles ([#2998](#2998)) ([057f20c](057f20c)) * **orchestrator:** write layer sizes (logical/mapped/diff) to object metadata ([#3122](#3122)) ([11869c0](11869c0)) * **orch:** harvest resume-prefetch trace on pause ([#3067](#3067)) ([97bd4a5](97bd4a5)) * **orch:** last-cycle memory prefetch on resume ([#3258](#3258)) ([ea94196](ea94196)) * **orch:** make resume-build -gdb work on real nodes + add copy-build -gdb ([#3108](#3108)) ([5385594](5385594)) * **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP) ([#3039](#3039)) ([a98cf2c](a98cf2c)) * **orch:** per-start UFFD startup working-set metric ([#2960](#2960)) ([dc386b2](dc386b2)) * **orch:** premade NixOS base-image support ([#3412](#3412)) ([4bd42d2](4bd42d2)) * **orch:** record envd init duration histogram on failure with success attribute ([#2749](#2749)) ([afa7458](afa7458)) * **orch:** snapshot fragmentation metrics ([#2931](#2931)) ([842b007](842b007)) * per-team events TTL limit (tier + addons) ([#3181](#3181)) ([f76b2cb](f76b2cb)) * **shared:** add OTEL instrumentation to AWS S3 storage client ([#3172](#3172)) ([25b0fd1](25b0fd1)) * **storage:** per-role storage URLs, env-free storage library ([#3246](#3246)) ([fcbe909](fcbe909)) * **storage:** stamp provenance custom metadata on uploaded objects (incl. headers) ([#3033](#3033)) ([ba8604e](ba8604e)) * **storage:** write-through compressed templates to NFS on upload ([#2827](#2827)) ([57503c1](57503c1)) ### Bug Fixes * added api and orch ([#3454](#3454)) ([d56e0a8](d56e0a8)) * **block:** rephrase misleading error message in pwritevAll ([#2816](#2816)) ([1555f1b](1555f1b)) * **cache:** use 512-byte units for stat.Blocks in FileSize ([#2949](#2949)) ([0f632a9](0f632a9)) * **clean-nfs-cache:** exclude zombies from delete_age ([#3191](#3191)) ([3fa2aeb](3fa2aeb)) * **compression:** correctness findings from compression audit ([#2803](#2803)) ([d21a6a9](d21a6a9)) * **copy-build:** resolve compression suffix for build data files ([#2859](#2859)) ([8966f7e](8966f7e)) * correct 3 CVES ([#3218](#3218)) ([076823b](076823b)) * **envd:** stop freezing socat cgroup across pause/resume ([#2923](#2923)) ([8b6f2b9](8b6f2b9)) * **inspect-build:** adapt validate to new Chunker upstream API ([#2989](#2989)) ([2e0d3da](2e0d3da)) * **nbd:** adjust status poll sleep from 100ns to 100µs ([02bf51b](02bf51b)) * **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid useless busy spinning ([#2884](#2884)) ([02bf51b](02bf51b)) * **nfsproxy:** deflake TestRoundTrip EADDRINUSE ([#2987](#2987)) ([55f4d18](55f4d18)) * **orch:** denormalize upload metric file type ([#2865](#2865)) ([b1646ca](b1646ca)) * **orch:** disable the chronyd seccomp filter on Alpine when using PHC ([#3453](#3453)) ([dfa9764](dfa9764)) * **orchestrator:** anchor rsync CWD to root in template file copy ([#2835](#2835)) ([7160db9](7160db9)) * **orchestrator:** atomically replace metadata ([#3321](#3321)) ([0c4ad6b](0c4ad6b)) * **orchestrator:** avoid serializing upload headers twice ([#2762](#2762)) ([9b7b149](9b7b149)) * **orchestrator:** chunk readiness bug in P2P->compressed ([#3185](#3185)) ([74a6e5b](74a6e5b)) * **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto… ([#3173](#3173)) ([88ff17c](88ff17c)) * **orchestrator:** discard poisoned nftables conn on firewall errors ([#3008](#3008)) ([03f10e0](03f10e0)) * **orchestrator:** drop stale pre-init logs ([#3297](#3297)) ([8ec4be5](8ec4be5)) * **orchestrator:** emit compression ratios as fractions, not BP ([#2772](#2772)) ([866f4c1](866f4c1)) * **orchestrator:** export dirty-page stall counter from process start ([#2992](#2992)) ([badc8ad](badc8ad)) * **orchestrator:** harden Firecracker process shutdown ([#2996](#2996)) ([df662e7](df662e7)) * **orchestrator:** harden shutdown network cleanup ([#3000](#3000)) ([de2f391](de2f391)) * **orchestrator:** implement Docker COPY merge semantics in template builds ([#3283](#3283)) ([9174104](9174104)) * **orchestrator:** keep dedup empty-pages telemetry scan-only ([#2991](#2991)) ([35d0832](35d0832)) * **orchestrator:** let build-cache threshold flag raise above its fal… ([#3175](#3175)) ([06393c3](06393c3)) * **orchestrator:** log missing egress proxy in startup reclaim instead of defaulting silently ([#3116](#3116)) ([6ca3163](6ca3163)) * **orchestrator:** make copy-build handle filesystem-only snapshots ([#3299](#3299)) ([62add04](62add04)) * **orchestrator:** measure ext4 free space from block groups ([#3282](#3282)) ([f18f05f](f18f05f)) * **orchestrator:** normalize upload metric file labels ([#2767](#2767)) ([6dec8b3](6dec8b3)) * **orchestrator:** order egress config/firewall updates to close BYOP enable race ([#3313](#3313)) ([7faa59e](7faa59e)) * **orchestrator:** order envd.service after local-fs.target ([#3043](#3043)) ([ea2663e](ea2663e)) * **orchestrator:** order envd.service after systemd-tmpfiles-setup ([#3130](#3130)) ([9481811](9481811)) * **orchestrator:** pause upload retain retry ([#2993](#2993)) ([4f81799](4f81799)) * **orchestrator:** pin tap device host-side MAC address ([#3271](#3271)) ([3c786ba](3c786ba)) * **orchestrator:** pin UFFD copy source buffers ([#2745](#2745)) ([837fa91](837fa91)) * **orchestrator:** preserve full ENV value across stdout chunks ([#2740](#2740)) ([4822e6d](4822e6d)) * **orchestrator:** read V3 ancestors as uncompressed instead of failing ([#2994](#2994)) ([c479dd3](c479dd3)) * **orchestrator:** reject standby while draining ([#3325](#3325)) ([475a7ee](475a7ee)) * **orchestrator:** report real V4 header compression ratio ([#2771](#2771)) ([ecd344e](ecd344e)) * **orchestrator:** resolve remaining P2P/compression/V5 issues ([#3015](#3015)) ([1e4379e](1e4379e)) * **orchestrator:** sanitize OCI pull errors ([#3096](#3096)) ([a3af6c0](a3af6c0)) * **orchestrator:** scope rootfs hash to provision default ([#3129](#3129)) ([475f955](475f955)) * **orchestrator:** stop Checks health-loop leaking ([#2739](#2739)) ([17e6e60](17e6e60)) * **orchestrator:** survive SIGBUS from failing disks under mmap'd caches ([#3385](#3385)) ([8694d08](8694d08)) * **orchestrator:** tolerate missing header for legacy templates ([#3026](#3026)) ([8a44bfe](8a44bfe)) * **orch:** fall back to ID_LIKE with a warning instead of rejecting ([#3459](#3459)) ([73399b3](73399b3)) * **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind flag) ([#3048](#3048)) ([efd3d4d](efd3d4d)) * **orch:** split scheduling base build id per artifact ([#2920](#2920)) ([3e35a2a](3e35a2a)) * **orch:** validate copy-build -gdb buckets before the snapshot copy ([#3446](#3446)) ([9be382f](9be382f)) * **shared:** never report a failed envd command stream as success ([#3281](#3281)) ([69c06b6](69c06b6)) * **storage:** compression upload & cache correctness fixes ([#3231](#3231)) ([980748f](980748f)) * **storage:** don't assume V4+ ancestor gaps are uncompressed ([#3447](#3447)) ([f828d12](f828d12)) * **uffd:** dedupe deferred page faults ([#2864](#2864)) ([9680a41](9680a41)) * WrapContextAsUserError should not misclassify internal timeouts as user cancellations ([#3155](#3155)) ([8f83959](8f83959)) ### Performance Improvements * **build:** cache resolved Diff per BuildId within File.ReadAt ([#2838](#2838)) ([53de07f](53de07f)) * **build:** parallelize fragmented backing reads ([#2872](#2872)) ([c7655a7](c7655a7)) * **clean-nfs-cache:** restore dirfd-relative statx ([#2766](#2766)) ([6bdbedb](6bdbedb)) * **header:** add V5 columnar varint header format ([#2847](#2847)) ([9dd931b](9dd931b)) * **header:** pack cached Header.Mapping into a compact form ([#2844](#2844)) ([7f0b13c](7f0b13c)) * **orchestrator:** add memfile dedup density threshold ([#2862](#2862)) ([7ccfa02](7ccfa02)) * **orchestrator:** avoid V3-ancestor header refresh ([#2999](#2999)) ([cb6aa0b](cb6aa0b)) * **orch:** metrics for dirty page throttling ([#2858](#2858)) ([d2aa554](d2aa554)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com> Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>

Lev Brouk added 2 commits May 20, 2026 16:03

levb requested review from ValentaTomas, dobrac and jakubno as code owners May 20, 2026 23:12

cla-bot Bot added the cla-signed label May 20, 2026

chatgpt-codex-connector Bot reviewed May 20, 2026

View reviewed changes

gemini-code-assist Bot reviewed May 20, 2026

View reviewed changes

djeebus approved these changes May 20, 2026

View reviewed changes

levb merged commit 6bdbedb into main May 21, 2026
54 checks passed

levb deleted the lev-clean-nfs-cache-dirfd-restore branch May 21, 2026 00:03

e2b-release-please Bot mentioned this pull request Jul 30, 2026

chore(main): release orchestrator 0.0.1 #3470

Merged

e2b-release-please Bot mentioned this pull request Aug 3, 2026

chore(main): release orchestrator 0.1.0 #3506

Closed

Navigation Menu

Uh oh!

perf(clean-nfs-cache): restore dirfd-relative statx - #2766

perf(clean-nfs-cache): restore dirfd-relative statx#2766
levb merged 2 commits into
mainfrom
lev-clean-nfs-cache-dirfd-restore

levb commented May 20, 2026

Uh oh!

codecov Bot commented May 20, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 20, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

djeebus left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

levb commented May 20, 2026

Uh oh!

codecov Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 3 Tests Failed:

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 20, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

djeebus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented May 20, 2026 •

edited

Loading