Skip to content

fix(uffd): dedupe deferred page faults - #2864

Merged
ValentaTomas merged 11 commits into
mainfrom
valenta/uffd-dedupe-deferred-eagain
Jun 1, 2026
Merged

fix(uffd): dedupe deferred page faults#2864
ValentaTomas merged 11 commits into
mainfrom
valenta/uffd-dedupe-deferred-eagain

Conversation

@ValentaTomas

Copy link
Copy Markdown
Member

Summary

Deduplicate deferred UFFD faults by address so repeated EAGAINs for the same page do not flood the retry queue or wake loop.

Test plan

  • Not run; Linux UFFD package depends on Linux cgo types and cannot execute on this macOS host.

@cla-bot cla-bot Bot added the cla-signed label May 29, 2026
@cursor

cursor Bot commented May 29, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Small, localized change to deferred fault retry bookkeeping in the Linux sandbox UFFD path, with unit tests and no auth or data-path impact.

Overview
Deferred userfaultfd retries now keep at most one queued fault per page address, so many threads faulting the same page after EAGAIN no longer multiply entries in the retry queue and poll wake loop. When a later fault for that page is a write, the kept entry is upgraded to a write fault so the retry resolves the page dirty instead of relying on a follow-up WP fault.

Reviewed by Cursor Bugbot for commit 2b7e091. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented May 29, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2707 2 2705 5
View the full list of 2 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 57.71% (Passed 740 times, Failed 1010 times)

Stack Traces | 69s run time
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (68.98s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 57.83% (Passed 730 times, Failed 1001 times)

Stack Traces | 199s run time
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1259}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory before tmpfs mount: 193 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Free memory before tmpfs mount: 791 MB\nMemory to use in integrity test (60% of free, min 64MB): 474 MB\n"}}
Executing command bash in sandbox iv6onr1rvybod2v97dlph (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"474+0 records in\n474+0 records out\n497025024 bytes (497 MB, 474 MiB) copied, 2.16223 s, 230 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 669 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=474\"\n\tUser time (seconds): 0.00\n\tSystem time (seconds): 1.99\n\tPercent of CPU this job got: 91%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:02.16\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2636\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 3\n\tMinor (reclaiming a frame) page faults: 344\n\tVoluntary context switches: 4\n\tInvoluntary context switches: 45\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox ihllff4vv3buv3t5d7ky1
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1275}}
Executing command bash in sandbox i3j7ic510elbcxuk4gsup (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"76a021f37a9659849c0e0c3697177da8237a2e38edb2d04d66382f4b625c6d16\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox ihllff4vv3buv3t5d7ky1
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1278}}
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
Executing command bash in sandbox ihllff4vv3buv3t5d7ky1 (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox ihllff4vv3buv3t5d7ky1: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (198.86s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces deduplication of deferred pagefaults by address in the deferredFaults struct to prevent redundant wakeups. It adds a byAddr map to track and deduplicate addresses, updates the Serve loop to only signal wakeups when a new pagefault is successfully deferred, and includes unit tests to verify this behavior. There are no review comments, and no issues were identified in the changes.

@ValentaTomas
ValentaTomas marked this pull request as ready for review May 29, 2026 22:04

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1824ece8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/orchestrator/pkg/sandbox/uffd/userfaultfd/deferred.go Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Comment thread packages/orchestrator/pkg/sandbox/uffd/userfaultfd/deferred.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 306359e1ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/orchestrator/pkg/sandbox/uffd/userfaultfd/deferred.go
Drop the dedupe map after draining so one burst of unique addresses does not retain map capacity for the sandbox lifetime.
Key deferred page faults by page boundary so same-page offsets coalesce before retry.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd691d5002

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/orchestrator/pkg/sandbox/uffd/userfaultfd/deferred.go Outdated
@ValentaTomas
ValentaTomas merged commit 9680a41 into main Jun 1, 2026
51 checks passed
@ValentaTomas
ValentaTomas deleted the valenta/uffd-dedupe-deferred-eagain branch June 1, 2026 02:20
charlie-e2b added a commit that referenced this pull request Jul 30, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([657559e](657559e))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([411b63e](411b63e))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([f8c7b5b](f8c7b5b))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([b22e820](b22e820))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([c684bd2](c684bd2))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([776ba39](776ba39))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([fda5e45](fda5e45))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([e58af28](e58af28))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P->compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([728bba3](728bba3))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([7167818](7167818))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([586ad74](586ad74))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([bfdbb24](bfdbb24))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([f551118](f551118))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([4be33ba](4be33ba))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([1abece1](1abece1))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([ea94196](ea94196))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([5385594](5385594))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([4bd42d2](4bd42d2))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([d56e0a8](d56e0a8))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([dfa9764](dfa9764))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P-&gt;compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([8694d08](8694d08))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([73399b3](73399b3))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([9be382f](9be382f))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([f828d12](f828d12))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants