Skip to content

fix(orchestrator): discard poisoned nftables conn on firewall errors - #3008

Merged
arkamar merged 3 commits into
mainfrom
fix/close-fw-connection-on-error
Jun 15, 2026
Merged

fix(orchestrator): discard poisoned nftables conn on firewall errors#3008
arkamar merged 3 commits into
mainfrom
fix/close-fw-connection-on-error

Conversation

@arkamar

@arkamar arkamar commented Jun 15, 2026

Copy link
Copy Markdown
Member

A failed buffer or Flush in ApplyRules leaves a partial batch or a sticky
error on the shared nftables.Conn, which the next Flush replays and
corrupts the slot firewall. nftables.Conn has no API to clear either, so on
any failure the conn is replaced via resetConn (transient-conn fallback if
reopening a lasting one fails). NewFirewall also leaked the lasting socket
on error paths and now closes it.

resetConn logs the recovery (warn on success, error if it failed) with the
tap interface and trace context.

arkamar added 2 commits June 15, 2026 10:54
@arkamar
fix(orchestrator): discard poisoned nftables conn on firewall errors
fdc5dc6 
A failed buffering step in ApplyRules left a partial netlink batch in the
shared nftables.Conn, and a failed Flush can leave a sticky serialization
error on it. Either one is committed or replayed by the next Flush,
corrupting the slot firewall. nftables.Conn exposes no API to clear the
buffer or the error (see google/nftables#324), so on any ApplyRules
failure the conn is replaced via resetConn (falling back to a transient
conn if reopening a lasting one fails).

NewFirewall additionally leaked the lasting netlink socket on every error
path after the conn was opened; it now closes the conn on failure.
@arkamar
feat(orchestrator): log firewall nftables conn reset for ops visibility
0c4e50f 
resetConn now logs the recovery after an ApplyRules failure: warn on
success, error when reopening or closing the conn failed (the firewall
stays usable via the transient-conn fallback). Both lines carry the
slot's tap interface and trace context — ApplyRules takes a ctx threaded
from the slot internet methods — so the event correlates with the
originating create/update RPC.
@cla-bot cla-bot Bot added the cla-signed label Jun 15, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@cursor

cursor Bot commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches per-sandbox egress firewall state where mistakes can affect isolation; the change is defensive recovery rather than new policy, with a degraded path if connection reset fully fails.

Overview
When sandbox firewall rule application fails, the shared nftables connection can keep a partial batch or sticky error that corrupts the next flush; this change replaces that connection after any ApplyRules failure and closes the lasting socket if firewall setup aborts early. Recovery is logged with tap interface and request trace context, and slot internet configure/update/reset paths pass that context through.

Reviewed by Cursor Bugbot for commit 8f9db0c. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 2.17391% with 45 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...kages/orchestrator/pkg/sandbox/network/firewall.go 0.00% 40 Missing ⚠️
packages/orchestrator/pkg/sandbox/network/slot.go 16.66% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jun 15, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

Ignoring the error from the fallback nftables.New() call in resetConn can result in fw.conn being assigned nil if both connection attempts fail, leading to a nil pointer dereference panic on subsequent firewall operations.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread packages/orchestrator/pkg/sandbox/network/firewall.go Outdated
@arkamar
fix(orchestrator): guard resetConn against a nil fallback conn
8f9db0c 
nftables.New() cannot fail today, but resetConn discarded its error and
would store a nil conn if that ever changes, panicking the next
ApplyRules. Check the transient fallback's error, join it into the result,
and only overwrite fw.conn when the new conn is non-nil. If both
constructors fail, keep the old conn and log that it is reused.
jakubno
jakubno approved these changes Jun 15, 2026
View reviewed changes
Comment thread packages/orchestrator/pkg/sandbox/network/firewall.go
@arkamar
arkamar merged commit 03f10e0 into main Jun 15, 2026
54 checks passed
@arkamar
arkamar deleted the fix/close-fw-connection-on-error branch June 15, 2026 17:32
@e2b-release-please e2b-release-please Bot mentioned this pull request Jul 30, 2026
Merged
charlie-e2b added a commit that referenced this pull request Jul 30, 2026
@e2b-release-please @charlie-e2b
chore(main): release orchestrator 0.0.1 (#3470)
836ca54 
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([657559e](657559e))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([411b63e](411b63e))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([f8c7b5b](f8c7b5b))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([b22e820](b22e820))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([c684bd2](c684bd2))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([776ba39](776ba39))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([fda5e45](fda5e45))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([e58af28](e58af28))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P->compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([728bba3](728bba3))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([7167818](7167818))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([586ad74](586ad74))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([bfdbb24](bfdbb24))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
@e2b-release-please @charlie-e2b
chore(main): release orchestrator 0.0.1 (#3470)
a5ed6cd 
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-30)


### Features

* **api:** add sandbox IAM workload token configuration
([13ddb3d](13ddb3d))
* **api:** add sandbox workload identity permission
([#3319](#3319))
([13ddb3d](13ddb3d))
* **api:** SOCKS5 egress proxy on sandbox network config (BYOP)
([#2642](#2642))
([1fc3820](1fc3820))
* **cfg:** add DISABLE_STARTUP_RECLAIM flag
([#3081](#3081))
([7677ca6](7677ca6))
* **clickhouse:** implement multi-cluster fan-out for events and stats
([#2925](#2925))
([39594c6](39594c6))
* dynamic sandbox log routing and ClickHouse-backed log reads
([#3236](#3236))
([1b19a3b](1b19a3b))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* **featureflags:** support per-service context providers
([#3100](#3100))
([65297c1](65297c1))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **metrics:** break down pause-snapshot latency by step
([#3426](#3426))
([f551118](f551118))
* **metrics:** label pause telemetry by fs_only
([#3425](#3425))
([4be33ba](4be33ba))
* **observability:** add kill_reason to sandbox.lifecycle.killed
([#2833](#2833))
([e45418f](e45418f))
* **observability:** include kill_reason in kill-path structured logs
([#2846](#2846))
([33c49f7](33c49f7))
* **orch:** add envd-version to LaunchDarkly sandbox context
([#3051](#3051))
([37d3b92](37d3b92))
* **orch:** add less, nftables, iputils-ping, and jq to base
provisioning ([#2736](#2736))
([a1e010e](a1e010e))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** debug a sandbox guest kernel with resume-build -gdb
([#3040](#3040))
([37bb0dc](37bb0dc))
* **orch:** decouple warm resume from memfile dedup
([#3166](#3166))
([77f25a0](77f25a0))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([1abece1](1abece1))
* **orchestrator/cgroup:** list and destroy leaked sandbox cgroups
([#3086](#3086))
([bce1d84](bce1d84))
* **orchestrator/nbd:** inspect and disconnect connected devices
([#3087](#3087))
([4d47148](4d47148))
* **orchestrator/network:** list slot namespaces
([#3089](#3089))
([c23dbc7](c23dbc7))
* **orchestrator/network:** list slot namespaces
([#3090](#3090))
([fbfce25](fbfce25))
* **orchestrator:** add -force-reboot to resume-build to cold-boot
memory-snaphsot builds
([#3208](#3208))
([cf8f15b](cf8f15b))
* **orchestrator:** add allocated resource metrics for sandboxes
([#2943](#2943))
([95cb6d3](95cb6d3))
* **orchestrator:** add dummy orchestrator binary for local API dev
([#2744](#2744))
([ab56e25](ab56e25))
* **orchestrator:** add NetworkAssignHook for sandbox lifecycle
extensions ([#3290](#3290))
([3261963](3261963))
* **orchestrator:** add soft-delete marker label to the check metric
([#3144](#3144))
([1ce64f8](1ce64f8))
* **orchestrator:** add v4HeaderForUncompressed FF bit
([#2669](#2669))
([1f459ee](1f459ee))
* **orchestrator:** always include execution metrics in sandbox webhook
events ([#2852](#2852))
([440edfe](440edfe))
* **orchestrator:** classify envd-init by exit type
([#3139](#3139))
([1e39a4f](1e39a4f))
* **orchestrator:** graceful sandbox drain on shutdown
([#3069](#3069))
([6ce68e3](6ce68e3))
* **orchestrator:** graceful template-build drain on shutdown
([#3079](#3079))
([1b3001c](1b3001c))
* **orchestrator:** improved read-path telemetry
([#3063](#3063))
([bc3fe84](bc3fe84))
* **orchestrator:** LD-gated ClickHouse write fan-out feature flag
([#3152](#3152))
([f046fcf](f046fcf))
* **orchestrator:** make build-reserved-disk-space-mb default 256MB
([#3065](#3065))
([d473f98](d473f98))
* **orchestrator:** record upload compression metrics
([#2761](#2761))
([9092e35](9092e35))
* **orchestrator:** report hugepage metrics to API
([#3182](#3182))
([7735bae](7735bae))
* **orchestrator:** run startup reclaim on boot
([#3123](#3123))
([79b838e](79b838e))
* **orchestrator:** single-instance flock on startup
([#3143](#3143))
([1320d6e](1320d6e))
* **orchestrator:** soft-delete consumer enforcement for storage index
([#3034](#3034))
([fbfc918](fbfc918))
* **orchestrator:** tag envd-init meters with start_type
([#3125](#3125))
([4466b48](4466b48))
* **orchestrator:** track and report last status change timestamp
([#2980](#2980))
([f79be77](f79be77))
* **orchestrator:** track sandbox lifecycles
([#2998](#2998))
([057f20c](057f20c))
* **orchestrator:** write layer sizes (logical/mapped/diff) to object
metadata ([#3122](#3122))
([11869c0](11869c0))
* **orch:** harvest resume-prefetch trace on pause
([#3067](#3067))
([97bd4a5](97bd4a5))
* **orch:** last-cycle memory prefetch on resume
([#3258](#3258))
([ea94196](ea94196))
* **orch:** make resume-build -gdb work on real nodes + add copy-build
-gdb ([#3108](#3108))
([5385594](5385594))
* **orch:** opt-in DSCP marker for sandbox egress (SANDBOX_EGRESS_DSCP)
([#3039](#3039))
([a98cf2c](a98cf2c))
* **orch:** per-start UFFD startup working-set metric
([#2960](#2960))
([dc386b2](dc386b2))
* **orch:** premade NixOS base-image support
([#3412](#3412))
([4bd42d2](4bd42d2))
* **orch:** record envd init duration histogram on failure with success
attribute ([#2749](#2749))
([afa7458](afa7458))
* **orch:** snapshot fragmentation metrics
([#2931](#2931))
([842b007](842b007))
* per-team events TTL limit (tier + addons)
([#3181](#3181))
([f76b2cb](f76b2cb))
* **shared:** add OTEL instrumentation to AWS S3 storage client
([#3172](#3172))
([25b0fd1](25b0fd1))
* **storage:** per-role storage URLs, env-free storage library
([#3246](#3246))
([fcbe909](fcbe909))
* **storage:** stamp provenance custom metadata on uploaded objects
(incl. headers) ([#3033](#3033))
([ba8604e](ba8604e))
* **storage:** write-through compressed templates to NFS on upload
([#2827](#2827))
([57503c1](57503c1))


### Bug Fixes

* added api and orch
([#3454](#3454))
([d56e0a8](d56e0a8))
* **block:** rephrase misleading error message in pwritevAll
([#2816](#2816))
([1555f1b](1555f1b))
* **cache:** use 512-byte units for stat.Blocks in FileSize
([#2949](#2949))
([0f632a9](0f632a9))
* **clean-nfs-cache:** exclude zombies from delete_age
([#3191](#3191))
([3fa2aeb](3fa2aeb))
* **compression:** correctness findings from compression audit
([#2803](#2803))
([d21a6a9](d21a6a9))
* **copy-build:** resolve compression suffix for build data files
([#2859](#2859))
([8966f7e](8966f7e))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **inspect-build:** adapt validate to new Chunker upstream API
([#2989](#2989))
([2e0d3da](2e0d3da))
* **nbd:** adjust status poll sleep from 100ns to 100µs
([02bf51b](02bf51b))
* **nbd:** change NBD status poll sleep from 100ns to 100µs to avoid
useless busy spinning
([#2884](#2884))
([02bf51b](02bf51b))
* **nfsproxy:** deflake TestRoundTrip EADDRINUSE
([#2987](#2987))
([55f4d18](55f4d18))
* **orch:** denormalize upload metric file type
([#2865](#2865))
([b1646ca](b1646ca))
* **orch:** disable the chronyd seccomp filter on Alpine when using PHC
([#3453](#3453))
([dfa9764](dfa9764))
* **orchestrator:** anchor rsync CWD to root in template file copy
([#2835](#2835))
([7160db9](7160db9))
* **orchestrator:** atomically replace metadata
([#3321](#3321))
([0c4ad6b](0c4ad6b))
* **orchestrator:** avoid serializing upload headers twice
([#2762](#2762))
([9b7b149](9b7b149))
* **orchestrator:** chunk readiness bug in P2P-&gt;compressed
([#3185](#3185))
([74a6e5b](74a6e5b))
* **orchestrator:** deschedule flaky eviction-loop race in TestDiffSto…
([#3173](#3173))
([88ff17c](88ff17c))
* **orchestrator:** discard poisoned nftables conn on firewall errors
([#3008](#3008))
([03f10e0](03f10e0))
* **orchestrator:** drop stale pre-init logs
([#3297](#3297))
([8ec4be5](8ec4be5))
* **orchestrator:** emit compression ratios as fractions, not BP
([#2772](#2772))
([866f4c1](866f4c1))
* **orchestrator:** export dirty-page stall counter from process start
([#2992](#2992))
([badc8ad](badc8ad))
* **orchestrator:** harden Firecracker process shutdown
([#2996](#2996))
([df662e7](df662e7))
* **orchestrator:** harden shutdown network cleanup
([#3000](#3000))
([de2f391](de2f391))
* **orchestrator:** implement Docker COPY merge semantics in template
builds ([#3283](#3283))
([9174104](9174104))
* **orchestrator:** keep dedup empty-pages telemetry scan-only
([#2991](#2991))
([35d0832](35d0832))
* **orchestrator:** let build-cache threshold flag raise above its fal…
([#3175](#3175))
([06393c3](06393c3))
* **orchestrator:** log missing egress proxy in startup reclaim instead
of defaulting silently
([#3116](#3116))
([6ca3163](6ca3163))
* **orchestrator:** make copy-build handle filesystem-only snapshots
([#3299](#3299))
([62add04](62add04))
* **orchestrator:** measure ext4 free space from block groups
([#3282](#3282))
([f18f05f](f18f05f))
* **orchestrator:** normalize upload metric file labels
([#2767](#2767))
([6dec8b3](6dec8b3))
* **orchestrator:** order egress config/firewall updates to close BYOP
enable race ([#3313](#3313))
([7faa59e](7faa59e))
* **orchestrator:** order envd.service after local-fs.target
([#3043](#3043))
([ea2663e](ea2663e))
* **orchestrator:** order envd.service after systemd-tmpfiles-setup
([#3130](#3130))
([9481811](9481811))
* **orchestrator:** pause upload retain retry
([#2993](#2993))
([4f81799](4f81799))
* **orchestrator:** pin tap device host-side MAC address
([#3271](#3271))
([3c786ba](3c786ba))
* **orchestrator:** pin UFFD copy source buffers
([#2745](#2745))
([837fa91](837fa91))
* **orchestrator:** preserve full ENV value across stdout chunks
([#2740](#2740))
([4822e6d](4822e6d))
* **orchestrator:** read V3 ancestors as uncompressed instead of failing
([#2994](#2994))
([c479dd3](c479dd3))
* **orchestrator:** reject standby while draining
([#3325](#3325))
([475a7ee](475a7ee))
* **orchestrator:** report real V4 header compression ratio
([#2771](#2771))
([ecd344e](ecd344e))
* **orchestrator:** resolve remaining P2P/compression/V5 issues
([#3015](#3015))
([1e4379e](1e4379e))
* **orchestrator:** sanitize OCI pull errors
([#3096](#3096))
([a3af6c0](a3af6c0))
* **orchestrator:** scope rootfs hash to provision default
([#3129](#3129))
([475f955](475f955))
* **orchestrator:** stop Checks health-loop leaking
([#2739](#2739))
([17e6e60](17e6e60))
* **orchestrator:** survive SIGBUS from failing disks under mmap'd
caches ([#3385](#3385))
([8694d08](8694d08))
* **orchestrator:** tolerate missing header for legacy templates
([#3026](#3026))
([8a44bfe](8a44bfe))
* **orch:** fall back to ID_LIKE with a warning instead of rejecting
([#3459](#3459))
([73399b3](73399b3))
* **orch:** prevent NBD dispatch read-loop stall on WRITE_ZEROES (behind
flag) ([#3048](#3048))
([efd3d4d](efd3d4d))
* **orch:** split scheduling base build id per artifact
([#2920](#2920))
([3e35a2a](3e35a2a))
* **orch:** validate copy-build -gdb buckets before the snapshot copy
([#3446](#3446))
([9be382f](9be382f))
* **shared:** never report a failed envd command stream as success
([#3281](#3281))
([69c06b6](69c06b6))
* **storage:** compression upload & cache correctness fixes
([#3231](#3231))
([980748f](980748f))
* **storage:** don't assume V4+ ancestor gaps are uncompressed
([#3447](#3447))
([f828d12](f828d12))
* **uffd:** dedupe deferred page faults
([#2864](#2864))
([9680a41](9680a41))
* WrapContextAsUserError should not misclassify internal timeouts as
user cancellations
([#3155](#3155))
([8f83959](8f83959))


### Performance Improvements

* **build:** cache resolved Diff per BuildId within File.ReadAt
([#2838](#2838))
([53de07f](53de07f))
* **build:** parallelize fragmented backing reads
([#2872](#2872))
([c7655a7](c7655a7))
* **clean-nfs-cache:** restore dirfd-relative statx
([#2766](#2766))
([6bdbedb](6bdbedb))
* **header:** add V5 columnar varint header format
([#2847](#2847))
([9dd931b](9dd931b))
* **header:** pack cached Header.Mapping into a compact form
([#2844](#2844))
([7f0b13c](7f0b13c))
* **orchestrator:** add memfile dedup density threshold
([#2862](#2862))
([7ccfa02](7ccfa02))
* **orchestrator:** avoid V3-ancestor header refresh
([#2999](#2999))
([cb6aa0b](cb6aa0b))
* **orch:** metrics for dirty page throttling
([#2858](#2858))
([d2aa554](d2aa554))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Co-authored-by: Charlie Wyse <charlie.wyse@e2b.dev>
@e2b-release-please e2b-release-please Bot mentioned this pull request Aug 3, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jakubno jakubno jakubno approved these changes
@ValentaTomas ValentaTomas Awaiting requested review from ValentaTomas ValentaTomas is a code owner
@dobrac dobrac Awaiting requested review from dobrac dobrac is a code owner
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arkamar @jakubno