Skip to content

fix(envd): avoid Start deadlock after request cancellation - #3256

Merged
arkamar merged 1 commit into
mainfrom
fix/envd-start-event-deadlock
Jul 10, 2026
Merged

fix(envd): avoid Start deadlock after request cancellation#3256
arkamar merged 1 commit into
mainfrom
fix/envd-start-event-deadlock

Conversation

@arkamar

@arkamar arkamar commented Jul 10, 2026

Copy link
Copy Markdown
Member

Start sent its bootstrap event directly to the unbuffered channel returned by MultiplexedChannel.Fork. If the stream sender observed request cancellation first, no receiver remained and the handler blocked on the send, preventing its process-event subscriptions from being released.

Replace the unnecessary single-subscriber multiplexer with a one-slot channel so bootstrap publication cannot block after cancellation. The regression test exercises this ordering through the Connect handler and verifies that server shutdown is no longer held open by the stuck request.

Start sent its bootstrap event directly to the unbuffered channel
returned by MultiplexedChannel.Fork. If the stream sender observed
request cancellation first, no receiver remained and the handler blocked
on the send, preventing its process-event subscriptions from being
released.

Replace the unnecessary single-subscriber multiplexer with a one-slot
channel so bootstrap publication cannot block after cancellation. The
regression test exercises this ordering through the Connect handler and
verifies that server shutdown is no longer held open by the stuck
request.
@cursor

cursor Bot commented Jul 10, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches concurrency and lifecycle of a core streaming RPC path; behavior change is narrow but cancellation ordering is easy to get wrong without the new test.

Overview
Fixes a deadlock in the process Start stream when the request context is cancelled before the bootstrap start event is consumed: the handler could block forever on publishing that event, leaving connections open and blocking server shutdown.

The bootstrap path no longer uses a single-subscriber multiplexer on an unbuffered fork; it uses a one-slot buffered channel so emitting the start event cannot block after the sender goroutine has already exited on cancellation. A regression test cancels the request at handler entry and asserts httptest server close completes within a timeout. Version bumps to 0.6.10.

Reviewed by Cursor Bugbot for commit 843de1c. Bugbot is set up for automated code reviews on this repo. Configure here.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces a multiplexed channel with a buffered channel of size 1 in handleStart to prevent blocking when a receiver goroutine exits early due to a cancelled context. It also updates the test helper newTestService to support middleware and adds a new test TestStart_CancelBeforeStartEvent to verify this behavior. Feedback was provided regarding the middleware wrapping order in the test helper, which currently executes middlewares in reverse (right-to-left) order.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread packages/envd/internal/services/process/start_test.go
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
3260 1 3259 9
View the top 3 failed test(s) by shortest run time
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory
Stack Traces | 0.01s run time
=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.01s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
Stack Traces | 0.01s run time
=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory
Stack Traces | 0.02s run time
=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir
Stack Traces | 0.81s run time
=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (0.81s)
Executing command cat in sandbox iqaic6su598z03cn9jckf (user: root)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestEgressFirewallDomainCaseInsensitive
Stack Traces | 3.74s run time
=== RUN   TestEgressFirewallDomainCaseInsensitive
=== PAUSE TestEgressFirewallDomainCaseInsensitive
=== CONT  TestEgressFirewallDomainCaseInsensitive
Executing command curl in sandbox imhnamkt0u557u86wj0et
    sandbox_network_out_test.go:68: Command [curl] output: event:{start:{pid:1081}}
    sandbox_network_out_test.go:68: Command [curl] output: event:{data:{stdout:"HTTP/2 301 \r\nlocation: https://www.google.com/\r\ncontent-type: text/html; charset=UTF-8\r\ncontent-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-HGQt1XY1dkdnlLBZM378ug' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle..../csp/gws/other-hp\r\ndate: Fri, 10 Jul 2026 15:22:02 GMT\r\nexpires: Sun, 09 Aug 2026 15:22:02 GMT\r\ncache-control: public, max-age=2592000\r\nserver: gws\r\ncontent-length: 220\r\nx-xss-protection: 0\r\nx-frame-options: SAMEORIGIN\r\nalt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000\r\n\r\n"}}
    sandbox_network_out_test.go:68: Command [curl] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_network_out_test.go:68: Command [curl] completed successfully in sandbox i1cg8l6bersb7esjd6n7m
Executing command curl in sandbox i1cg8l6bersb7esjd6n7m
    sandbox_network_out_test.go:651: Command [curl] output: event:{start:{pid:1083}}
    sandbox_network_out_test.go:651: 
        	Error Trace:	.../api/sandboxes/sandbox_network_out_test.go:78
        	            				.../api/sandboxes/sandbox_network_out_test.go:651
        	Error:      	"failed to execute command curl in sandbox i1cg8l6bersb7esjd6n7m: invalid_argument: protocol error: incomplete envelope: unexpected EOF" does not contain "failed with exit code"
        	Test:       	TestEgressFirewallDomainCaseInsensitive
        	Messages:   	Expected connection failure message
Executing command curl in sandbox il50sapidnyzn8l73i7nn
--- FAIL: TestEgressFirewallDomainCaseInsensitive (3.74s)
github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestMaskRequestHostAPIParameter
Stack Traces | 5.08s run time
=== RUN   TestMaskRequestHostAPIParameter
=== PAUSE TestMaskRequestHostAPIParameter
=== CONT  TestMaskRequestHostAPIParameter
    mask_request_host_test.go:44: Command [python3] output: event:{start:{pid:1138}}
    mask_request_host_test.go:68: Command [cat] output: event:{start:{pid:1139}}
    mask_request_host_test.go:68: Command [cat] output: event:{data:{stderr:"cat: /tmp/nc_output.txt: No such file or directory\n"}}
    mask_request_host_test.go:68: Command [cat] output: event:{end:{exit_code:1 exited:true status:"exit status 1" error:"exit status 1"}}
    mask_request_host_test.go:69: 
        	Error Trace:	.../tests/proxies/mask_request_host_test.go:69
        	Error:      	Received unexpected error:
        	            	command cat in sandbox i7vofmnxkugkqqn42giio failed with exit code 1
        	Test:       	TestMaskRequestHostAPIParameter
--- FAIL: TestMaskRequestHostAPIParameter (5.08s)
View the full list of 1 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 30.77% (Passed 9 times, Failed 4 times)

Stack Traces | 0.01s run time
=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, targeted deadlock fix with a solid regression test.

Extended reasoning...

Overview

Touches only packages/envd/internal/services/process/start.go (~10 lines), a new regression test in start_test.go, and the required envd version bump (0.6.9 → 0.6.10). The fix replaces handler.MultiplexedChannel — which was being used with a single subscriber — with a plain chan rpc.ProcessEvent_Start of buffer 1. This ensures the bootstrap send after proc.Start() can never block if the sender goroutine has already exited via ctx.Done().

Security risks

None. The change is confined to intra-handler goroutine synchronization inside envd; no auth, crypto, or network-visible surface is affected.

Level of scrutiny

Low-to-moderate. Envd runs inside guest microVMs and the handler is on a well-trafficked code path, but the semantic change is narrow: buffered channel semantics for a single-shot bootstrap event. Correctness verified by inspection — if the sender goroutine wins ctx.Done(), the buffered event is silently dropped (correct, since the client has disconnected); if it wins the start path, behavior is identical to before. The <-exitChan join at the end still guarantees the outer handler waits for the sender.

Other factors

The regression test cancels the request context at HTTP handler entry via middleware, which reliably exercises the racing ordering, and bounds the server close in a goroutine with a 20s watchdog — a stuck handler would fail loudly. The PR description accurately describes both the bug and the fix. No outstanding reviewer feedback.

@arkamar
arkamar merged commit 04317f8 into main Jul 10, 2026
43 checks passed
@arkamar
arkamar deleted the fix/envd-start-event-deadlock branch July 10, 2026 15:54
arkamar added a commit that referenced this pull request Jul 10, 2026
…annel (#3257)

The bidirectional return type existed only so start.go could write a
bootstrap event directly into its fork channel; that hack was removed in
04317f8 ("fix(envd): avoid Start deadlock after request
cancellation (#3256)"), so let the compiler enforce that subscribers
never send.
charlie-e2b pushed a commit that referenced this pull request Jul 29, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-29)


### Features

* **envd:** add --no-cgroups flag to disable cgroup management
([#2811](#2811))
([e10814c](e10814c))
* **envd:** add optional EntryInfo to watch FilesystemEvent
([#2930](#2930))
([bbbc7c8](bbbc7c8))
* **envd:** allow opting into watching network mounts
([#2982](#2982))
([9799dd0](9799dd0))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([f8c7b5b](f8c7b5b))


### Bug Fixes

* added envd to artifact repository
([#3432](#3432))
([6c4f0e2](6c4f0e2))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** avoid Start deadlock after request cancellation
([#3256](#3256))
([04317f8](04317f8))
* **envd:** bound the in-memory logs queue
([#2676](#2676))
([05c9939](05c9939))
* **envd:** discard output when no subscriber is connected
([#2639](#2639))
([8cf1795](8cf1795))
* **envd:** fall back to lazy unmount when forced NFS umount fails
([#2683](#2683))
([5346a0d](5346a0d))
* **envd:** ignore closed pty read errors
([#2769](#2769))
([6118672](6118672))
* **envd:** include suppressed count in exporter error logs
([#2680](#2680))
([35c1141](35c1141))
* **envd:** make /init lock ctx-aware to prevent retry pile-up
([#2702](#2702))
([173afd4](173afd4))
* **envd:** make CA install lock ctx-aware
([#2690](#2690))
([83ee89f](83ee89f))
* **envd:** replace env vars in /init instead of merging
([#2706](#2706))
([1b52e9a](1b52e9a))
* **envd:** replace time.Sleep with ticker in ScanAndBroadcast for
prompt shutdown ([#3374](#3374))
([002fd9f](002fd9f))
* **envd:** self-heal MMDS routing on /init lookup failure
([#2701](#2701))
([90944d5](90944d5))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **envd:** stop misleading CA install cancel errors on rapid /init
([#3206](#3206))
([91d09e4](91d09e4))
* **envd:** suppress repeat MMDS poll failures
([#2678](#2678))
([73d691a](73d691a))
* **envd:** tolerate busy tmpfs cleanup in tests
([#2938](#2938))
([a485834](a485834))
* **envd:** use constant-time comparison for signature validation
([#3145](#3145))
([fcf92fa](fcf92fa))
* **envd:** use WithoutCancel for CA cleanup goroutine ctx
([#3207](#3207))
([ee7bf84](ee7bf84))


### Performance Improvements

* **envd:** stop logging streamed payload content
([#2755](#2755))
([db3868c](db3868c))
* **sandbox:** keep envd logging out of journald
([#2675](#2675))
([f6943ca](f6943ca))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
jakubno pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


## 0.0.1 (2026-07-29)


### Features

* **envd:** add --no-cgroups flag to disable cgroup management
([#2811](#2811))
([e10814c](e10814c))
* **envd:** add optional EntryInfo to watch FilesystemEvent
([#2930](#2930))
([bbbc7c8](bbbc7c8))
* **envd:** allow opting into watching network mounts
([#2982](#2982))
([9799dd0](9799dd0))
* **envd:** give envd realtime IO priority, reset for user processes
([#2681](#2681))
([f4bd1b2](f4bd1b2))
* **envd:** split collapse stats into real migrations vs already-huge
([#3021](#3021))
([0d77614](0d77614))
* **envd:** support user-defined file metadata via xattrs
([#2732](#2732))
([da8fbe4](da8fbe4))
* freeze user cgroup across pause/resume to keep envd /init responsive
([#2688](#2688))
([eceb741](eceb741))
* **orch:** collapse envd's heap into 2 MiB hugepages before pause to
cut cold-resume faults
([#2997](#2997))
([6677f73](6677f73))
* **orch:** distro-aware template base-image provisioning
([#3411](#3411))
([1abece1](1abece1))


### Bug Fixes

* added envd to artifact repository
([#3432](#3432))
([b7024ba](b7024ba))
* correct 3 CVES ([#3218](#3218))
([076823b](076823b))
* **envd:** avoid Start deadlock after request cancellation
([#3256](#3256))
([04317f8](04317f8))
* **envd:** bound the in-memory logs queue
([#2676](#2676))
([05c9939](05c9939))
* **envd:** discard output when no subscriber is connected
([#2639](#2639))
([8cf1795](8cf1795))
* **envd:** fall back to lazy unmount when forced NFS umount fails
([#2683](#2683))
([5346a0d](5346a0d))
* **envd:** ignore closed pty read errors
([#2769](#2769))
([6118672](6118672))
* **envd:** include suppressed count in exporter error logs
([#2680](#2680))
([35c1141](35c1141))
* **envd:** make /init lock ctx-aware to prevent retry pile-up
([#2702](#2702))
([173afd4](173afd4))
* **envd:** make CA install lock ctx-aware
([#2690](#2690))
([83ee89f](83ee89f))
* **envd:** replace env vars in /init instead of merging
([#2706](#2706))
([1b52e9a](1b52e9a))
* **envd:** replace time.Sleep with ticker in ScanAndBroadcast for
prompt shutdown ([#3374](#3374))
([002fd9f](002fd9f))
* **envd:** self-heal MMDS routing on /init lookup failure
([#2701](#2701))
([90944d5](90944d5))
* **envd:** stop freezing socat cgroup across pause/resume
([#2923](#2923))
([8b6f2b9](8b6f2b9))
* **envd:** stop misleading CA install cancel errors on rapid /init
([#3206](#3206))
([91d09e4](91d09e4))
* **envd:** suppress repeat MMDS poll failures
([#2678](#2678))
([73d691a](73d691a))
* **envd:** tolerate busy tmpfs cleanup in tests
([#2938](#2938))
([a485834](a485834))
* **envd:** use constant-time comparison for signature validation
([#3145](#3145))
([fcf92fa](fcf92fa))
* **envd:** use WithoutCancel for CA cleanup goroutine ctx
([#3207](#3207))
([ee7bf84](ee7bf84))


### Performance Improvements

* **envd:** stop logging streamed payload content
([#2755](#2755))
([db3868c](db3868c))
* **sandbox:** keep envd logging out of journald
([#2675](#2675))
([f6943ca](f6943ca))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: e2b-release-please[bot] <298072688+e2b-release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants