feat: freeze user cgroup across pause/resume to keep envd /init responsive

[ValentaTomas]

Freezes user/pty/socat cgroups before pause and unfreezes them on resume via envd /init. Gated by envd >= 0.6.0 and the default-off freeze-user-cgroup feature flag.

[cursor]

PR Summary

High Risk
Touches pause/resume critical path by freezing/unfreezing cgroups and changing /init behavior; failures could leave sandboxes frozen or affect resume responsiveness. Also shifts access-token validation out of SetData, which could be risky if SetData is ever called from another path without the new PostInit guard.

Overview
Adds native POST /freeze and POST /unfreeze to envd to directly toggle cgroup.freeze for user/pty/socat cgroups, and makes /init always attempt an unfreeze (even for stale timestamps) but only after access-token validation.

Orchestrator reclaim/pause flow optionally calls /freeze behind a default-off freeze-user-cgroup flag and an envd >=0.6.0 gate, runs reclaim commands in a new “system” cgroup, and registers an error-path cleanup to call /unfreeze if pause fails.

^{Reviewed by Cursor Bugbot for commit d682685. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Code Review

The error returned by utils.IsGTEVersion in packages/orchestrator/pkg/sandbox/reclaim.go is ignored, which could hide configuration issues or malformed version strings; this error should be logged to ensure visibility and easier debugging.

[ValentaTomas]

Superseded by #2700 (move envd into a dedicated network namespace), which addresses the root cause.

[codecov]

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
2671	5	2666	7

View the full list of 5 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSnapshotTemplateCreateSandbox

Flake rate in main: 50.05% (Passed 462 times, Failed 463 times)

Stack Traces | 0s run time

=== RUN   TestSnapshotTemplateCreateSandbox
=== PAUSE TestSnapshotTemplateCreateSandbox
=== CONT  TestSnapshotTemplateCreateSandbox
--- FAIL: TestSnapshotTemplateCreateSandbox (0.00s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSnapshotTemplateCreateSandbox/overwritten_snapshot_build_is_served_immediately_on_sandbox_create

Flake rate in main: 50.00% (Passed 462 times, Failed 462 times)

Stack Traces | 32.4s run time

=== RUN   TestSnapshotTemplateCreateSandbox/overwritten_snapshot_build_is_served_immediately_on_sandbox_create
=== PAUSE TestSnapshotTemplateCreateSandbox/overwritten_snapshot_build_is_served_immediately_on_sandbox_create
=== CONT  TestSnapshotTemplateCreateSandbox/overwritten_snapshot_build_is_served_immediately_on_sandbox_create
    snapshot_template_test.go:295: 
        	Error Trace:	.../api/sandboxes/snapshot_template_test.go:295
        	Error:      	Received unexpected error:
        	            	failed to execute command /bin/sh in sandbox izizr878ju111va4pkrum: unavailable: HTTP status 502 Bad Gateway
        	Test:       	TestSnapshotTemplateCreateSandbox/overwritten_snapshot_build_is_served_immediately_on_sandbox_create
        	Messages:   	failed to write marker file
--- FAIL: TestSnapshotTemplateCreateSandbox/overwritten_snapshot_build_is_served_immediately_on_sandbox_create (32.42s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestDeleteTemplateFromAnotherTeamAccessToken

Flake rate in main: 50.11% (Passed 460 times, Failed 462 times)

Stack Traces | 193s run time

=== RUN   TestDeleteTemplateFromAnotherTeamAccessToken
=== PAUSE TestDeleteTemplateFromAnotherTeamAccessToken
=== CONT  TestDeleteTemplateFromAnotherTeamAccessToken
    delete_template_test.go:67: Build failed: {<nil> build was cancelled <nil>}
--- FAIL: TestDeleteTemplateFromAnotherTeamAccessToken (193.42s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 62.67% (Passed 467 times, Failed 784 times)

Stack Traces | 89s run time

=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (88.97s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 62.94% (Passed 457 times, Failed 776 times)

Stack Traces | 127s run time

=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1263}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory before tmpfs mount: 191 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Free memory before tmpfs mount: 793 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Memory to use in integrity test (80% of free, min 64MB): 634 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"634+0 records in\n634+0 records out\n664797184 bytes (665 MB, 634 MiB) copied, 2.98237 s, 223 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=634\"\n\tUser time (seconds): 0.00\n\tSystem time (seconds): 2.95\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:02.98\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2656\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 3\n\tMinor (reclaiming a frame) page faults: 343\n\tVoluntary context switches: 4\n\tInvoluntary context switches: 33\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 832 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox icopjx6ykm529frx4pbiz
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1280}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"8aec4e56d16d919ff08c8b300f00ad9b681b207c0721bea4f6bdf9d76d967319\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox icopjx6ykm529frx4pbiz
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1283}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"8aec4e56d16d919ff08c8b300f00ad9b681b207c0721bea4f6bdf9d76d967319\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox icopjx6ykm529frx4pbiz
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1287}}
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
Executing command bash in sandbox icopjx6ykm529frx4pbiz (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox icopjx6ykm529frx4pbiz: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (127.41s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Cgroups stay frozen if init steps fail before unfreeze
  • Moved the cgroup unfreeze logic into a defer statement to guarantee execution even when CA bundle install or NFS setup fails.

Or push these changes by commenting:

@cursor push 051c174648

Preview (051c174648)

diff --git a/packages/envd/internal/api/init.go b/packages/envd/internal/api/init.go
--- a/packages/envd/internal/api/init.go
+++ b/packages/envd/internal/api/init.go
@@ -171,6 +171,16 @@
 		return err
 	}
 
+	// Unfreeze cgroups that were frozen pre-snapshot; idempotent if not frozen.
+	// This must happen unconditionally even if later steps fail, so use defer.
+	defer func() {
+		for _, pt := range []cgroups.ProcessType{cgroups.ProcessTypeUser, cgroups.ProcessTypePTY, cgroups.ProcessTypeSocat} {
+			if err := a.cgroupManager.Unfreeze(pt); err != nil {
+				logger.Warn().Err(err).Msgf("unfreeze %s cgroup", pt)
+			}
+		}
+	}()
+
 	if data.Timestamp != nil {
 		// Check if current time differs significantly from the received timestamp
 		if shouldSetSystemTime(time.Now(), *data.Timestamp) {
@@ -227,13 +237,6 @@
 		}
 	}
 
-	// Unfreeze cgroups that were frozen pre-snapshot; idempotent if not frozen.
-	for _, pt := range []cgroups.ProcessType{cgroups.ProcessTypeUser, cgroups.ProcessTypePTY, cgroups.ProcessTypeSocat} {
-		if err := a.cgroupManager.Unfreeze(pt); err != nil {
-			logger.Warn().Err(err).Msgf("unfreeze %s cgroup", pt)
-		}
-	}
-
 	return nil
 }

_{You can send follow-ups to the cloud agent here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da643dccc5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Partial freeze not rolled back
  • Added call to bestEffortUnfreeze on freeze error to roll back any partially frozen cgroups before pause proceeds.

_{You can send follow-ups to the cloud agent here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3dea922b6c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: PostFreeze does not short-circuit despite stated intent
  • Reverted PostFreeze to short-circuit on first error and enhanced test to verify short-circuit behavior by tracking freeze attempts.

Or push these changes by commenting:

@cursor push 025d5f1edd

Preview (025d5f1edd)

diff --git a/packages/envd/internal/api/init.go b/packages/envd/internal/api/init.go
--- a/packages/envd/internal/api/init.go
+++ b/packages/envd/internal/api/init.go
@@ -262,7 +262,7 @@
 	logger := a.logger.With().Str(string(logs.OperationIDKey), logs.AssignOperationID()).Logger()
 
 	if err := a.freezeLock.Acquire(r.Context(), 1); err != nil {
-		w.WriteHeader(http.StatusServiceUnavailable)
+		jsonError(w, http.StatusServiceUnavailable, err)
 
 		return
 	}
@@ -292,7 +292,7 @@
 	logger := a.logger.With().Str(string(logs.OperationIDKey), logs.AssignOperationID()).Logger()
 
 	if err := a.freezeLock.Acquire(context.WithoutCancel(ctx), 1); err != nil {
-		jsonError(w, http.StatusInternalServerError, err)
+		jsonError(w, http.StatusServiceUnavailable, err)
 
 		return
 	}

diff --git a/packages/envd/internal/api/init_test.go b/packages/envd/internal/api/init_test.go
--- a/packages/envd/internal/api/init_test.go
+++ b/packages/envd/internal/api/init_test.go
@@ -600,6 +600,7 @@
 	mu               sync.Mutex
 	frozen           []cgroups.ProcessType
 	freezeErr        error
+	freezeAttempts   []cgroups.ProcessType
 	unfrozen         []cgroups.ProcessType
 	unfreezeAttempts []cgroups.ProcessType
 	unfreezeErr      error
@@ -612,6 +613,7 @@
 func (f *fakeCgroupManager) Freeze(pt cgroups.ProcessType) error {
 	f.mu.Lock()
 	defer f.mu.Unlock()
+	f.freezeAttempts = append(f.freezeAttempts, pt)
 	if f.freezeErr != nil {
 		return f.freezeErr
 	}
@@ -669,6 +671,7 @@
 
 		assert.Equal(t, http.StatusInternalServerError, rec.Code)
 		assert.Empty(t, mgr.frozen)
+		assert.Len(t, mgr.freezeAttempts, 1, "should short-circuit after first error")
 	})
 }

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 3dea922. Configure here.}

[arkamar]

I think it is ready, but please, check if my changes look ok to you.