feat(orchestrator): startup resource reclaim package

[wj-e2b]

Add the startupreclaim package, which best-effort reclaims host resources
left behind by sandboxes that did not shut down cleanly: orphaned firecracker
processes (matched to ns- via /proc netns inode, with cmdline-ancestry
fallback), connected NBD devices, slot network namespaces, sandbox cgroups,
and stale sockets/fifos/overlay files. Failures are logged and counted via
OTel metrics, never fatal.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[cursor]

PR Summary

High Risk
Startup code kills process groups, mutates iptables/routes/netns, disconnects NBD, and deletes cache/temp files; incorrect matching or ordering could disrupt unrelated host processes or networking.

Overview
Adds best-effort startup reclaim so the orchestrator can clean host leftovers after unclean sandbox shutdowns: orphaned firecracker process groups (via /proc cmdline basename match), connected NBD devices, leaked network slots (discovered from persisted netns entries), sandbox cgroups, and stale sockets/fifos/overlay files. A central Run runs those steps in a fixed order (VMMs before network teardown), records OTel counters, and logs failures without aborting startup.

Network slot lifecycle is tightened so reclaim can rediscover partial teardowns: CreateNetwork idempotently tears down a stale slot before reuse, and RemoveNetwork deletes the named namespace only after host-side iptables/routes/veth cleanup succeeds, keeping the netns anchor when teardown fails so a later reclaim can retry.

^{Reviewed by Cursor Bugbot for commit 642c3ce. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Code Review

This pull request exports the netNamespacesDir constant as NetNamespacesDir and introduces helper functions NamespaceName, SlotIndexFromNamespace, and ListSlotNamespaces to manage network namespaces, along with corresponding unit tests. I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

LGTM, straightforward additive helpers following the same pattern as the recent cgroup/nbd listing PRs.

Extended reasoning...

Overview

This PR exports the netns directory constant (netNamespacesDir → NetNamespacesDir), adds two small naming helpers (NamespaceName, SlotIndexFromNamespace), and a ListSlotNamespaces function that scans a directory for ns-<idx> entries and returns sorted slot indices. The slot.go changes are purely the rename to use the exported constant. A unit test covers the parsing and listing behavior.

Security risks

None. This is read-only directory enumeration with bounds validation (idx < 1 || idx > vrtSlotsSize) and no change to slot acquisition, firewall rule logic, or namespace lifecycle.

Level of scrutiny

Low. This is a mechanical export + additive helpers that mirror the recent feat(orchestrator/cgroup): list and destroy leaked sandbox cgroups and feat(orchestrator/nbd): inspect and disconnect connected devices PRs — clearly part of a broader leaked-resource discovery effort. The helpers are pure, well-tested, and properly filter directories/invalid names.

Other factors

The test correctly verifies that directories are skipped, invalid names are rejected, and results are sorted numerically (ns-2 before ns-10). The bounds check on the parsed index appropriately rejects ns-0 and out-of-range slots. No outstanding reviewer feedback to address.

[dobrac]

why does this need to be public?

[dobrac]

if this is used only in tests, not sure if we need the method at all

[codecov]

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
3089	5	3084	8

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Stack Traces | 0.01s run time

=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Stack Traces | 0.02s run time

=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Stack Traces | 0.66s run time

=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (0.66s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[rguliyev]

@wj-e2b could you please add monitoring for reclaims labeled by instance, reclaim type, status (success, fail), (more?)

[wj-e2b]

@wj-e2b could you please add monitoring for reclaims labeled by instance, reclaim type, status (success, fail), (more?)

This already exists

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 642c3ce. Configure here.}

[cursor]

foreignNs stale after netns reclaim

High Severity

If startupreclaim.Run runs after network.NewStorageLocal, leaked ns-* entries are removed from disk but StorageLocal.foreignNs still lists them from init. Acquire skips those names before checking availability, so reclaimed slot indices stay unusable for that process even when /run/netns is clean.

 

^{Reviewed by Cursor Bugbot for commit 642c3ce. Configure here.}

[cursor]

Nil egress proxy skips rule cleanup

Medium Severity

Config.withDefaults replaces a nil EgressProxy with NewNoopEgressProxy(), so startup network reclaim’s RemoveNetwork never runs real OnSlotDelete iptables cleanup. Leaked slots can keep egress firewall rules after reclaim when Run is wired like production storage but omits the live egress implementation.

 

^{Reviewed by Cursor Bugbot for commit 642c3ce. Configure here.}

[dobrac]

can we maybe refactor the const so it can be reused?