feat(orch): debug a sandbox guest kernel with resume-build -gdb

[kalyazin]

Why

Host- and UFFD-side telemetry can't show what happens inside a resumed guest — which process/VMA is faulting, kernel state during the resume. This adds a way to attach gdb to a resumed sandbox guest with source-level kernel symbols, for diagnosing resume behaviour (page-fault attribution, scheduler/VM state) on a dev node.

What

resume-build -gdb resumes a snapshot under a --features gdb Firecracker held at the kernel entry breakpoint, loads the guest kernel's DWARF symbols, and hands over a ready gdb session (interactive, or scripted via -gdb-exec / -gdb-script). Commit-by-commit:

• feat(orch): allow skipping the envd readiness wait on resume — Config.SkipEnvdWait gates the post-resume WaitForEnvd in ResumeSandbox (the guest is held at the breakpoint and never boots envd).
• feat(orch): add fc-debug.gdb guest-kernel debugging macros — reusable gdb macros: fc-faults [N] (attributes guest page faults to comm/pid/VMA), fc-curr, fc-task, fc-regions, fc-va. Targets Linux 6.1.x x86_64.
• feat(orch): add resume-build -gdb … — the orchestration: arms FIRECRACKER_GDB_SOCKET, stages the debug FC binary, resumes in the background and connects gdb once FC binds the socket (the stub holds the snapshot load open until a debugger attaches — resuming first would deadlock), generates the init script, prints a debug-context block, drives gdb. Debug artifacts (firecracker-debug, vmlinux.debug) are fetched by version from e2b-prod-public-builds (override base via E2B_GDB_ARTIFACTS_URL; override paths via -gdb-fc / -gdb-symbols). Symbols load at offset 0 — FC boots the uncompressed vmlinux ELF directly, so image KASLR never runs and there's no slide to recover.
• docs(orch): add … runbook — gdb-debugging.md (usage, macros, observer-effect notes, customer-data rules).
• feat(orch): honor PROXY_PORT in create-build — lets create-build run alongside a live orchestrator on an alternate port (needed to build the test snapshot on a node).

Depends on the fc-versions and fc-kernels release pipelines publishing firecracker-debug-<arch> and vmlinux.debug to the public builds bucket. Until those land, the fetch 404s and you build the artifacts and pass -gdb-fc / -gdb-symbols (or set E2B_GDB_ARTIFACTS_URL). Debug-only --features gdb Firecracker must never be promoted to a prod version.

Validation

• End-to-end on the dev cluster: built a DWARF kernel + --features gdb FC on the cluster, create-build cold-booted the DWARF kernel into a snapshot, then resume-build -gdb (no flags) fetched both artifacts by version, recovered a ready session, and produced real fault attribution:
  • FAULT comm=systemd-journal pid=283 addr=0x7f83… vma=0x7f83…-… flags=0xfb
  • fc-curr → comm=systemd-journal pid=283 …; fc-regions → page_offset_base=0xffff888000000000 …
• Unit tests: debugArtifactsBaseURL, resolveOrFetch (override→local→fetch precedence, 404), proxyPort parsing/fallback.
• golangci-lint clean; every commit builds and passes tests on its own.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Changes the resume sandbox lifecycle (optional envd skip) and temporarily stages a debug Firecracker binary on disk; impact on production traffic is limited if only resume-build sets these flags, but misconfiguration or use on non-dev nodes could leave wrong FC binaries or skip readiness checks unexpectedly.

Overview
This adds a dev-only path to resume a build snapshot under a gdb-enabled Firecracker, hold the guest at the kernel entry breakpoint, load version-matched firecracker-debug and vmlinux.debug (with overrides), and hand off an interactive or scripted gdb session—including macros for attributing guest page faults. Resume no longer waits for envd when SkipEnvdWait is set (used only for this flow), gdb mode forces egress off and rejects mixing with pause/cmd/bench flags, and create-build can take an alternate sandbox proxy port via PROXY_PORT so it can run beside a live orchestrator.

^{Reviewed by Cursor Bugbot for commit 4b8f14b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
3077	2	3075	8

View the top 3 failed test(s) by shortest run time

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestMaskRequestHostAPIParameter

Stack Traces | 5.2s run time

=== RUN   TestMaskRequestHostAPIParameter
=== PAUSE TestMaskRequestHostAPIParameter
=== CONT  TestMaskRequestHostAPIParameter
    mask_request_host_test.go:44: Command [python3] output: event:{start:{pid:1120}}
Executing command cat in sandbox ici4a8bdcl81hmkk3vdsl (user: root)
    mask_request_host_test.go:68: Command [cat] output: event:{start:{pid:1121}}
    mask_request_host_test.go:68: Command [cat] output: event:{data:{stderr:"cat: /tmp/nc_output.txt: No such file or directory\n"}}
    mask_request_host_test.go:68: Command [cat] output: event:{end:{exit_code:1 exited:true status:"exit status 1" error:"exit status 1"}}
    mask_request_host_test.go:69: 
        	Error Trace:	.../tests/proxies/mask_request_host_test.go:69
        	Error:      	Received unexpected error:
        	            	command cat in sandbox ici4a8bdcl81hmkk3vdsl failed with exit code 1
        	Test:       	TestMaskRequestHostAPIParameter
--- FAIL: TestMaskRequestHostAPIParameter (5.20s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestTemplateBuildWithTagInAlias

Stack Traces | 155s run time

=== RUN   TestTemplateBuildWithTagInAlias
=== PAUSE TestTemplateBuildWithTagInAlias
=== CONT  TestTemplateBuildWithTagInAlias
    template_tags_test.go:475: Build failed: {<nil> build was cancelled <nil>}
--- FAIL: TestTemplateBuildWithTagInAlias (154.99s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestTemplateTagAssignFromSourceTag

Stack Traces | 158s run time

=== RUN   TestTemplateTagAssignFromSourceTag
=== PAUSE TestTemplateTagAssignFromSourceTag
=== CONT  TestTemplateTagAssignFromSourceTag
    template_tags_test.go:59: Build failed: {<nil> build was cancelled <nil>}
--- FAIL: TestTemplateTagAssignFromSourceTag (158.38s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[gemini-code-assist]

Code Review

Using time.After inside a select loop in waitForSocket creates a new timer on every iteration, which can cause a significant resource leak of timers; replacing it with a reused time.Ticker avoids this issue. Additionally, the write operation to the temporary GDB init script file in writeInitScript does not check for errors, which can lead to silent failures or truncated scripts if the disk is full or write permissions are restricted.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using time.After inside a select loop creates a new timer on every iteration, which can cause a significant resource leak of timers if GDB takes a while to connect or if the context is canceled. Replacing this with a reused time.Ticker avoids leaking timers and ensures efficient resource usage.

Suggested change

	func waitForSocket(ctx context.Context, path string, timeout time.Duration) error {
	deadline := time.Now().Add(timeout)
	for {
	if info, err := os.Stat(path); err == nil && info.Mode()&os.ModeSocket != 0 {
	return nil
	}
	if time.Now().After(deadline) {
	return errors.New("timeout")
	}
	select {
	case <-ctx.Done():
	return ctx.Err()
	case <-time.After(50 * time.Millisecond):
	}
	}
	}
	func waitForSocket(ctx context.Context, path string, timeout time.Duration) error {
	deadline := time.Now().Add(timeout)
	ticker := time.NewTicker(50 * time.Millisecond)
	defer ticker.Stop()
	for {
	if info, err := os.Stat(path); err == nil && info.Mode()&os.ModeSocket != 0 {
	return nil
	}
	if time.Now().After(deadline) {
	return errors.New("timeout")
	}
	select {
	case <-ctx.Done():
	return ctx.Err()
	case <-ticker.C:
	}
	}
	}

[kalyazin]

Good catch — switched waitForSocket to a single reused time.Ticker (created once, defer ticker.Stop()) instead of allocating via time.After each iteration. Fixed in 2e27ece.

[gemini-code-assist]

The write operation to the temporary GDB init script file does not check for errors, which can lead to silent failures or truncated scripts if the disk is full or write permissions are restricted. Explicitly checking the write error and closing the file descriptor before returning ensures robust error handling.

func writeInitScript(symbols, socket string) (string, error) {
	macroLib, err := macroLibPath()
	if err != nil {
		return "", err
	}
	f, err := os.CreateTemp("", "fc-debug-init-*.gdb")
	if err != nil {
		return "", err
	}
	defer f.Close()
	_, err = fmt.Fprintf(f, "set pagination off\nset confirm off\nsource %s\n# FC boots the uncompressed vmlinux ELF directly, so KASLR never relocates the image:\n# symbols sit at their link-time addresses (offset 0).\nadd-symbol-file %s -o 0x0\ntarget remote %s\n", macroLib, symbols, socket)
	if err != nil {
		return "", err
	}
	if err := f.Close(); err != nil {
		return "", err
	}

	return f.Name(), nil
}

[kalyazin]

Fixed in 2e27ece — writeInitScript now checks the Fprintf write error and the explicit Close error, and removes the temp file on the error path (so a failed write does not leave a truncated init script behind).

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8ab8d03. Configure here.}

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[cursor]

Stale comment

[cursor]

🔒 Agentic Security Review
Severity: HIGH
[Privacy Guard] The new -gdb execution path does not enforce -no-egress before entering gdbMode, even though this flow is explicitly documented as handling real customer snapshots with -no-egress as a binding rule.
Impact: A debugging run against customer snapshot state can accidentally execute with outbound network still enabled, creating a real exfiltration path for guest-derived sensitive data.

 

^{Reviewed by Cursor Security Reviewer for commit c3978c8. Configure here.}

[kalyazin]

Enforced in 638113a: -gdb now sets noEgress = true before the egress proxy is constructed, so a debugging run can't leave egress open regardless of whether the operator remembers -no-egress.

For the record, the practical risk was already near-zero: in gdb mode the guest is held at Firecracker's entry breakpoint and never boots — no envd, no kernel network init, no workload — so the frozen guest has nothing running that could exfiltrate. The change makes the runbook's "always -no-egress" mechanical rather than a manual step, as defense-in-depth.

[cursor]

🔒 Agentic Security Review
Severity: MEDIUM
The -gdb-socket value is used directly in _ = os.Remove(socket) without containment or type checks, and this path is run via sudo -E in the documented workflow.
Impact: If automation or a tool-controlled input can set -gdb-socket, this enables deletion of arbitrary root-accessible filesystem entries outside the intended temp/socket area.

 

^{Reviewed by Cursor Security Reviewer for commit c3978c8. Configure here.}

[kalyazin]

False positive for this tool. resume-build is a manual developer CLI the operator runs as sudo -E; -gdb-socket is an operator-supplied flag, not untrusted/automation-controlled input, so there's no trust boundary being crossed — anyone who can pass the flag already has a root shell and can delete files directly. os.Remove (not RemoveAll) unlinks a single path, and this is the standard "clear the stale socket before bind" step (mirrored in teardown). No change planned.