feat(observability): include kill_reason in kill-path structured logs

[rguliyev]

kill_reason already rides on the webhook event, the ClickHouse sandbox_events row, and the orchestrator.sandbox.killed counter. Adds it to the kill-path structured logs too, so log-only investigation in Loki has the same attribution.

[linear-code]

ENG-4105

[gemini-code-assist]

Code Review

This pull request adds the kill_reason field to various sandbox-related logs across the API and orchestrator services to improve traceability. It also introduces a String() method for the KillReason type to normalize empty values to 'unknown' and includes corresponding unit tests. There are no review comments, and no critical issues or bugs were identified in the changes.

[codecov]

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
2684	3	2681	7

View the full list of 4 ❄️ flaky test(s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestCommandKillNextApp

Flake rate in main: 43.29% (Passed 668 times, Failed 510 times)

Stack Traces | 212s run time

=== RUN   TestCommandKillNextApp
=== PAUSE TestCommandKillNextApp
=== CONT  TestCommandKillNextApp
Executing command cat in sandbox i1st82lhzpsui97x2wiv6 (user: root)
    process_test.go:28: Command [npx] output: event:{start:{pid:1268}}
    process_test.go:28: Command [npx] output: event:{data:{stderr:"npm WARN exec"}}
    process_test.go:28: Command [npx] output: event:{data:{stderr:" The following package was not found and will be installed: create-next-app@16.2.6\n"}}
Executing command python in sandbox icsjxaktylkxzgcqjdp1z
    process_test.go:28: Command [npx] output: event:{data:{stdout:"Creating a new Next.js app in .../home/user/nextapp.\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"Using npm.\n\nInitializing project with template: app-tw \n\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"\nInstalling dependencies:\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"- next\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"- react\n- react-dom\n\nInstalling devDependencies:\n- @tailwindcss/postcss\n- @types/node\n- @types/react\n- @types/react-dom\n- eslint\n- eslint-config-next\n- tailwindcss\n- typescript\n\n"}}
    process_test.go:29: 
        	Error Trace:	.../tests/envd/process_test.go:29
        	Error:      	Received unexpected error:
        	            	failed to execute command npx in sandbox io4nrugtkg972jhnbzjfm: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestCommandKillNextApp
--- FAIL: TestCommandKillNextApp (211.94s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 58.15% (Passed 673 times, Failed 935 times)

Stack Traces | 62.6s run time

=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (62.62s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 58.28% (Passed 663 times, Failed 926 times)

Stack Traces | 116s run time

=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1257}}
Executing command bash in sandbox iy5bosfgzlwk6r06m386k (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory before tmpfs mount: 193 MB\nFree memory before tmpfs mount: 791 MB\nMemory to use in integrity test (80% of free, min 64MB): 632 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"632+0 records in\n632+0 records out\n662700032 bytes (663 MB, 632 MiB) copied, 2.74559 s, 241 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" being timed: \""}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"dd"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" if=/dev/urandom of=/mnt/testfile bs=1M "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"count=632\"\n\tUse"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"r time "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"(secon"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ds): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0.00"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"S"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"y"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"("}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"c"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"d"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:")"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:":"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"2.73"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"P"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"r"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"c"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"f"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"C"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"P"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"U"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"h"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"i"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"j"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"b"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"g"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:":"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:02.75\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAv"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"rage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMax"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"imum resident set size (kbytes): 2632\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor (reclaiming a frame) page faults: 345\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 14\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 829 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox ibvn1ajsgxp56ra14kn6a
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1273}}
Executing command bash in sandbox ighkyx1wfn5j9zpu7hk25 (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"bf15bef11584de62ec9f7241f04f42b2d82c47c0f6770f18b09bb71018f5936f\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox ibvn1ajsgxp56ra14kn6a
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1276}}
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
Executing command bash in sandbox ibvn1ajsgxp56ra14kn6a (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox ibvn1ajsgxp56ra14kn6a: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (115.91s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestSandboxAutoResumeViaProxy

Flake rate in main: 44.18% (Passed 667 times, Failed 528 times)

Stack Traces | 12.5s run time

=== RUN   TestSandboxAutoResumeViaProxy
=== PAUSE TestSandboxAutoResumeViaProxy
=== CONT  TestSandboxAutoResumeViaProxy
    auto_resume_test.go:116: 
        	Error Trace:	.../tests/proxies/auto_resume_test.go:116
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestSandboxAutoResumeViaProxy
--- FAIL: TestSandboxAutoResumeViaProxy (12.54s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}