Skip to content

feat(api): cleaner evicts sandboxes older than StaleCutoff#2640

Merged
jakubno merged 4 commits into
mainfrom
cleaner-evict-expired
May 13, 2026
Merged

feat(api): cleaner evicts sandboxes older than StaleCutoff#2640
jakubno merged 4 commits into
mainfrom
cleaner-evict-expired

Conversation

@jakubno
Copy link
Copy Markdown
Member

@jakubno jakubno commented May 13, 2026

The Redis cleaner now removes sandboxes whose EndTime is older than StaleCutoff, in addition to pruning orphaned index entries. Recently expired sandboxes are left to the evictor to avoid races.

@jakubno
feat(api): cleaner evicts sandboxes older than StaleCutoff
8bc7d4b 
The Redis cleaner now removes sandboxes whose EndTime is older than
StaleCutoff, in addition to pruning orphaned index entries. Recently
expired sandboxes are left to the evictor to avoid races.
@cla-bot cla-bot Bot added the cla-signed label May 13, 2026
@cursor
Copy link
Copy Markdown

cursor Bot commented May 13, 2026 •
edited
Loading

PR Summary

Medium Risk
Automatically deletes Redis sandbox records based on EndTime, which can cause unexpected data loss if clocks/state transitions are off or EndTime is incorrect; failures in the new removal loop are only logged, so cleanup can silently be partial.

Overview
The cleaner now calls Remove on sandboxes returned by ExpiredItems when their EndTime is older than sandbox.StaleCutoff, and adds tests to verify stale expired sandboxes are fully removed while recently expired ones are preserved. Potential issues: this introduces time-based deletion logic that depends on accurate EndTime/clock behavior and may race with other removal/evictor flows, and per-sandbox removal errors don’t fail the cleanup cycle (they’re only logged).

Reviewed by Cursor Bugbot for commit cadc7dd. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 13, 2026 •
edited
Loading

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
2620 6 2614 5
View the top 1 failed test(s) by shortest run time 
github.com/e2b-dev/infra/packages/api/internal/cache/templates::TestTemplateMetadataCache_Get_NotFound
Stack Traces | 0.35s run time 
=== RUN   TestTemplateMetadataCache_Get_NotFound
=== PAUSE TestTemplateMetadataCache_Get_NotFound
=== CONT  TestTemplateMetadataCache_Get_NotFound
    template_metadata_cache_test.go:38: 
        	Error Trace:	.../pkg/testutils/db.go:75
        	            				.../cache/templates/template_metadata_cache_test.go:38
        	Error:      	Received unexpected error:
        	            	run postgres: generic container: create container: Error response from daemon: Head "https://registry-1.docker..../postgres/manifests/16-alpine": Get "https://auth.docker.io/token?account=githubactions&scope=repository%3Alibrary%2Fpostgres%3Apull&service=registry.docker.io": EOF
        	Test:       	TestTemplateMetadataCache_Get_NotFound
        	Messages:   	Failed to start postgres container
--- FAIL: TestTemplateMetadataCache_Get_NotFound (0.35s)
2026/05/13 15:43:53 OK   20000101000000_auth.sql (4.22ms)
2026/05/13 15:43:53 OK   20000101000001_rls_for_migration_table.sql (922.41µs)
2026/05/13 15:43:53 OK   20231124185944_create_schemas_and_tables.sql (49.56ms)
2026/05/13 15:43:53 OK   20231220094836_create_triggers_and_policies.sql (4ms)
2026/05/13 15:43:53 OK   20231222181015_add_env_resources.sql (1.35ms)
2026/05/13 15:43:53 OK   20240103104619_add_team_email.sql (1.53ms)
2026/05/13 15:43:53 OK   20240106121919_add_team_ban_fields.sql (1.46ms)
2026/05/13 15:43:53 OK   20240202120312_add_kernel_version.sql (1.28ms)
2026/05/13 15:43:53 OK   20240219190940_add_max_length_hours.sql (1.04ms)
2026/05/13 15:43:53 OK   20240221023613_add_firecracker_version.sql (1.09ms)
2026/05/13 15:43:53 OK   20240221215408_update_firecracker_version.sql (1.45ms)
2026/05/13 15:43:53 OK   20240305221944_remove_tier_resources.sql (1.9ms)
2026/05/13 15:43:53 OK   20240315165236_create_env_builds.sql (6ms)
2026/05/13 15:43:53 OK   20240605070918_refactor_triggers_and_policies.sql (4.58ms)
2026/05/13 15:43:53 OK   20240625095352_add_envd_version.sql (988.71µs)
2026/05/13 15:43:53 OK   20240728094137_add_default_team_flag.sql (2.18ms)
2026/05/13 15:43:53 OK   20240909142106_drop_team_policies.sql (1.09ms)
2026/05/13 15:43:53 OK   20241120222814_add_team_api_key_metadata.sql (3.74ms)
2026/05/13 15:43:53 OK   20241121225404_add_team_api_key_id.sql (5.04ms)
2026/05/13 15:43:53 OK   20241127174604_add_env_creator.sql (1.91ms)
2026/05/13 15:43:53 OK   20241206124325_add_user_team_adder.sql (1.68ms)
2026/05/13 15:43:53 OK   20241213142106_create_snapshots.sql (2.83ms)
2026/05/13 15:43:53 OK   20250106142106_remove_team_is_default.sql (1.2ms)
2026/05/13 15:43:53 OK   20250206105106_add_snapshot_constraints.sql (2.98ms)
2026/05/13 15:43:53 OK   20250211160814_add_token_hashes.sql (5.96ms)
2026/05/13 15:43:53 OK   20250306105106_add_indexes.sql (4.66ms)
2026/05/13 15:43:53 OK   20250404151700_add_snapshots_sbx_started_at.sql (5.36ms)
2026/05/13 15:43:53 OK   20250409113306_add_envd_secured_to_snapshot.sql (1.25ms)
2026/05/13 15:43:53 OK   20250506112836_builds_status_index.sql (1.31ms)
2026/05/13 15:43:53 OK   20250507134356_add_max_specs_to_tier.sql (1.37ms)
2026/05/13 15:43:53 OK   20250513111201_tier_fix_max_memory.sql (1.3ms)
2026/05/13 15:43:53 OK   20250522105042_users_teams_add_created_at.sql (1.11ms)
2026/05/13 15:43:53 OK   20250528203546_ready_command.sql (864.9µs)
2026/05/13 15:43:53 OK   20250606204750_optimize_hashed_key_schema.sql (1.74ms)
2026/05/13 15:43:53 OK   20250606213446_deployment_cluster.sql (3.9ms)
2026/05/13 15:43:53 OK   20250624001047_deploy_cluster_policy.sql (994.87µs)
2026/05/13 15:43:53 OK   20250624001048_cluster_for_templates.sql (2.38ms)
2026/05/13 15:43:53 OK   20250624001049_cluster_for_builds.sql (1.2ms)
2026/05/13 15:43:53 OK   20250624232413_add_build_reason.sql (687.44µs)
2026/05/13 15:43:53 OK   20250708135400_snapshots_migrations.sql (1.73ms)
2026/05/13 15:43:53 OK   20250708135401_snapshot_pause_node_id.sql (785.82µs)
2026/05/13 15:43:53 OK   20250714132924_cluster_sandbox_domain.sql (672.83µs)
2026/05/13 15:43:53 OK   20250728085406_add_create_network_flag.sql (771.44µs)
2026/05/13 15:43:53 OK   20250802144312_update_firecracker_to_v1_12_1.sql (826.42µs)
2026/05/13 15:43:53 OK   20250815181502_change_build_reason_to_json.sql (4.51ms)
2026/05/13 15:43:53 OK   20250818114512_auto_pause.sql (1.09ms)
2026/05/13 15:43:53 OK   20250820102103_add_snapshots_indexes.sql (3.77ms)
2026/05/13 15:43:53 OK   20250824185633_build_node_not_nullable.sql (1.2ms)
2026/05/13 15:43:53 OK   20250824185634_snapshot_node_not_nullable.sql (1.03ms)
2026/05/13 15:43:53 OK   20250825100000_remove_default_keys.sql (875.37µs)
2026/05/13 15:43:53 OK   20250825102440_add_hash_indexes.sql (2.59ms)
2026/05/13 15:43:53 OK   20250825102800_hash_existing_keys.sql (4.13ms)
2026/05/13 15:43:53 OK   20250825102900_drop_mask_columns.sql (1.7ms)
2026/05/13 15:43:53 OK   20250901161352_add_concurrent_template_builds_to_tier.sql (1.39ms)
2026/05/13 15:43:53 OK   20250905134524_fix_env_builds_non_null_fields.sql (1.03ms)
2026/05/13 15:43:53 OK   20250910063940_make_raw_keys_nullable.sql (2.19ms)
2026/05/13 15:43:53 OK   20250910072612_access_tokens_id_non_nullable.sql (734.1µs)
2026/05/13 15:43:53 OK   20250910124212_remove_raw_keys.sql (1.94ms)
2026/05/13 15:43:53 OK   20250923094021_add_team_id_to_snapshots.sql (2.35ms)
2026/05/13 15:43:53 OK   20250923103546_add_snapshot_list_index.sql (2.01ms)
2026/05/13 15:43:53 OK   20250923103614_make_team_id_non_nullable.sql (900.55µs)
2026/05/13 15:43:53 OK   20251009170758_unique_snapshots.sql (1.06ms)
2026/05/13 15:43:53 OK   20251011200438_create_addons_table.sql (7.74ms)
2026/05/13 15:43:53 OK   20251018100653_add_build_version.sql (842.78µs)
2026/05/13 15:43:53 OK   20251026192416_addons_idempotency.sql (1.38ms)
2026/05/13 15:43:53 OK   20251030130958_add_env_index_to_snapshots.sql (1.18ms)
2026/05/13 15:43:53 OK   20251106172810_add_config_to_snapshots.sql (648.85µs)
2026/05/13 15:43:53 OK   20251121101953_build_node_id_nullable.sql (1.15ms)
2026/05/13 15:43:53 OK   20251127000000_add_machine_info_to_env_builds.sql (1.31ms)
2026/05/13 15:43:53 OK   20251216135834_index_snapshots_base_env_id.sql (1.31ms)
2026/05/13 15:43:53 OK   20251217000000_create_public_users_table.sql (5.28ms)
2026/05/13 15:43:53 OK   20251218160000_allow_m_n_builds_with_tags.sql (10.71ms)
2026/05/13 15:43:53 OK   20251218170000_optimize_build_assignment_indexes.sql (2ms)
2026/05/13 15:43:53 OK   20260121175429_add_team_slug.sql (11.95ms)
2026/05/13 15:43:53 OK   20260121175430_add_env_aliases_namespace.sql (2.32ms)
2026/05/13 15:43:53 OK   20260127120000_add_env_aliases_uuid_pkey.sql (7.94ms)
2026/05/13 15:43:53 OK   20260129105527_populate_env_aliases_namespace.sql (1.7ms)
2026/05/13 15:43:53 OK   20260204172712_remove_build_assignment_triggers.sql (3.09ms)
2026/05/13 15:43:53 OK   20260210120000_add_env_builds_created_at_index.sql (1.46ms)
2026/05/13 15:43:53 OK   20260210120001_add_env_and_build_source_columns.sql (3.5ms)
2026/05/13 15:43:53 OK   20260210120002_add_status_group_column.sql (6.01ms)
2026/05/13 15:43:53 OK   20260211120000_add_snapshot_templates.sql (4.09ms)
2026/05/13 15:43:53 OK   20260216120000_add_envs_team_id_source_index.sql (2.39ms)
2026/05/13 15:43:53 OK   20260218120000_add_team_id_to_env_builds.sql (7.15ms)
2026/05/13 15:43:53 OK   20260225120000_add_env_builds_team_status_group_index.sql (1.25ms)
2026/05/13 15:43:53 OK   20260228120000_snapshot_template_origin_node.sql (967µs)
2026/05/13 15:43:53 OK   20260304120000_volumes.sql (4.41ms)
2026/05/13 15:43:53 OK   20260305120000_add_env_builds_team_active_partial_index.sql (1.37ms)
2026/05/13 15:43:53 OK   20260305130000_create_active_template_builds.sql (3.23ms)
2026/05/13 15:43:53 OK   20260309120000_add_team_sandbox_scheduling_labels.sql (1.07ms)
2026/05/13 15:43:53 OK   20260310120000_add_snapshots_metadata_gin_index.sql (27.08ms)
2026/05/13 15:43:53 OK   20260312120000_fix_snapshots_jsonb_null_metadata.sql (2.92ms)
2026/05/13 15:43:53 OK   20260313120000_fix_snapshots_created_at.sql (2.04ms)
2026/05/13 15:43:53 OK   20260314120000_fix_snapshots_metadata_sql_null_trigger.sql (647.82µs)
2026/05/13 15:43:53 OK   20260316120000_users_teams_uuid_pkey.sql (5.35ms)
2026/05/13 15:43:53 OK   20260316130000_repoint_user_fks_to_public_users.sql (17.08ms)
2026/05/13 15:43:53 OK   20260413120000_active_template_builds_fk_envs.sql (3.27ms)
2026/05/13 15:43:53 OK   20260416120000_remove_user_team_provision_triggers.sql (8.55ms)
2026/05/13 15:43:53 OK   20260423170000_cluster_auth_org_id.sql (1.73ms)
2026/05/13 15:43:53 goose: successfully migrated database to version: 20260423170000
2026/05/13 15:43:53 OK   20000101000000_auth.sql (4.26ms)
2026/05/13 15:43:53 OK   20000101000001_rls_for_migration_table.sql (822.12µs)
2026/05/13 15:43:53 OK   20231124185944_create_schemas_and_tables.sql (20.84ms)
2026/05/13 15:43:53 OK   20231220094836_create_triggers_and_policies.sql (3.34ms)
2026/05/13 15:43:53 OK   20231222181015_add_env_resources.sql (1.03ms)
2026/05/13 15:43:53 OK   20240103104619_add_team_email.sql (1.71ms)
2026/05/13 15:43:53 OK   20240106121919_add_team_ban_fields.sql (1.08ms)
2026/05/13 15:43:53 OK   20240202120312_add_kernel_version.sql (1.28ms)
2026/05/13 15:43:53 OK   20240219190940_add_max_length_hours.sql (990.63µs)
2026/05/13 15:43:54 OK   20240221023613_add_firecracker_version.sql (940.93µs)
2026/05/13 15:43:54 OK   20240221215408_update_firecracker_version.sql (1.08ms)
2026/05/13 15:43:54 OK   20240305221944_remove_tier_resources.sql (1.75ms)
2026/05/13 15:43:54 OK   20240315165236_create_env_builds.sql (5.56ms)
2026/05/13 15:43:54 OK   20240605070918_refactor_triggers_and_policies.sql (4.32ms)
2026/05/13 15:43:54 OK   20240625095352_add_envd_version.sql (1.05ms)
2026/05/13 15:43:54 OK   20240728094137_add_default_team_flag.sql (2.47ms)
2026/05/13 15:43:54 OK   20240909142106_drop_team_policies.sql (1.27ms)
2026/05/13 15:43:54 OK   20241120222814_add_team_api_key_metadata.sql (3.61ms)
2026/05/13 15:43:54 OK   20241121225404_add_team_api_key_id.sql (4.96ms)
2026/05/13 15:43:54 OK   20241127174604_add_env_creator.sql (1.73ms)
2026/05/13 15:43:54 OK   20241206124325_add_user_team_adder.sql (1.74ms)
2026/05/13 15:43:54 OK   20241213142106_create_snapshots.sql (2.3ms)
2026/05/13 15:43:54 OK   20250106142106_remove_team_is_default.sql (1.11ms)
2026/05/13 15:43:54 OK   20250206105106_add_snapshot_constraints.sql (2.67ms)
2026/05/13 15:43:54 OK   20250211160814_add_token_hashes.sql (5.93ms)
2026/05/13 15:43:54 OK   20250306105106_add_indexes.sql (4.39ms)
2026/05/13 15:43:54 OK   20250404151700_add_snapshots_sbx_started_at.sql (4.44ms)
2026/05/13 15:43:54 OK   20250409113306_add_envd_secured_to_snapshot.sql (1.15ms)
2026/05/13 15:43:54 OK   20250506112836_builds_status_index.sql (1.23ms)
2026/05/13 15:43:54 OK   20250507134356_add_max_specs_to_tier.sql (1.27ms)
2026/05/13 15:43:54 OK   20250513111201_tier_fix_max_memory.sql (1.09ms)
2026/05/13 15:43:54 OK   20250522105042_users_teams_add_created_at.sql (1.21ms)
2026/05/13 15:43:54 OK   20250528203546_ready_command.sql (1.03ms)
2026/05/13 15:43:54 OK   20250606204750_optimize_hashed_key_schema.sql (2.01ms)
2026/05/13 15:43:54 OK   20250606213446_deployment_cluster.sql (3.87ms)
2026/05/13 15:43:54 OK   20250624001047_deploy_cluster_policy.sql (854.39µs)
2026/05/13 15:43:54 OK   20250624001048_cluster_for_templates.sql (1.82ms)
2026/05/13 15:43:54 OK   20250624001049_cluster_for_builds.sql (749.21µs)
2026/05/13 15:43:54 OK   20250624232413_add_build_reason.sql (763.6µs)
2026/05/13 15:43:54 OK   20250708135400_snapshots_migrations.sql (1.24ms)
2026/05/13 15:43:54 OK   20250708135401_snapshot_pause_node_id.sql (632.8µs)
2026/05/13 15:43:54 OK   20250714132924_cluster_sandbox_domain.sql (657.1µs)
2026/05/13 15:43:54 OK   20250728085406_add_create_network_flag.sql (776.69µs)
2026/05/13 15:43:54 OK   20250802144312_update_firecracker_to_v1_12_1.sql (843.35µs)
2026/05/13 15:43:54 OK   20250815181502_change_build_reason_to_json.sql (4.46ms)
2026/05/13 15:43:54 OK   20250818114512_auto_pause.sql (1.14ms)
2026/05/13 15:43:54 OK   20250820102103_add_snapshots_indexes.sql (4.12ms)
2026/05/13 15:43:54 OK   20250824185633_build_node_not_nullable.sql (1.12ms)
2026/05/13 15:43:54 OK   20250824185634_snapshot_node_not_nullable.sql (1.01ms)
2026/05/13 15:43:54 OK   20250825100000_remove_default_keys.sql (865.17µs)
2026/05/13 15:43:54 OK   20250825102440_add_hash_indexes.sql (2.64ms)
2026/05/13 15:43:54 OK   20250825102800_hash_existing_keys.sql (4.54ms)
2026/05/13 15:43:54 OK   20250825102900_drop_mask_columns.sql (1.38ms)
2026/05/13 15:43:54 OK   20250901161352_add_concurrent_template_builds_to_tier.sql (1.17ms)
2026/05/13 15:43:54 OK   20250905134524_fix_env_builds_non_null_fields.sql (989.44µs)
2026/05/13 15:43:54 OK   20250910063940_make_raw_keys_nullable.sql (1.91ms)
2026/05/13 15:43:54 OK   20250910072612_access_tokens_id_non_nullable.sql (592.99µs)
2026/05/13 15:43:54 OK   20250910124212_remove_raw_keys.sql (1.78ms)
2026/05/13 15:43:54 OK   20250923094021_add_team_id_to_snapshots.sql (2.39ms)
2026/05/13 15:43:54 OK   20250923103546_add_snapshot_list_index.sql (2.03ms)
2026/05/13 15:43:54 OK   20250923103614_make_team_id_non_nullable.sql (1.02ms)
2026/05/13 15:43:54 OK   20251009170758_unique_snapshots.sql (1.17ms)
2026/05/13 15:43:54 OK   20251011200438_create_addons_table.sql (8.17ms)
2026/05/13 15:43:54 OK   20251018100653_add_build_version.sql (833.67µs)
2026/05/13 15:43:54 OK   20251026192416_addons_idempotency.sql (1.22ms)
2026/05/13 15:43:54 OK   20251030130958_add_env_index_to_snapshots.sql (1.23ms)
2026/05/13 15:43:54 OK   20251106172810_add_config_to_snapshots.sql (644.15µs)
2026/05/13 15:43:54 OK   20251121101953_build_node_id_nullable.sql (616.21µs)
2026/05/13 15:43:54 OK   20251127000000_add_machine_info_to_env_builds.sql (1.04ms)
2026/05/13 15:43:54 OK   20251216135834_index_snapshots_base_env_id.sql (1.24ms)
2026/05/13 15:43:54 OK   20251217000000_create_public_users_table.sql (4.64ms)
2026/05/13 15:43:54 OK   20251218160000_allow_m_n_builds_with_tags.sql (10.13ms)
2026/05/13 15:43:54 OK   20251218170000_optimize_build_assignment_indexes.sql (2.08ms)
2026/05/13 15:43:54 OK   20260121175429_add_team_slug.sql (11.09ms)
2026/05/13 15:43:54 OK   20260121175430_add_env_aliases_namespace.sql (1.83ms)
2026/05/13 15:43:54 OK   20260127120000_add_env_aliases_uuid_pkey.sql (6.83ms)
2026/05/13 15:43:54 OK   20260129105527_populate_env_aliases_namespace.sql (1.56ms)
2026/05/13 15:43:54 OK   20260204172712_remove_build_assignment_triggers.sql (3.03ms)
2026/05/13 15:43:54 OK   20260210120000_add_env_builds_created_at_index.sql (1.19ms)
2026/05/13 15:43:54 OK   20260210120001_add_env_and_build_source_columns.sql (2.58ms)
2026/05/13 15:43:54 OK   20260210120002_add_status_group_column.sql (4.82ms)
2026/05/13 15:43:54 OK   20260211120000_add_snapshot_templates.sql (3.68ms)
2026/05/13 15:43:54 OK   20260216120000_add_envs_team_id_source_index.sql (2.43ms)
2026/05/13 15:43:54 OK   20260218120000_add_team_id_to_env_builds.sql (8.48ms)
2026/05/13 15:43:54 OK   20260225120000_add_env_builds_team_status_group_index.sql (1.26ms)
2026/05/13 15:43:54 OK   20260228120000_snapshot_template_origin_node.sql (906.64µs)
2026/05/13 15:43:54 OK   20260304120000_volumes.sql (3.83ms)
2026/05/13 15:43:54 OK   20260305120000_add_env_builds_team_active_partial_index.sql (1.47ms)
2026/05/13 15:43:54 OK   20260305130000_create_active_template_builds.sql (3.46ms)
2026/05/13 15:43:54 OK   20260309120000_add_team_sandbox_scheduling_labels.sql (1.05ms)
2026/05/13 15:43:54 OK   20260310120000_add_snapshots_metadata_gin_index.sql (28.26ms)
2026/05/13 15:43:54 OK   20260312120000_fix_snapshots_jsonb_null_metadata.sql (2.56ms)
2026/05/13 15:43:54 OK   20260313120000_fix_snapshots_created_at.sql (1.98ms)
2026/05/13 15:43:54 OK   20260314120000_fix_snapshots_metadata_sql_null_trigger.sql (745.79µs)
2026/05/13 15:43:54 OK   20260316120000_users_teams_uuid_pkey.sql (6.4ms)
2026/05/13 15:43:54 OK   20260316130000_repoint_user_fks_to_public_users.sql (18.29ms)
2026/05/13 15:43:54 OK   20260413120000_active_template_builds_fk_envs.sql (2.89ms)
2026/05/13 15:43:54 OK   20260416120000_remove_user_team_provision_triggers.sql (8.57ms)
2026/05/13 15:43:54 OK   20260423170000_cluster_auth_org_id.sql (1.6ms)
2026/05/13 15:43:54 goose: successfully migrated database to version: 20260423170000
2026/05/13 15:43:54 OK   20000101000000_auth.sql (4.22ms)
2026/05/13 15:43:54 OK   20000101000001_rls_for_migration_table.sql (837.55µs)
2026/05/13 15:43:54 OK   20231124185944_create_schemas_and_tables.sql (20.12ms)
2026/05/13 15:43:54 OK   20231220094836_create_triggers_and_policies.sql (3.55ms)
2026/05/13 15:43:54 OK   20231222181015_add_env_resources.sql (1.12ms)
2026/05/13 15:43:54 OK   20240103104619_add_team_email.sql (1.55ms)
2026/05/13 15:43:54 OK   20240106121919_add_team_ban_fields.sql (1.41ms)
2026/05/13 15:43:54 OK   20240202120312_add_kernel_version.sql (1.35ms)
2026/05/13 15:43:54 OK   20240219190940_add_max_length_hours.sql (1.03ms)
2026/05/13 15:43:54 OK   20240221023613_add_firecracker_version.sql (951.82µs)
2026/05/13 15:43:54 OK   20240221215408_update_firecracker_version.sql (1.45ms)
2026/05/13 15:43:54 OK   20240305221944_remove_tier_resources.sql (1.83ms)
2026/05/13 15:43:54 OK   20240315165236_create_env_builds.sql (6.01ms)
2026/05/13 15:43:54 OK   20240605070918_refactor_triggers_and_policies.sql (5.17ms)
2026/05/13 15:43:54 OK   20240625095352_add_envd_version.sql (1.18ms)
2026/05/13 15:43:54 OK   20240728094137_add_default_team_flag.sql (2.22ms)
2026/05/13 15:43:54 OK   20240909142106_drop_team_policies.sql (1.27ms)
2026/05/13 15:43:54 OK   20241120222814_add_team_api_key_metadata.sql (3.57ms)
2026/05/13 15:43:54 OK   20241121225404_add_team_api_key_id.sql (4.8ms)
2026/05/13 15:43:54 OK   20241127174604_add_env_creator.sql (1.88ms)
2026/05/13 15:43:54 OK   20241206124325_add_user_team_adder.sql (1.76ms)
2026/05/13 15:43:54 OK   20241213142106_create_snapshots.sql (2.33ms)
2026/05/13 15:43:54 OK   20250106142106_remove_team_is_default.sql (1.31ms)
2026/05/13 15:43:54 OK   20250206105106_add_snapshot_constraints.sql (2.85ms)
2026/05/13 15:43:54 OK   20250211160814_add_token_hashes.sql (6.67ms)
2026/05/13 15:43:54 OK   20250306105106_add_indexes.sql (4.79ms)
2026/05/13 15:43:54 OK   20250404151700_add_snapshots_sbx_started_at.sql (4.67ms)
2026/05/13 15:43:54 OK   20250409113306_add_envd_secured_to_snapshot.sql (1.16ms)
2026/05/13 15:43:54 OK   20250506112836_builds_status_index.sql (1.3ms)
2026/05/13 15:43:54 OK   20250507134356_add_max_specs_to_tier.sql (1.48ms)
2026/05/13 15:43:54 OK   20250513111201_tier_fix_max_memory.sql (1.08ms)
2026/05/13 15:43:54 OK   20250522105042_users_teams_add_created_at.sql (953.59µs)
2026/05/13 15:43:54 OK   20250528203546_ready_command.sql (915.31µs)
2026/05/13 15:43:54 OK   20250606204750_optimize_hashed_key_schema.sql (1.66ms)
2026/05/13 15:43:54 OK   20250606213446_deployment_cluster.sql (3.66ms)
2026/05/13 15:43:54 OK   20250624001047_deploy_cluster_policy.sql (773.14µs)
2026/05/13 15:43:54 OK   20250624001048_cluster_for_templates.sql (1.85ms)
2026/05/13 15:43:54 OK   20250624001049_cluster_for_builds.sql (771.16µs)
2026/05/13 15:43:54 OK   20250624232413_add_build_reason.sql (728.23µs)
2026/05/13 15:43:54 OK   20250708135400_snapshots_migrations.sql (1.21ms)
2026/05/13 15:43:54 OK   20250708135401_snapshot_pause_node_id.sql (932.9µs)
2026/05/13 15:43:54 OK   20250714132924_cluster_sandbox_domain.sql (615.37µs)
2026/05/13 15:43:54 OK   20250728085406_add_create_network_flag.sql (690.99µs)
2026/05/13 15:43:54 OK   20250802144312_update_firecracker_to_v1_12_1.sql (788.2µs)
2026/05/13 15:43:54 OK   20250815181502_change_build_reason_to_json.sql (4.72ms)
2026/05/13 15:43:54 OK   20250818114512_auto_pause.sql (1.06ms)
2026/05/13 15:43:54 OK   20250820102103_add_snapshots_indexes.sql (3.58ms)
2026/05/13 15:43:54 OK   20250824185633_build_node_not_nullable.sql (884.5µs)
2026/05/13 15:43:54 OK   20250824185634_snapshot_node_not_nullable.sql (835.13µs)
2026/05/13 15:43:54 OK   20250825100000_remove_default_keys.sql (683.91µs)
2026/05/13 15:43:54 OK   20250825102440_add_hash_indexes.sql (2.49ms)
2026/05/13 15:43:54 OK   20250825102800_hash_existing_keys.sql (3.99ms)
2026/05/13 15:43:54 OK   20250825102900_drop_mask_columns.sql (1.23ms)
2026/05/13 15:43:54 OK   20250901161352_add_concurrent_template_builds_to_tier.sql (1.1ms)
2026/05/13 15:43:54 OK   20250905134524_fix_env_builds_non_null_fields.sql (962.96µs)
2026/05/13 15:43:54 OK   20250910063940_make_raw_keys_nullable.sql (2.08ms)
2026/05/13 15:43:54 OK   20250910072612_access_tokens_id_non_nullable.sql (613.25µs)
2026/05/13 15:43:54 OK   20250910124212_remove_raw_keys.sql (1.82ms)
2026/05/13 15:43:54 OK   20250923094021_add_team_id_to_snapshots.sql (2.24ms)
2026/05/13 15:43:54 OK   20250923103546_add_snapshot_list_index.sql (1.9ms)
2026/05/13 15:43:54 OK   20250923103614_make_team_id_non_nullable.sql (923.74µs)
2026/05/13 15:43:54 OK   20251009170758_unique_snapshots.sql (1.01ms)
2026/05/13 15:43:54 OK   20251011200438_create_addons_table.sql (8.32ms)
2026/05/13 15:43:54 OK   20251018100653_add_build_version.sql (786.48µs)
2026/05/13 15:43:54 OK   20251026192416_addons_idempotency.sql (1.27ms)
2026/05/13 15:43:54 OK   20251030130958_add_env_index_to_snapshots.sql (1.12ms)
2026/05/13 15:43:54 OK   20251106172810_add_config_to_snapshots.sql (805.16µs)
2026/05/13 15:43:54 OK   20251121101953_build_node_id_nullable.sql (629.51µs)
2026/05/13 15:43:54 OK   20251127000000_add_machine_info_to_env_builds.sql (989.27µs)
2026/05/13 15:43:54 OK   20251216135834_index_snapshots_base_env_id.sql (1.19ms)
2026/05/13 15:43:54 OK   20251217000000_create_public_users_table.sql (4.83ms)
2026/05/13 15:43:54 OK   20251218160000_allow_m_n_builds_with_tags.sql (10.01ms)
2026/05/13 15:43:54 OK   20251218170000_optimize_build_assignment_indexes.sql (1.87ms)
2026/05/13 15:43:54 OK   20260121175429_add_team_slug.sql (10.97ms)
2026/05/13 15:43:54 OK   20260121175430_add_env_aliases_namespace.sql (1.62ms)
2026/05/13 15:43:54 OK   20260127120000_add_env_aliases_uuid_pkey.sql (7.16ms)
2026/05/13 15:43:54 OK   20260129105527_populate_env_aliases_namespace.sql (1.63ms)
2026/05/13 15:43:54 OK   20260204172712_remove_build_assignment_triggers.sql (2.97ms)
2026/05/13 15:43:54 OK   20260210120000_add_env_builds_created_at_index.sql (1.18ms)
2026/05/13 15:43:54 OK   20260210120001_add_env_and_build_source_columns.sql (2.6ms)
2026/05/13 15:43:54 OK   20260210120002_add_status_group_column.sql (4.91ms)
2026/05/13 15:43:54 OK   20260211120000_add_snapshot_templates.sql (4.45ms)
2026/05/13 15:43:54 OK   20260216120000_add_envs_team_id_source_index.sql (2.15ms)
2026/05/13 15:43:54 OK   20260218120000_add_team_id_to_env_builds.sql (6.92ms)
2026/05/13 15:43:54 OK   20260225120000_add_env_builds_team_status_group_index.sql (1.2ms)
2026/05/13 15:43:54 OK   20260228120000_snapshot_template_origin_node.sql (782.74µs)
2026/05/13 15:43:54 OK   20260304120000_volumes.sql (3.72ms)
2026/05/13 15:43:54 OK   20260305120000_add_env_builds_team_active_partial_index.sql (1.25ms)
2026/05/13 15:43:54 OK   20260305130000_create_active_template_builds.sql (3.01ms)
2026/05/13 15:43:54 OK   20260309120000_add_team_sandbox_scheduling_labels.sql (1.13ms)
2026/05/13 15:43:54 OK   20260310120000_add_snapshots_metadata_gin_index.sql (26.82ms)
2026/05/13 15:43:54 OK   20260312120000_fix_snapshots_jsonb_null_metadata.sql (2.61ms)
2026/05/13 15:43:54 OK   20260313120000_fix_snapshots_created_at.sql (1.85ms)
2026/05/13 15:43:54 OK   20260314120000_fix_snapshots_metadata_sql_null_trigger.sql (690.9µs)
2026/05/13 15:43:54 OK   20260316120000_users_teams_uuid_pkey.sql (5.44ms)
2026/05/13 15:43:54 OK   20260316130000_repoint_user_fks_to_public_users.sql (18.33ms)
2026/05/13 15:43:54 OK   20260413120000_active_template_builds_fk_envs.sql (2.69ms)
2026/05/13 15:43:54 OK   20260416120000_remove_user_team_provision_triggers.sql (7.43ms)
2026/05/13 15:43:54 OK   20260423170000_cluster_auth_org_id.sql (1.87ms)
2026/05/13 15:43:54 goose: successfully migrated database to version: 20260423170000
2026/05/13 15:43:56 OK   20000101000000_auth.sql (9.78ms)
2026/05/13 15:43:56 OK   20000101000001_rls_for_migration_table.sql (1.96ms)
2026/05/13 15:43:56 OK   20231124185944_create_schemas_and_tables.sql (56.91ms)
2026/05/13 15:43:56 OK   20231220094836_create_triggers_and_policies.sql (4.75ms)
2026/05/13 15:43:56 OK   20231222181015_add_env_resources.sql (1.67ms)
2026/05/13 15:43:56 OK   20240103104619_add_team_email.sql (11.29ms)
2026/05/13 15:43:56 OK   20240106121919_add_team_ban_fields.sql (9.29ms)
2026/05/13 15:43:56 OK   20240202120312_add_kernel_version.sql (13.08ms)
2026/05/13 15:43:56 OK   20240219190940_add_max_length_hours.sql (2.64ms)
2026/05/13 15:43:56 OK   20240221023613_add_firecracker_version.sql (2ms)
2026/05/13 15:43:56 OK   20240221215408_update_firecracker_version.sql (1.7ms)
2026/05/13 15:43:56 OK   20240305221944_remove_tier_resources.sql (9.51ms)
2026/05/13 15:43:56 OK   20240315165236_create_env_builds.sql (13.84ms)
2026/05/13 15:43:56 OK   20240605070918_refactor_triggers_and_policies.sql (14.75ms)
2026/05/13 15:43:56 OK   20240625095352_add_envd_version.sql (14.28ms)
2026/05/13 15:43:56 OK   20240728094137_add_default_team_flag.sql (3.05ms)
2026/05/13 15:43:56 OK   20240909142106_drop_team_policies.sql (1.51ms)
2026/05/13 15:43:56 OK   20241120222814_add_team_api_key_metadata.sql (19.46ms)
2026/05/13 15:43:56 OK   20241121225404_add_team_api_key_id.sql (34.44ms)
2026/05/13 15:43:56 OK   20241127174604_add_env_creator.sql (18.73ms)
2026/05/13 15:43:56 OK   20241206124325_add_user_team_adder.sql (3.78ms)
2026/05/13 15:43:56 OK   20241213142106_create_snapshots.sql (9.63ms)
2026/05/13 15:43:56 OK   20250106142106_remove_team_is_default.sql (5.43ms)
2026/05/13 15:43:56 OK   20250206105106_add_snapshot_constraints.sql (11.31ms)
2026/05/13 15:43:56 OK   20250211160814_add_token_hashes.sql (53.44ms)
2026/05/13 15:43:56 OK   20250306105106_add_indexes.sql (43.5ms)
2026/05/13 15:43:56 OK   20250404151700_add_snapshots_sbx_started_at.sql (18.93ms)
2026/05/13 15:43:56 OK   20250409113306_add_envd_secured_to_snapshot.sql (1.94ms)
2026/05/13 15:43:56 OK   20250506112836_builds_status_index.sql (13.07ms)
2026/05/13 15:43:56 OK   20250507134356_add_max_specs_to_tier.sql (1.92ms)
2026/05/13 15:43:56 OK   20250513111201_tier_fix_max_memory.sql (1.88ms)
2026/05/13 15:43:56 OK   20250522105042_users_teams_add_created_at.sql (1.26ms)
2026/05/13 15:43:56 OK   20250528203546_ready_command.sql (2.91ms)
2026/05/13 15:43:56 OK   20250606204750_optimize_hashed_key_schema.sql (17.99ms)
2026/05/13 15:43:56 OK   20250606213446_deployment_cluster.sql (26.53ms)
2026/05/13 15:43:56 OK   20250624001047_deploy_cluster_policy.sql (2.83ms)
2026/05/13 15:43:56 OK   20250624001048_cluster_for_templates.sql (5ms)
2026/05/13 15:43:56 OK   20250624001049_cluster_for_builds.sql (1.18ms)
2026/05/13 15:43:56 OK   20250624232413_add_build_reason.sql (14.66ms)
2026/05/13 15:43:56 OK   20250708135400_snapshots_migrations.sql (19.01ms)
2026/05/13 15:43:56 OK   20250708135401_snapshot_pause_node_id.sql (2.28ms)
2026/05/13 15:43:56 OK   20250714132924_cluster_sandbox_domain.sql (5.5ms)
2026/05/13 15:43:56 OK   20250728085406_add_create_network_flag.sql (2.74ms)
2026/05/13 15:43:56 OK   20250802144312_update_firecracker_to_v1_12_1.sql (2.57ms)
2026/05/13 15:43:56 OK   20250815181502_change_build_reason_to_json.sql (26.76ms)
2026/05/13 15:43:56 OK   20250818114512_auto_pause.sql (1.53ms)
2026/05/13 15:43:56 OK   20250820102103_add_snapshots_indexes.sql (29.69ms)
2026/05/13 15:43:56 OK   20250824185633_build_node_not_nullable.sql (18.58ms)
2026/05/13 15:43:56 OK   20250824185634_snapshot_node_not_nullable.sql (7.34ms)
2026/05/13 15:43:56 OK   20250825100000_remove_default_keys.sql (7.88ms)
2026/05/13 15:43:56 OK   20250825102440_add_hash_indexes.sql (22.33ms)
2026/05/13 15:43:57 OK   20250825102800_hash_existing_keys.sql (49.96ms)
2026/05/13 15:43:57 OK   20250825102900_drop_mask_columns.sql (16.37ms)
2026/05/13 15:43:57 OK   20250901161352_add_concurrent_template_builds_to_tier.sql (12.22ms)
2026/05/13 15:43:57 OK   20250905134524_fix_env_builds_non_null_fields.sql (1.81ms)
2026/05/13 15:43:57 OK   20250910063940_make_raw_keys_nullable.sql (18.47ms)
2026/05/13 15:43:57 OK   20250910072612_access_tokens_id_non_nullable.sql (4.67ms)
2026/05/13 15:43:57 OK   20250910124212_remove_raw_keys.sql (25.47ms)
2026/05/13 15:43:57 OK   20250923094021_add_team_id_to_snapshots.sql (5.1ms)
2026/05/13 15:43:57 OK   20250923103546_add_snapshot_list_index.sql (9.32ms)
2026/05/13 15:43:57 OK   20250923103614_make_team_id_non_nullable.sql (1.67ms)
2026/05/13 15:43:57 OK   20251009170758_unique_snapshots.sql (14.53ms)
2026/05/13 15:43:57 OK   20251011200438_create_addons_table.sql (25.58ms)
2026/05/13 15:43:57 OK   20251018100653_add_build_version.sql (3.85ms)
2026/05/13 15:43:57 OK   20251026192416_addons_idempotency.sql (4.2ms)
2026/05/13 15:43:57 OK   20251030130958_add_env_index_to_snapshots.sql (13.26ms)
2026/05/13 15:43:57 OK   20251106172810_add_config_to_snapshots.sql (2.73ms)
2026/05/13 15:43:57 OK   20251121101953_build_node_id_nullable.sql (5.66ms)
2026/05/13 15:43:57 OK   20251127000000_add_machine_info_to_env_builds.sql (2.48ms)
2026/05/13 15:43:57 OK   20251216135834_index_snapshots_base_env_id.sql (4.27ms)
2026/05/13 15:43:57 OK   20251217000000_create_public_users_table.sql (28.1ms)
2026/05/13 15:43:57 OK   20251218160000_allow_m_n_builds_with_tags.sql (169.02ms)
2026/05/13 15:43:57 OK   20251218170000_optimize_build_assignment_indexes.sql (21.93ms)
2026/05/13 15:43:57 OK   20260121175429_add_team_slug.sql (29.05ms)
2026/05/13 15:43:57 OK   20260121175430_add_env_aliases_namespace.sql (31.25ms)
2026/05/13 15:43:57 OK   20260127120000_add_env_aliases_uuid_pkey.sql (40.42ms)
2026/05/13 15:43:57 OK   20260129105527_populate_env_aliases_namespace.sql (79.46ms)
2026/05/13 15:43:57 OK   20260204172712_remove_build_assignment_triggers.sql (9.09ms)
2026/05/13 15:43:57 OK   20260210120000_add_env_builds_created_at_index.sql (1.57ms)
2026/05/13 15:43:57 OK   20260210120001_add_env_and_build_source_columns.sql (22.51ms)
2026/05/13 15:43:57 OK   20260210120002_add_status_group_column.sql (41.29ms)
2026/05/13 15:43:57 OK   20260211120000_add_snapshot_templates.sql (50.71ms)
2026/05/13 15:43:57 OK   20260216120000_add_envs_team_id_source_index.sql (18.07ms)
2026/05/13 15:43:57 OK   20260218120000_add_team_id_to_env_builds.sql (68.47ms)
2026/05/13 15:43:57 OK   20260225120000_add_env_builds_team_status_group_index.sql (21.15ms)
2026/05/13 15:43:57 OK   20260228120000_snapshot_template_origin_node.sql (2.35ms)
2026/05/13 15:43:57 OK   20260304120000_volumes.sql (22.84ms)
2026/05/13 15:43:57 OK   20260305120000_add_env_builds_team_active_partial_index.sql (4.5ms)
2026/05/13 15:43:57 OK   20260305130000_create_active_template_builds.sql (35.68ms)
2026/05/13 15:43:57 OK   20260309120000_add_team_sandbox_scheduling_labels.sql (58.01ms)
2026/05/13 15:43:58 OK   20260310120000_add_snapshots_metadata_gin_index.sql (94.25ms)
2026/05/13 15:43:58 OK   20260312120000_fix_snapshots_jsonb_null_metadata.sql (44.44ms)
2026/05/13 15:43:58 OK   20260313120000_fix_snapshots_created_at.sql (19.61ms)
2026/05/13 15:43:58 OK   20260314120000_fix_snapshots_metadata_sql_null_trigger.sql (1.2ms)
2026/05/13 15:43:58 OK   20260316120000_users_teams_uuid_pkey.sql (9.58ms)
2026/05/13 15:43:58 OK   20260316130000_repoint_user_fks_to_public_users.sql (29.4ms)
2026/05/13 15:43:58 OK   20260413120000_active_template_builds_fk_envs.sql (3.87ms)
2026/05/13 15:43:58 OK   20260416120000_remove_user_team_provision_triggers.sql (28.79ms)
2026/05/13 15:43:58 OK   20260423170000_cluster_auth_org_id.sql (2.37ms)
2026/05/13 15:43:58 goose: successfully migrated database to version: 20260423170000
2026/05/13 15:43:58 OK   20000101000000_auth.sql (5.15ms)
2026/05/13 15:43:58 OK   20000101000001_rls_for_migration_table.sql (2.31ms)
2026/05/13 15:43:58 OK   20231124185944_create_schemas_and_tables.sql (61.15ms)
2026/05/13 15:43:58 OK   20231220094836_create_triggers_and_policies.sql (8.16ms)
2026/05/13 15:43:58 OK   20231222181015_add_env_resources.sql (20.4ms)
2026/05/13 15:43:58 OK   20240103104619_add_team_email.sql (2.99ms)
2026/05/13 15:43:58 OK   20240106121919_add_team_ban_fields.sql (1.54ms)
2026/05/13 15:43:58 OK   20240202120312_add_kernel_version.sql (4.91ms)
2026/05/13 15:43:58 OK   20240219190940_add_max_length_hours.sql (1.39ms)
2026/05/13 15:43:58 OK   20240221023613_add_firecracker_version.sql (1.81ms)
2026/05/13 15:43:58 OK   20240221215408_update_firecracker_version.sql (22.37ms)
2026/05/13 15:43:58 OK   20240305221944_remove_tier_resources.sql (6.16ms)
2026/05/13 15:43:58 OK   20240315165236_create_env_builds.sql (9.65ms)
2026/05/13 15:43:58 OK   20240605070918_refactor_triggers_and_policies.sql (7.11ms)
2026/05/13 15:43:58 OK   20240625095352_add_envd_version.sql (3ms)
2026/05/13 15:43:58 OK   20240728094137_add_default_team_flag.sql (17.77ms)
2026/05/13 15:43:58 OK   20240909142106_drop_team_policies.sql (4.46ms)
2026/05/13 15:43:58 OK   20241120222814_add_team_api_key_metadata.sql (27.36ms)
2026/05/13 15:43:58 OK   20241121225404_add_team_api_key_id.sql (13.06ms)
2026/05/13 15:43:58 OK   20241127174604_add_env_creator.sql (13.49ms)
2026/05/13 15:43:58 OK   20241206124325_add_user_team_adder.sql (6.72ms)
2026/05/13 15:43:58 OK   20241213142106_create_snapshots.sql (15.81ms)
2026/05/13 15:43:58 OK   20250106142106_remove_team_is_default.sql (2.18ms)
2026/05/13 15:43:58 OK   20250206105106_add_snapshot_constraints.sql (3.78ms)
2026/05/13 15:43:58 OK   20250211160814_add_token_hashes.sql (21.67ms)
2026/05/13 15:43:58 OK   20250306105106_add_indexes.sql (8.12ms)
2026/05/13 15:43:58 OK   20250404151700_add_snapshots_sbx_started_at.sql (17.8ms)
2026/05/13 15:43:58 OK   20250409113306_add_envd_secured_to_snapshot.sql (3.08ms)
2026/05/13 15:43:58 OK   20250506112836_builds_status_index.sql (3.11ms)
2026/05/13 15:43:58 OK   20250507134356_add_max_specs_to_tier.sql (3.01ms)
2026/05/13 15:43:58 OK   20250513111201_tier_fix_max_memory.sql (1.88ms)
2026/05/13 15:43:58 OK   20250522105042_users_teams_add_created_at.sql (1.81ms)
2026/05/13 15:43:58 OK   20250528203546_ready_command.sql (1.76ms)
2026/05/13 15:43:58 OK   20250606204750_optimize_hashed_key_schema.sql (2.95ms)
2026/05/13 15:43:58 OK   20250606213446_deployment_cluster.sql (6.36ms)
2026/05/13 15:43:58 OK   20250624001047_deploy_cluster_policy.sql (1.42ms)
2026/05/13 15:43:58 OK   20250624001048_cluster_for_templates.sql (2.33ms)
2026/05/13 15:43:58 OK   20250624001049_cluster_for_builds.sql (5.28ms)
2026/05/13 15:43:58 OK   20250624232413_add_build_reason.sql (3.08ms)
2026/05/13 15:43:58 OK   20250708135400_snapshots_migrations.sql (3.69ms)
2026/05/13 15:43:58 OK   20250708135401_snapshot_pause_node_id.sql (17.2ms)
2026/05/13 15:43:58 OK   20250714132924_cluster_sandbox_domain.sql (1.8ms)
2026/05/13 15:43:58 OK   20250728085406_add_create_network_flag.sql (1.69ms)
2026/05/13 15:43:58 OK   20250802144312_update_firecracker_to_v1_12_1.sql (2.66ms)
2026/05/13 15:43:58 OK   20250815181502_change_build_reason_to_json.sql (19.08ms)
2026/05/13 15:43:58 OK   20250818114512_auto_pause.sql (2.85ms)
2026/05/13 15:43:58 OK   20250820102103_add_snapshots_indexes.sql (7.21ms)
2026/05/13 15:43:58 OK   20250824185633_build_node_not_nullable.sql (8.74ms)
2026/05/13 15:43:58 OK   20250824185634_snapshot_node_not_nullable.sql (3.35ms)
2026/05/13 15:43:58 OK   20250825100000_remove_default_keys.sql (1.58ms)
2026/05/13 15:43:58 OK   20250825102440_add_hash_indexes.sql (4.31ms)
2026/05/13 15:43:58 OK   20250825102800_hash_existing_keys.sql (21.52ms)
2026/05/13 15:43:58 OK   20250825102900_drop_mask_columns.sql (2.38ms)
2026/05/13 15:43:58 OK   20250901161352_add_concurrent_template_builds_to_tier.sql (3.73ms)
2026/05/13 15:43:58 OK   20250905134524_fix_env_builds_non_null_fields.sql (1.47ms)
2026/05/13 15:43:58 OK   20250910063940_make_raw_keys_nullable.sql (3.81ms)
2026/05/13 15:43:58 OK   20250910072612_access_tokens_id_non_nullable.sql (22.92ms)
2026/05/13 15:43:58 OK   20250910124212_remove_raw_keys.sql (2.72ms)
2026/05/13 15:43:58 OK   20250923094021_add_team_id_to_snapshots.sql (4.76ms)
2026/05/13 15:43:58 OK   20250923103546_add_snapshot_list_index.sql (14.62ms)
2026/05/13 15:43:58 OK   20250923103614_make_team_id_non_nullable.sql (2.17ms)
2026/05/13 15:43:58 OK   20251009170758_unique_snapshots.sql (28.99ms)
2026/05/13 15:43:58 OK   20251011200438_create_addons_table.sql (35.74ms)
2026/05/13 15:43:58 OK   20251018100653_add_build_version.sql (6.83ms)
2026/05/13 15:43:58 OK   20251026192416_addons_idempotency.sql (2.79ms)
2026/05/13 15:43:58 OK   20251030130958_add_env_index_to_snapshots.sql (2.41ms)
2026/05/13 15:43:58 OK   20251106172810_add_config_to_snapshots.sql (4.23ms)
2026/05/13 15:43:58 OK   20251121101953_build_node_id_nullable.sql (4.08ms)
2026/05/13 15:43:58 OK   20251127000000_add_machine_info_to_env_builds.sql (2.97ms)
2026/05/13 15:43:58 OK   20251216135834_index_snapshots_base_env_id.sql (15.6ms)
2026/05/13 15:43:58 OK   20251217000000_create_public_users_table.sql (18.91ms)
2026/05/13 15:43:58 OK   20251218160000_allow_m_n_builds_with_tags.sql (47.74ms)
2026/05/13 15:43:58 OK   20251218170000_optimize_build_assignment_indexes.sql (9.9ms)
2026/05/13 15:43:58 OK   20260121175429_add_team_slug.sql (59.15ms)
2026/05/13 15:43:58 OK   20260121175430_add_env_aliases_namespace.sql (16.11ms)
2026/05/13 15:43:59 OK   20260127120000_add_env_aliases_uuid_pkey.sql (39.04ms)
2026/05/13 15:43:59 OK   20260129105527_populate_env_aliases_namespace.sql (13.67ms)
2026/05/13 15:43:59 OK   20260204172712_remove_build_assignment_triggers.sql (3.87ms)
2026/05/13 15:43:59 OK   20260210120000_add_env_builds_created_at_index.sql (15.6ms)
2026/05/13 15:43:59 OK   20260210120001_add_env_and_build_source_columns.sql (18.02ms)
2026/05/13 15:43:59 OK   20260210120002_add_status_group_column.sql (19.8ms)
View the full list of 16 ❄️ flaky test(s) 
github.com/e2b-dev/infra/tests/integration/internal/tests/api/metrics::TestTeamMetrics

Flake rate in main: 70.28% (Passed 151 times, Failed 357 times)

Stack Traces | 1.38s run time 
=== RUN   TestTeamMetrics
=== PAUSE TestTeamMetrics
=== CONT  TestTeamMetrics
    team_metrics_test.go:61: 
        	Error Trace:	.../api/metrics/team_metrics_test.go:61
        	Error:      	Should be true
        	Test:       	TestTeamMetrics
        	Messages:   	MaxConcurrentSandboxes should be >= 0
--- FAIL: TestTeamMetrics (1.38s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestUpdateNetworkConfig

Flake rate in main: 76.51% (Passed 159 times, Failed 518 times)

Stack Traces | 202s run time 
=== RUN   TestUpdateNetworkConfig
=== PAUSE TestUpdateNetworkConfig
=== CONT  TestUpdateNetworkConfig
    sandbox_network_out_test.go:30: Building custom template for network egress tests...
    template.go:44: network-egress-test: [info] Building template lhxeflu93i7rheyj9u3s/5ed962e0-09ed-4044-b4d7-38b18ae176c3
    template.go:44: network-egress-test: [info] [base] FROM ubuntu:22.04 [ffd709f131f42dfab282de47a91dd2c139e900c1c11fc574b49b517a05ef0a32]
    template.go:44: network-egress-test: [info] Base Docker image size: 30 MB
    template.go:44: network-egress-test: [info] Creating file system and pulling Docker image
    template.go:44: network-egress-test: [info] Uncompressing layer sha256:f63eb04151bcac21ad049f8d781b97b219aba392c5457907f8f3e88e43eb48ec 30 MB
    template.go:44: network-egress-test: [info] Uncompressing layer sha256:4fbe8582ef6b0f68beb2aa89e809795a1365578ffe54bae2d5f312cf1fe137f3 12 MB
    template.go:44: network-egress-test: [info] Uncompressing layer sha256:8c4b1b28875140ed3abacaf16ad0d696f6bef912f52d2148f261a23e3349465b 168 B
    template.go:44: network-egress-test: [info] Layers extracted
    template.go:44: network-egress-test: [info] Root filesystem structure: bin, boot, dev, etc, home, lib, lib32, lib64, libx32, media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, var
    template.go:44: network-egress-test: [info] Provisioning sandbox template
    template.go:44: network-egress-test: [info] Provisioning was successful, cleaning up
    template.go:44: network-egress-test: [info] Sandbox template provisioned
    template.go:44: network-egress-test: [info] [base] DEFAULT USER user [90bdd4afa342293c931373351bf578872dec9179214ba3e8bf9edba311466213]
    template.go:44: network-egress-test: [info] [builder 1/1] RUN sudo apt-get update && sudo apt-get install -y curl iputils-ping dnsutils openssh-client gnupg && sudo rm -rf .../lib/apt/lists/* [2463a9871c5acb096dc00a6b03e4965da11a9e03c5f0618cbf556435df9fae17]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Reading package lists...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Reading package lists...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Building dependency tree...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Reading state information...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: curl is already the newest version (7.81.0-1ubuntu1.24).
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: openssh-client is already the newest version (1:8.9p1-3ubuntu0.15).
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: openssh-client set to manually installed.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: The following additional packages will be installed:
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: bind9-dnsutils bind9-host bind9-libs dirmngr gnupg-l10n gnupg-utils gpg
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libassuan0 libicu70
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: libksba8 liblmdb0 libmaxminddb0 libnpth0 libuv1 libxml2 pinentry-curses
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Suggested packages:
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: scdaemon mmdb-bin pinentry-doc
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: The following NEW packages will be installed:
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: bind9-dnsutils bind9-host bind9-libs dirmngr dnsutils gnupg gnupg-l10n
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: iputils-ping libassuan0 libicu70 libksba8 liblmdb0 libmaxminddb0 libnpth0
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: libuv1 libxml2 pinentry-curses
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: 0 upgraded, 24 newly installed, 0 to remove and 2 not upgraded.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Need to get 15.3 MB of archives.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: After this operation, 48.2 MB of additional disk space will be used.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 iputils-ping amd64 3:20211215-1ubuntu0.1 [43.0 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libicu70 amd64 70.1-2 [10.6 MB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libxml2 amd64 2.9.13+dfsg-1ubuntu0.11 [765 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblmdb0 amd64 0.9.24-1build2 [47.6 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libmaxminddb0 amd64 1.5.2-1build2 [24.7 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuv1 amd64 1.43.0-1ubuntu0.1 [92.7 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 bind9-libs amd64 1:9.18.39-0ubuntu0.22.04.3 [1262 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 bind9-host amd64 1:9.18.39-0ubuntu0.22.04.3 [52.5 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 bind9-dnsutils amd64 1:9.18.39-0ubuntu0.22.04.3 [158 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libassuan0 amd64 2.5.5-1build1 [38.2 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgconf amd64 2.2.27-3ubuntu2.5 [94.3 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libksba8 amd64 1.6.0-2ubuntu0.2 [119 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libnpth0 amd64 1.6-3build2 [8664 B]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dirmngr amd64 2.2.27-3ubuntu2.5 [293 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 dnsutils all 1:9.18.39-0ubuntu0.22.04.3 [3924 B]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-l10n all 2.2.27-3ubuntu2.5 [54.5 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-utils amd64 2.2.27-3ubuntu2.5 [309 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg amd64 2.2.27-3ubuntu2.5 [519 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 pinentry-curses amd64 1.1.1-1build2 [34.4 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-agent amd64 2.2.27-3ubuntu2.5 [209 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-client amd64 2.2.27-3ubuntu2.5 [62.7 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-server amd64 2.2.27-3ubuntu2.5 [57.6 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgsm amd64 2.2.27-3ubuntu2.5 [197 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg all 2.2.27-3ubuntu2.5 [315 kB]
    template.go:44: network-egress-test: [info] [builder 1/1] [stderr]: debconf: delaying package configuration, since apt-utils is not installed
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Fetched 15.3 MB in 0s (44.9 MB/s)
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package iputils-ping.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 75%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 80%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 85%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 90%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 95%
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: (Reading database ... 100%(Reading database ... 12395 files and directories currently installed.)
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../00-iputils-ping_3%3a20211215-1ubuntu0.1_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking iputils-ping (3:20211215-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libicu70:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../01-libicu70_70.1-2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libicu70:amd64 (70.1-2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libxml2:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../02-libxml2_2.9.13+dfsg-1ubuntu0.11_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libxml2:amd64 (2.9.13+dfsg-1ubuntu0.11) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package liblmdb0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../03-liblmdb0_0.9.24-1build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking liblmdb0:amd64 (0.9.24-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libmaxminddb0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../04-libmaxminddb0_1.5.2-1build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libmaxminddb0:amd64 (1.5.2-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libuv1:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../05-libuv1_1.43.0-1ubuntu0.1_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libuv1:amd64 (1.43.0-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package bind9-libs:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../06-bind9-libs_1%3a9.18.39-0ubuntu0.22.04.3_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking bind9-libs:amd64 (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package bind9-host.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../07-bind9-host_1%3a9.18.39-0ubuntu0.22.04.3_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking bind9-host (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package bind9-dnsutils.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../08-bind9-dnsutils_1%3a9.18.39-0ubuntu0.22.04.3_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking bind9-dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libassuan0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../09-libassuan0_2.5.5-1build1_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libassuan0:amd64 (2.5.5-1build1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpgconf.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../10-gpgconf_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpgconf (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libksba8:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../11-libksba8_1.6.0-2ubuntu0.2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libksba8:amd64 (1.6.0-2ubuntu0.2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package libnpth0:amd64.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../12-libnpth0_1.6-3build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking libnpth0:amd64 (1.6-3build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package dirmngr.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../13-dirmngr_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking dirmngr (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package dnsutils.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../14-dnsutils_1%3a9.18.39-0ubuntu0.22.04.3_all.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gnupg-l10n.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../15-gnupg-l10n_2.2.27-3ubuntu2.5_all.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gnupg-l10n (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gnupg-utils.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../16-gnupg-utils_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gnupg-utils (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../17-gpg_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package pinentry-curses.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../18-pinentry-curses_1.1.1-1build2_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking pinentry-curses (1.1.1-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg-agent.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../19-gpg-agent_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg-agent (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg-wks-client.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../20-gpg-wks-client_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg-wks-client (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpg-wks-server.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../21-gpg-wks-server_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpg-wks-server (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gpgsm.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../22-gpgsm_2.2.27-3ubuntu2.5_amd64.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gpgsm (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Selecting previously unselected package gnupg.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Preparing to unpack .../23-gnupg_2.2.27-3ubuntu2.5_all.deb ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Unpacking gnupg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libksba8:amd64 (1.6.0-2ubuntu0.2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up liblmdb0:amd64 (0.9.24-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libmaxminddb0:amd64 (1.5.2-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libnpth0:amd64 (1.6-3build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libassuan0:amd64 (2.5.5-1build1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libuv1:amd64 (1.43.0-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gnupg-l10n (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpgconf (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up iputils-ping (3:20211215-1ubuntu0.1) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libicu70:amd64 (70.1-2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gnupg-utils (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up pinentry-curses (1.1.1-1build2) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg-agent (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent-browser.socket → .../systemd/user/gpg-agent-browser.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent-extra.socket → .../systemd/user/gpg-agent-extra.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent-ssh.socket → .../systemd/user/gpg-agent-ssh.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/gpg-agent.socket → .../systemd/user/gpg-agent.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpgsm (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up dirmngr (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Created symlink .../systemd/user/sockets.target.wants/dirmngr.socket → .../systemd/user/dirmngr.socket.
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg-wks-server (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up libxml2:amd64 (2.9.13+dfsg-1ubuntu0.11) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gpg-wks-client (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up bind9-libs:amd64 (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up gnupg (2.2.27-3ubuntu2.5) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up bind9-host (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up bind9-dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Setting up dnsutils (1:9.18.39-0ubuntu0.22.04.3) ...
    template.go:44: network-egress-test: [info] [builder 1/1] [stdout]: Processing triggers for libc-bin (2.35-0ubuntu3.13) ...
    template.go:44: network-egress-test: [info] [finalize] Finalizing template build [678d8c6396745e76c1e6fcb6ec74a06eb1cc3bffc1101dbaec5c89848fb75045]
    template.go:44: network-egress-test: [info] [optimize] Optimizing template [027a14840d213f9902f523344074c7f19992129998b9867f8c379d0012f44fe9]
    template.go:44: network-egress-test: [info] Build finished, took 2m40s
    sandbox_network_out_test.go:32: Build completed successfully
    sandbox_network_out_test.go:49: Network test template built: lhxeflu93i7rheyj9u3s
--- FAIL: TestUpdateNetworkConfig (202.28s)

github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false

Flake rate in main: 76.99% (Passed 153 times, Failed 512 times)

Stack Traces | 7.18s run time 
=== RUN   TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false
Executing command curl in sandbox izwgw646011kt3o6nf5xc
    sandbox_network_update_test.go:372: Command [curl] output: event:{start:{pid:1364}}
    sandbox_network_update_test.go:372: Command [curl] output: event:{end:{exit_code:35  exited:true  status:"exit status 35"  error:"exit status 35"}}
Executing command curl in sandbox izwgw646011kt3o6nf5xc
    sandbox_network_update_test.go:372: Command [curl] output: event:{start:{pid:1365}}
    sandbox_network_update_test.go:372: Command [curl] output: event:{end:{exit_code:35  exited:true  status:"exit status 35"  error:"exit status 35"}}
Executing command curl in sandbox izwgw646011kt3o6nf5xc
    sandbox_network_update_test.go:391: Command [curl] output: event:{start:{pid:1366}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{data:{stdout:"HTTP/2 302 \r\nx-content-type-options: nosniff\r\nlocation: https://dns.google/\r\ndate: Wed, 13 May 2026 15:54:57 GMT\r\ncontent-type: text/html; charset=UTF-8\r\nserver: HTTP server (unknown)\r\ncontent-length: 216\r\nx-xss-protection: 0\r\nx-frame-options: SAMEORIGIN\r\nalt-svc: h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000\r\n\r\n"}}
    sandbox_network_update_test.go:391: Command [curl] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_network_update_test.go:391: Command [curl] completed successfully in sandbox izwgw646011kt3o6nf5xc
    sandbox_network_update_test.go:391: 
        	Error Trace:	.../api/sandboxes/sandbox_network_out_test.go:74
        	            				.../api/sandboxes/sandbox_network_update_test.go:60
        	            				.../api/sandboxes/sandbox_network_update_test.go:391
        	Error:      	An error is expected but got nil.
        	Test:       	TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false
        	Messages:   	https://8.8.8.8 should be blocked
--- FAIL: TestUpdateNetworkConfig/pause_resume_preserves_allow_internet_access_false (7.18s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost

Flake rate in main: 56.91% (Passed 265 times, Failed 350 times)

Stack Traces | 0s run time 
=== RUN   TestBindLocalhost
=== PAUSE TestBindLocalhost
=== CONT  TestBindLocalhost
--- FAIL: TestBindLocalhost (0.00s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_0_0_0_0

Flake rate in main: 62.94% (Passed 149 times, Failed 253 times)

Stack Traces | 9.77s run time 
=== RUN   TestBindLocalhost/bind_0_0_0_0
=== PAUSE TestBindLocalhost/bind_0_0_0_0
=== CONT  TestBindLocalhost/bind_0_0_0_0
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1265}}
Executing command python in sandbox i3fgn8px3f8cikkz65h7z
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_0_0_0_0
        	Messages:   	Unexpected status code 502 for bind address 0.0.0.0
--- FAIL: TestBindLocalhost/bind_0_0_0_0 (9.77s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_::1

Flake rate in main: 64.61% (Passed 149 times, Failed 272 times)

Stack Traces | 8.38s run time 
=== RUN   TestBindLocalhost/bind_::1
=== PAUSE TestBindLocalhost/bind_::1
=== CONT  TestBindLocalhost/bind_::1
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1265}}
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_::1
        	Messages:   	Unexpected status code 502 for bind address ::1
--- FAIL: TestBindLocalhost/bind_::1 (8.38s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestBindLocalhost/bind_localhost

Flake rate in main: 64.52% (Passed 149 times, Failed 271 times)

Stack Traces | 9.2s run time 
=== RUN   TestBindLocalhost/bind_localhost
=== PAUSE TestBindLocalhost/bind_localhost
=== CONT  TestBindLocalhost/bind_localhost
Executing command python in sandbox i2dyl0ulblex8ev85d2f3
    localhost_bind_test.go:69: Command [python] output: event:{start:{pid:1265}}
    localhost_bind_test.go:90: 
        	Error Trace:	.../tests/envd/localhost_bind_test.go:90
        	Error:      	Not equal: 
        	            	expected: 200
        	            	actual  : 502
        	Test:       	TestBindLocalhost/bind_localhost
        	Messages:   	Unexpected status code 502 for bind address localhost
--- FAIL: TestBindLocalhost/bind_localhost (9.20s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir

Flake rate in main: 53.75% (Passed 191 times, Failed 222 times)

Stack Traces | 0.38s run time 
=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
Executing command update-ca-certificates in sandbox i8j4wbck2ypww8zil44o7 (user: root)
--- FAIL: TestListDir (0.38s)
Executing command python in sandbox izj66i3u7xtd6victx4o2

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory

Flake rate in main: 57.79% (Passed 149 times, Failed 204 times)

Stack Traces | 0.02s run time 
=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:97: 
        	Error Trace:	.../tests/envd/filesystem_test.go:97
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory

Flake rate in main: 57.79% (Passed 149 times, Failed 204 times)

Stack Traces | 0.02s run time 
=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:97: 
        	Error Trace:	.../tests/envd/filesystem_test.go:97
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)

Flake rate in main: 57.79% (Passed 149 times, Failed 204 times)

Stack Traces | 0.02s run time 
=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:97: 
        	Error Trace:	.../tests/envd/filesystem_test.go:97
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.02s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 57.79% (Passed 149 times, Failed 204 times)

Stack Traces | 0.01s run time 
=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:97: 
        	Error Trace:	.../tests/envd/filesystem_test.go:97
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 66.24% (Passed 159 times, Failed 312 times)

Stack Traces | 84.2s run time 
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:26: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (84.16s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 67.25% (Passed 149 times, Failed 306 times)

Stack Traces | 28.6s run time 
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1283}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory before tmpfs mount: 186 MB\nFree memory before tmpfs mount: 798 MB\nMemory to use in integrity test (80% of free, min 64MB): 638 MB\n"}}
Executing command bash in sandbox i80vi1trmpp9i4rfjzgi8 (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"638+0 records in\n638+0 records out\n668991488 bytes (669 MB, 638 MiB) copied, 2.96794 s, 225 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"C"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"o"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 830 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"and being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=638\"\n\tUser time (seconds): 0.00\n\tSystem time (seconds): 2.94\n\tPercent of CPU this job got: 99%\n\tElapsed (wall clock) time (h:mm:ss or m:ss): 0:02.97\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverage total size (kbytes): 0\n\tMaximum resident set size (kbytes): 2628\n\tAverage resident set size (kbytes): 0\n\tMajor (requiring I/O) page faults: 2\n\tMinor (reclaiming a frame) page faults: 345\n\tVoluntary context switches: 3\n\tInvoluntary context switches: 54\n\tSwaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0\n\tSocket messages received: 0\n\tSignals delivered: 0\n\tPage size (bytes): 4096\n\tExit status: 0\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox iozy5zpdb8n2xfksqterc
Executing command bash in sandbox iozy5zpdb8n2xfksqterc (user: root)
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{start:{pid:1299}}
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{data:{stdout:"12bc864714ebee10754794df20911cc3d320ccbf8fa99590b6ddf1f3a652a59e\n"}}
    sandbox_memory_integrity_test.go:74: Command [bash] output: event:{end:{exited:true status:"exit status 0"}}
    sandbox_memory_integrity_test.go:74: Command [bash] completed successfully in sandbox iozy5zpdb8n2xfksqterc
Executing command bash in sandbox iozy5zpdb8n2xfksqterc (user: root)
    sandbox_memory_integrity_test.go:99: Command [bash] output: event:{start:{pid:1302}}
    sandbox_memory_integrity_test.go:100: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:100
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox iozy5zpdb8n2xfksqterc: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (28.57s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestSandboxAutoResumeViaProxy

Flake rate in main: 55.16% (Passed 152 times, Failed 187 times)

Stack Traces | 21.3s run time 
=== RUN   TestSandboxAutoResumeViaProxy
=== PAUSE TestSandboxAutoResumeViaProxy
=== CONT  TestSandboxAutoResumeViaProxy
    auto_resume_test.go:97: [Status code: 502] Response body: {"sandboxId":"irxp1hxlsjgwcjjqanjwd","message":"The sandbox is running but port is not open","port":8000,"code":502}
    auto_resume_test.go:97: [Status code: 502] Response body: {"sandboxId":"irxp1hxlsjgwcjjqanjwd","message":"The sandbox is running but port is not open","port":8000,"code":502}
    auto_resume_test.go:97: [Status code: 502] Response body: {"sandboxId":"irxp1hxlsjgwcjjqanjwd","message":"The sandbox is running but port is not open","port":8000,"code":502}
    auto_resume_test.go:97: [Status code: 502] Response body: {"sandboxId":"irxp1hxlsjgwcjjqanjwd","message":"The sandbox is running but port is not open","port":8000,"code":502}
    auto_resume_test.go:116: 
        	Error Trace:	.../tests/proxies/auto_resume_test.go:116
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestSandboxAutoResumeViaProxy
--- FAIL: TestSandboxAutoResumeViaProxy (21.32s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestSandboxProxyWorkingPort

Flake rate in main: 54.76% (Passed 152 times, Failed 184 times)

Stack Traces | 11.5s run time 
=== RUN   TestSandboxProxyWorkingPort
=== PAUSE TestSandboxProxyWorkingPort
=== CONT  TestSandboxProxyWorkingPort
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Executing command ls in sandbox ikzqhotkq9ozfvmcnhiek
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Executing command ls in sandbox ijdsx2frizm7gjl2ay3im
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Executing command ls in sandbox ikzqhotkq9ozfvmcnhiek
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Executing command ls in sandbox ijdsx2frizm7gjl2ay3im
    closed_port_test.go:63: Error: Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    closed_port_test.go:64: 
        	Error Trace:	.../tests/proxies/closed_port_test.go:64
        	Error:      	Expected value not to be nil.
        	Test:       	TestSandboxProxyWorkingPort
--- FAIL: TestSandboxProxyWorkingPort (11.51s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

gemini-code-assist[bot]
gemini-code-assist Bot reviewed May 13, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Redis sandbox cleaner to proactively remove expired sandboxes that exceed the StaleCutoff threshold, ensuring cleaner storage state. The reviewer identified a potential race condition in the new evictExpired logic, noting that storage.Remove is called without verifying the current sandbox state, which could lead to premature deletion if a sandbox's lifetime is extended concurrently.

Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go
@jakubno jakubno marked this pull request as ready for review May 13, 2026 13:41
claude[bot]
claude Bot reviewed May 13, 2026
View reviewed changes
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small, well-tested addition with the race window already addressed by the StaleCutoff guard.

Extended reasoning...

Overview

This PR extends the Redis sandbox cleaner to also remove sandboxes whose EndTime is older than sandbox.StaleCutoff, in addition to its existing job of pruning orphaned index entries. The change is ~20 lines in cleaner.go plus two new focused tests in cleaner_test.go covering both the eviction path and the StaleCutoff race-guard.

Security risks

None. This is internal cleanup logic operating on Redis-backed sandbox state. Remove is idempotent and the operation is scoped to entries already classified as expired by ExpiredItems. Multi-pod safety is preserved (the existing idempotency contract still holds).

Level of scrutiny

Low-to-moderate. It is operational/cleanup code rather than auth or data-handling, the change is small and self-contained, and the new behavior is gated by an explicit StaleCutoff window that intentionally leaves recently-expired sandboxes to the evictor to avoid a race. The PR author addressed gemini-code-assist's race concern in the timeline by noting Redis is used as a write-through cache and the StaleCutoff window provides sufficient buffer.

Other factors

The two inline nits flagged (a stale doc comment and a misleading 'after ErrNotFound' log message) are comment-only and do not affect runtime behavior — worth fixing but not blocking. Test coverage for the new behavior is good: both the eviction path and the negative guard (recently-expired survives) are exercised.

Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go
Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go Outdated
@jakubno
chore(api): clarify cleaner comments and warning message
f4e8023 
Address PR feedback: the inline comment in RunOnce contradicted what
evictExpired actually does, and the warning message referenced a
non-existent ErrNotFound precondition.
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 13, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4e80239d6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go
sitole
sitole reviewed May 13, 2026
View reviewed changes
Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go Outdated
sitole
sitole reviewed May 13, 2026
View reviewed changes
Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go Outdated
sitole
sitole approved these changes May 13, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@sitole sitole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just details

@jakubno
chore(api): address cleaner review feedback
e33eea1 
- drop items.go line reference from RunOnce comment
- promote stale-remove failure log from Warn to Error
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 13, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e33eea1afa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go
@jakubno jakubno force-pushed the cleaner-evict-expired branch from 2de6e99 to cadc7dd Compare May 13, 2026 15:42
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed May 13, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cadc7ddea5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/api/internal/sandbox/storage/redis/cleaner.go
@jakubno jakubno merged commit 0f12a54 into main May 13, 2026
53 of 54 checks passed
@jakubno jakubno deleted the cleaner-evict-expired branch May 13, 2026 16:06
ValentaTomas pushed a commit that referenced this pull request May 13, 2026
@jakubno @ValentaTomas
feat(api): cleaner evicts sandboxes older than StaleCutoff (#2640)
1e1fbe8 
The Redis cleaner now removes sandboxes whose EndTime is older than
StaleCutoff, in addition to pruning orphaned index entries. Recently
expired sandboxes are left to the evictor to avoid races.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@claude claude[bot] claude[bot] left review comments

@sitole sitole sitole approved these changes

@ValentaTomas ValentaTomas Awaiting requested review from ValentaTomas ValentaTomas is a code owner

@dobrac dobrac Awaiting requested review from dobrac dobrac is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jakubno @sitole @bchalios