Skip to content

fix: prevent hang on shutdown#1783

Merged
dobrac merged 1 commit intomainfrom
fix/prevent-hang-on-shutdown
Jan 27, 2026
Merged

fix: prevent hang on shutdown#1783
dobrac merged 1 commit intomainfrom
fix/prevent-hang-on-shutdown

Conversation

@dobrac
Copy link
Copy Markdown
Contributor

@dobrac dobrac commented Jan 27, 2026

Try to prevent hang here:

2026-01-27T10:10:44.082Z  INFO  Shutdown signal received  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:10:44.083Z  INFO  Service status changed  {"service": "template-manager", "internal": true, "pid": 1943188, "status": "Draining"}
2026-01-27T10:10:59.084Z  INFO  Waiting for all build jobs to finish  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:10:59.084Z  INFO  Waiting for consumers to check build status  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:11:14.085Z  INFO  Template build queue cleaned  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:11:14.085Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "grpc server", "forced": false}
2026-01-27T10:11:14.085Z  INFO  Shutting down grpc server  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:11:14.088Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "http server", "forced": false}
2026-01-27T10:11:14.089Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "cmux server", "forced": false}
2026-01-27T10:11:14.089Z  INFO  Shutting down cmux server  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:11:14.089Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "template server", "forced": false}
2026-01-27T10:11:14.089Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "hyperloop server", "forced": false}
2026-01-27T10:11:14.090Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "template manager sandbox logger", "forced": false}
2026-01-27T10:11:14.090Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "network pool", "forced": false}
2026-01-27T10:11:14.090Z  INFO  Closing network pool  {"service": "template-manager", "internal": true, "pid": 1943188}
2026-01-27T10:11:21.702Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "device pool", "forced": false}
2026-01-27T10:11:21.702Z  INFO  Closing device pool  {"service": "template-manager", "internal": true, "pid": 1943188, "used_slots": 65}
2026-01-27T10:11:21.703Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "tcp egress firewall", "forced": false}
2026-01-27T10:11:21.703Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "sandbox proxy", "forced": false}
2026-01-27T10:11:21.704Z  INFO  closing  {"service": "template-manager", "internal": true, "pid": 1943188, "service": "sandbox observer", "forced": false}

Note

Improves shutdown reliability and observability.

  • Adds 10s timeout when shutting down sandbox metrics meter exporter to avoid hangs
  • Tracks active template builds using an atomic counter; increments/decrements around build goroutines and logs active_builds during Wait
  • Logs start of drain phase with sandbox_count for better shutdown diagnostics

Written by Cursor Bugbot for commit d5cc2b6. This will update automatically on new commits. Configure here.

@dobrac dobrac added the bug Something isn't working label Jan 27, 2026
@dobrac dobrac enabled auto-merge (squash) January 27, 2026 10:50
@dobrac dobrac merged commit a4ddce5 into main Jan 27, 2026
31 checks passed
@dobrac dobrac deleted the fix/prevent-hang-on-shutdown branch January 27, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants