Skip to content

fix: [sc-89434] Close msgQueue per reconnect cycle to prevent goroutine leak#60

Merged
mlataza merged 1 commit into
mainfrom
bug/sc-89434/goroutine-leak-msgqueue-never-closed
Apr 27, 2026
Merged

fix: [sc-89434] Close msgQueue per reconnect cycle to prevent goroutine leak#60
mlataza merged 1 commit into
mainfrom
bug/sc-89434/goroutine-leak-msgqueue-never-closed

Conversation

@mlataza

@mlataza mlataza commented Apr 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Worker goroutines leaked on every MQTT reconnect cycle because msgQueue was created once before the reconnection loop and never closed, so workers had no signal to exit between cycles.
  • msgQueue creation and the worker pool are now moved inside the reconnection loop so each connection attempt gets a fresh queue and a fresh sync.WaitGroup.
  • A drainWorkers() closure (close(msgQueue) + wg.Wait()) is called at every exit point before the loop resets: connection lost, subscribe failure, connect failure, stop signal, and MQTT options error. Workers drain any buffered messages and exit cleanly before the next cycle begins.

Changes

  • cmd/agent_smith/service.go: removed pre-loop worker/queue setup; added per-iteration worker pool with sync.WaitGroup and drainWorkers() calls at all five exit/continue paths; added "sync" import.
  • cmd/agent_smith/message_queue_test.go: added TestWorkerPool_NoLeakAcrossReconnectCycles — runs 5 simulated connection cycles, each creating workers, enqueuing a message, closing the queue, and waiting; asserts goroutine count does not grow beyond baseline + 3.

Test plan

  • Run go test ./cmd/agent_smith/... — all tests pass including TestWorkerPool_NoLeakAcrossReconnectCycles
  • Deploy to a machine with intermittent network connectivity, simulate 5+ MQTT disconnection/reconnection cycles, and confirm goroutine count remains stable via go tool pprof or /debug/pprof/goroutine
  • Send a command after reconnection and confirm the result is posted back correctly

…ne leak

Move msgQueue creation and worker pool into each iteration of the MQTT
reconnection loop. Add drainWorkers() (close + WaitGroup.Wait()) at every
exit point — connection lost, subscribe failure, connect failure, stop
signal, and options error — so worker goroutines from each cycle drain and
exit before the next cycle begins.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mlataza
mlataza merged commit ca1514a into main Apr 27, 2026
21 checks passed
@mlataza
mlataza deleted the bug/sc-89434/goroutine-leak-msgqueue-never-closed branch April 27, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant