CI failure
The failure occurred in the same time window as the CI alert. The complete 2,037-line failing-job log was available and reviewed.
Failure evidence
The coderd/x/nats test process crashed, causing 11 tests/subtests to be reported as (unknown). These are crash fallout, not 11 independent test failures.
=== FAIL: coderd/x/nats TestPubsub_ClusterTLS/RotationOverlap (unknown)
panic: send on closed channel
goroutine 1439 [running]:
github.com/nats-io/nats-server/v2/server.(*Server).reloadAuthorization(...)
.../github.com/nats-io/nats-server/v2@v2.14.2/server/reload.go:2298
github.com/nats-io/nats-server/v2/server.(*Server).applyOptions(...)
.../github.com/nats-io/nats-server/v2@v2.14.2/server/reload.go:2050
github.com/nats-io/nats-server/v2/server.(*Server).ReloadOptions(...)
.../github.com/nats-io/nats-server/v2@v2.14.2/server/reload.go:1474
github.com/coder/coder/v2/coderd/x/nats.(*Pubsub).setPeerAddresses(...)
/home/runner/work/coder/coder/coderd/x/nats/cluster.go:131
github.com/coder/coder/v2/coderd/x/nats.(*Pubsub).runPeerRefresh(...)
/home/runner/work/coder/coder/coderd/x/nats/cluster.go:84
created by github.com/coder/coder/v2/coderd/x/nats.New in goroutine 361
/home/runner/work/coder/coder/coderd/x/nats/pubsub.go:420
Immediately before the panic, test cleanup logged nats server shut down. Other affected (unknown) results include TestPubsub_ClusterTLS, Mesh, SANMismatch, MixedTLSAndPlaintext, WrongCARejected, TestPubsub_ClusterTLS_CacheSwap, and TestPubsub_ClusterTLS_RealCA.
Error analysis / root cause
Classification: process crash caused by a code-level shutdown race.
New starts go p.runPeerRefresh(). Close cancels p.ctx, closes connections, and calls p.Server.Shutdown() / WaitForShutdown(), but does not join the peer-refresh goroutine or synchronize server shutdown with clusterMu.
setPeerAddresses checks p.ctx.Err() while holding clusterMu, then later calls p.Server.ReloadOptions. A refresh that passes the context check just before cancellation can therefore enter ReloadOptions while Close shuts down the NATS server. NATS reloadAuthorization then sends on a server channel that shutdown has already closed.
Likely fix direction: track/join runPeerRefresh before shutting down the embedded server, or otherwise serialize cancellation/server shutdown against the whole setPeerAddresses/ReloadOptions critical section.
No Go race-detector warning was present (WARNING: DATA RACE / race detected during execution of test). No OOM, signal: killed, allocation failure, disk exhaustion, or file-descriptor exhaustion indicator was present.
Assignment analysis
Blame/history targets:
git blame -L 120,317 coderd/x/nats/tls_internal_test.go
git blame -L 72,135 coderd/x/nats/cluster.go
git blame -L 405,423 coderd/x/nats/pubsub.go
git log --oneline -10 --follow coderd/x/nats/tls_internal_test.go
git log --oneline -20 -- coderd/x/nats/cluster.go
git log --oneline -10 --follow coderd/x/nats/pubsub.go
The failing cluster-TLS test file has one substantive commit: ad29777 (feat: NATS mTLS pubsub implementation, PR #26902), authored by cstyan. That same commit is the most recent substantive modifier of the crashing peer-refresh implementation and Pubsub lifecycle code. Assignment is therefore based on test-function and implementation ownership, not on the author of the failing CI commit.
Assignee: cstyan.
Duplicate search
Searched open and closed coder/internal issues, including the last 30 days, for:
TestPubsub_ClusterTLS, TestPubsub_ClusterTLS/RotationOverlap, and the test familysend on closed channelreloadAuthorization, ReloadOptions, and runPeerRefreshcoderd/x/nats, tls_internal_test.go, NATS shutdown, panic, process crash, and unknown failures- data-race and OOM signatures
No matching issue was found. Related NATS flakes are different failure modes: #1617, #1607, #1612, and #1552.
Reproduction
The race is timing-dependent. Stress the parallel cluster-TLS cleanup path:
go test ./coderd/x/nats -run 'TestPubsub_ClusterTLS($|_)' -count=100
A focused regression test should force a peer refresh to pause after the context check, invoke Close, then release the refresh and verify that shutdown waits without calling ReloadOptions on a stopped server.
CI failure
test-go-pg-17)bd5d640f1e4ed8c607d6d00f7cc8392721ca8679The failure occurred in the same time window as the CI alert. The complete 2,037-line failing-job log was available and reviewed.
Failure evidence
The
coderd/x/natstest process crashed, causing 11 tests/subtests to be reported as(unknown). These are crash fallout, not 11 independent test failures.Immediately before the panic, test cleanup logged
nats server shut down. Other affected(unknown)results includeTestPubsub_ClusterTLS,Mesh,SANMismatch,MixedTLSAndPlaintext,WrongCARejected,TestPubsub_ClusterTLS_CacheSwap, andTestPubsub_ClusterTLS_RealCA.Error analysis / root cause
Classification: process crash caused by a code-level shutdown race.
Newstartsgo p.runPeerRefresh().Closecancelsp.ctx, closes connections, and callsp.Server.Shutdown()/WaitForShutdown(), but does not join the peer-refresh goroutine or synchronize server shutdown withclusterMu.setPeerAddresseschecksp.ctx.Err()while holdingclusterMu, then later callsp.Server.ReloadOptions. A refresh that passes the context check just before cancellation can therefore enterReloadOptionswhileCloseshuts down the NATS server. NATSreloadAuthorizationthen sends on a server channel that shutdown has already closed.Likely fix direction: track/join
runPeerRefreshbefore shutting down the embedded server, or otherwise serialize cancellation/server shutdown against the wholesetPeerAddresses/ReloadOptionscritical section.No Go race-detector warning was present (
WARNING: DATA RACE/race detected during execution of test). No OOM,signal: killed, allocation failure, disk exhaustion, or file-descriptor exhaustion indicator was present.Assignment analysis
Blame/history targets:
The failing cluster-TLS test file has one substantive commit:
ad29777(feat: NATS mTLS pubsub implementation, PR #26902), authored bycstyan. That same commit is the most recent substantive modifier of the crashing peer-refresh implementation and Pubsub lifecycle code. Assignment is therefore based on test-function and implementation ownership, not on the author of the failing CI commit.Assignee:
cstyan.Duplicate search
Searched open and closed
coder/internalissues, including the last 30 days, for:TestPubsub_ClusterTLS,TestPubsub_ClusterTLS/RotationOverlap, and the test familysend on closed channelreloadAuthorization,ReloadOptions, andrunPeerRefreshcoderd/x/nats,tls_internal_test.go, NATS shutdown, panic, process crash, and unknown failuresNo matching issue was found. Related NATS flakes are different failure modes: #1617, #1607, #1612, and #1552.
Reproduction
The race is timing-dependent. Stress the parallel cluster-TLS cleanup path:
A focused regression test should force a peer refresh to pause after the context check, invoke
Close, then release the refresh and verify that shutdown waits without callingReloadOptionson a stopped server.