pkg/sql/colenc/colenc_test: [deadlock in GetStorageClusterActiveVersion] TestEncoderEqualityDatums failed

[cockroach-teamcity]

pkg/sql/colenc/colenc_test.TestEncoderEqualityDatums failed on master @ 934684d8134fd8bb34eae1a37f3aa83a4ac066b7:

Fatal error:

panic: test timed out after 16m37s
running tests:
	TestEncoderEqualityDatums (16m37s)

Stack:

goroutine 233773828 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

=== RUN   TestEncoderEqualityDatums
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestEncoderEqualityDatums4163817714
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:157: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.

Parameters:

• attempt=1
• deadlock=true
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-37933

[rytaft]

Seems like a deadlock here:

goroutine 458753 [runnable]:
github.com/sasha-s/go-deadlock.(*lockOrder).preLock(0xbf92e00, {0xc018b08820, 0x5, 0x32}, {0x5c8d4a0, 0xc00b53bc78})
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:275 +0x985
github.com/sasha-s/go-deadlock.preLock(...)
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:159
github.com/sasha-s/go-deadlock.lock(0xc010f87e38, {0x5c8d4a0, 0xc00b53bc78})
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:176 +0x77
github.com/sasha-s/go-deadlock.(*Mutex).Lock(...)
	github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:85
github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).GetStorageClusterActiveVersion(0xc00b53bb80)
	github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:583 +0x75
github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startTenantAutoUpgradeLoop.func1({0x7e8c658, 0xc010eadce0})
	github.com/cockroachdb/cockroach/pkg/server/tenant_auto_upgrade.go:45 +0x125
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 325533
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe

[rickystewart]

Probable tenant upgrade deadlock problem internal discussion. Pro-actively closing.