roachtest: cluster monitor hanging after unexpected node closure

[cockroach-teamcity]

roachtest.import/tpch/nodes=8 failed with artifacts on master @ ed3a25e3c9459cede2f80babbfc9d44a836b6c12:

VMs preempted during the test run : projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-13762710-1706682693-107-n8cpu4-0005

**Other Failure**
(test_runner.go:1136).runTest: test timed out (10h0m0s)
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-35788

[DrewKimball]

The import seems to have succeeded in ~2 hours, while node 5 died about 1.5 hours in, probably due to VM preemption. I wonder if the test runner isn't able to handle dead nodes, and hangs instead?

[DrewKimball]

The test uses a roachtest/cluster.Monitor to watch for unexpected node deaths, but maybe something about the nature of this node death (ssh failures due to VM preemption) prevents that from working correctly. I see this bit in the stack trace repeated 7 times (indicating that one node is already finished):

goroutine 7338029 [IO wait, 599 minutes]:
internal/poll.runtime_pollWait(0x7f9740938e98, 0x72)
	GOROOT/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc00478c960?, 0xc0011d3000?, 0x1)
	GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00478c960, {0xc0011d3000, 0x1000, 0x1000})
	GOROOT/src/internal/poll/fd_unix.go:164 +0x27a
os.(*File).read(...)
	GOROOT/src/os/file_posix.go:29
os.(*File).Read(0xc003cbaeb8, {0xc0011d3000?, 0x2?, 0xc004af0db0?})
	GOROOT/src/os/file.go:118 +0x52
bufio.(*Reader).fill(0xc004af0f50)
	GOROOT/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).ReadSlice(0xc004af0f50, 0x1?)
	GOROOT/src/bufio/bufio.go:379 +0x29
bufio.(*Reader).ReadLine(0xc004af0f50)
	GOROOT/src/bufio/bufio.go:408 +0x25
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Monitor.func2.3({0x85bfb40?, 0xc003cbaeb8?})
	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:956 +0x1e5
created by github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Monitor.func2 in goroutine 7337904
	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:952 +0xea5

There are another 7 of these:

goroutine 7337904 [semacquire, 599 minutes]:
sync.runtime_Semacquire(0xc0053640c0?)
	GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc0042fc220?)
	GOROOT/src/sync/waitgroup.go:116 +0x48
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Monitor.func2(0x0)
	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1021 +0xf59
created by github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Monitor in goroutine 7337903
	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:759 +0x19b

...but 8 of these:

goroutine 7337979 [chan receive, 599 minutes]:
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Monitor.func2.4()
	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1017 +0x35
created by github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Monitor.func2 in goroutine 7338018
	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1016 +0xf4c

I'm suspicious of this line:

cockroach/pkg/roachprod/install/cluster_synced.go

Lines 957 to 959 in cc4fdff

	if err == io.EOF {
	return
	}

which never invokes sendEvent to inform the caller that a node has closed.

[DrewKimball]

cc @cockroach-dev-inf does this look like something your team might address?

[jlinder]

This looks more like something @cockroachdb/test-eng would address as they own roachprod + roachtest.

[renatolabs]

Ack, thanks for redirecting, we'll look into it.

Also, it's pretty weird that this issue wasn't filed under #118528 with the rest of the other VM preemption failures. Might have something to do with the test timeout.

[blathers-crl]

cc @cockroachdb/test-eng