roachtest: clearrange/zfs/checks=true failed

[cockroach-teamcity]

roachtest.clearrange/zfs/checks=true failed with artifacts on master @ 909e91dc1ea45e8223658c1144067332df7ed1ec:

(cluster.go:2347).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/clearrange/zfs/checks=true/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=16
• ROACHTEST_encrypted=false
• ROACHTEST_fs=zfs
• ROACHTEST_localSSD=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-37629

[itsbilal]

This one is weird. n4 seems to have crashed inexplicably, causing the monitor to trip up. But the logs for n4 don't say anything descriptive; we see cockroach exited with code 10 but no panics or any fatal errors in the logs.

[RaduBerinde]

10 is "disk full":

cockroach/pkg/cli/exit/codes.go

Line 63 in e16a8ed

func DiskFull() Code { return Code{10} }

[jbowens]

I think n4 was likely seeing the effect of space amplification due to eventually file-only snapshots. The disks seem tiny so we don't have a ton of leeway. n4 had a large LSM relative to the other nodes, but not especially so. I think we should add a timeseries metric for the size of zombie sstables. I'll file an issue.

https://grafana.testeng.crdb.io/d/StorageAvKxELVz/storage?from=1712588792350&to=1712589191384&var-cluster=teamcity-14738640-1712555393-97-n10cpu16&orgId=1&var-instances=All

[jbowens]

I'm going to close it out; we have a path forward to resolution (cockroachdb/pebble#3500) and better diagnostics for confirming the cause (#122110).