roachtest: backup/2TB/n10cpu4 failed

[cockroach-teamcity]

roachtest.backup/2TB/n10cpu4 failed with artifacts on release-23.2.5-rc @ e7576d10aaaa61940cebf8bd8f157ce0e1651e7b:

(monitor.go:153).Wait: monitor failure: unexpectedly found job 961836988879241217 in state failed
test artifacts and logs in: /artifacts/backup/2TB/n10cpu4/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: backup/2TB/n10cpu4 failed #122434 roachtest: backup/2TB/n10cpu4 failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-23.2 release-blocker]
• roachtest: backup/2TB/n10cpu4 failed #121886 roachtest: backup/2TB/n10cpu4 failed [C-test-failure O-roachtest O-robot P-1 T-disaster-recovery branch-master]
• roachtest: backup/2TB/n10cpu4 failed [client connection force closed via ClientConn.Close] #121755 roachtest: backup/2TB/n10cpu4 failed [C-test-failure O-roachtest O-robot P-3 T-disaster-recovery branch-release-24.1]

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38294

[msbutler]

another gcp flake:

6.unredacted/cockroach.log:E240420 08:04:22.609414 195757 jobs/adopt.go:482 ⋮ [T1,Vsystem,n6] 1648  job 961836988879241217: adoption completed with error failed to run backup: running distributed backup to export 1103 ranges: ‹googleapi: got HTTP response code 503 with body: ›