roachtest: backup-restore/mixed-version failed

[cockroach-teamcity]

roachtest.backup-restore/mixed-version failed with artifacts on release-23.2.6-rc @ 212dffab088f28ff592219b98869bc34f8691ecc:

(mixedversion.go:592).Run: mixed-version test failure while running step 38 (run "plan and run backups"): error computing contents for backup 2_v22.2.19-to-v23.1.21_full-planned-and-executed-on-v23.1.21-incremental-planned-and-executed-on-v22.2.19_cluster: error running command (./cockroach sql --certs-dir certs -e "SELECT * FROM [SHOW GRANTS] AS OF SYSTEM TIME '1718178368013552384.0000000000'" --port {pgport:1}): COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/backup-restore/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=true
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-39504

[msbutler]

Well this is an odd one. We got the following error after restore completed. I guess restore forgot to turn the schema back online?

Wraps: (4) error computing contents for backup 2_v22.2.19-to-v23.1.21_full-planned-and-executed-on-v23.1.21-incremental-planned-and-executed-on-v22.2.19_cluster
Wraps: (5) error running command (./cockroach sql --certs-dir certs -e "SELECT * FROM [SHOW GRANTS] AS OF SYSTEM TIME '1718178368013552384.0000000000'" --port {pgport:1})
Wraps: (6) Node 1. Command with error:
  | ```
  | ./cockroach sql --certs-dir certs -e "SELECT * FROM [SHOW GRANTS] AS OF SYSTEM TIME '1718178368013552384.0000000000'" --port {pgport:1}
  | ```
  | stdout: <empty>
  | stderr:ERROR: schema "public" is offline: restoring
  | Failed running "sql"

[msbutler]

ah, thanks to the excellent logging, we were in the following mixed version state, which implies that the previous restore was non-mvcc:

(1) test failed:
  | test random seed: 6463111699098982353
  |
  |                                     n1           n2           n3           n4
  | released versions                   v22.2.19     v22.2.19     v22.2.19     v23.1.21
  | logical binary versions             22.2         22.2         22.2         23.1
  | cluster versions before failure     22.2         22.2         22.2         22.2
  | cluster versions after failure      22.2         22.2         22.2         22.2

Given that 22.2 is out of support, i'm not going to investigate this further.

[msbutler]

Closing this out as 22.2 is EOL