roachtest: add failure message to test if post validations fail

[aliher1911]

In #96949 we added post-test validations for roachtests.
When validation fails GH task is created with an error like

(cluster.go:1387).FailOnInvalidDescriptors: invalid descriptors check failed: pq: query execution canceled due to statement timeout

but linked test log in artifacts end with

11:04:14 test_runner.go:990: tearing down after success; see teardown.log

indicating that test was successful.

It would be helpful to have this failures consistent with "normal" test failures and be visible in artifacts.zip.

Example failure: #97912

Jira issue: CRDB-25228

[blathers-crl]

cc @cockroachdb/test-eng

[smg260]

Post validations are done during teardown, by which stage we've replaced the test logger with a teardown logger. Possible solutions include

• move these validations up into runTest, so we can report a failure in the test.log
• emit a cautionary log statement to check teardown.log for post validation failures

The validation failures should still show up in the artifacts as a failure_x.log.

@aliher1911 just confirming that you expect to see some indication that a post validation failed in test.log, and not just the presence of the failure.log

[aliher1911]

Ideally all I want is not to chase errors. In vanilla test failure case I see the error in test log and I can go to the source to see what failed. If there was some remote command involved, I can go to artifacts and find one that failed to see the actual output. With new post validations, there would be no test failure to go to. So I need to look through artifacts to find which one of the failed commands is the real failure. There could be more than one because recovery tests fail some things in successful test runs.

From my point of view having validators in test.log is a good idea as teardown.log is the last place I would go to when trying to fix broken test. But I don't have a big picture.