roachtest: identify apt failures automatically and mark them as flakes

[renatolabs]

Every couple of weeks, we get a few roachtest failures because the Ubuntu mirrors were down for a period of time (for the latest occurrence at the time of writing, see #103297).

Ideally, we want these issues to be marked as infrastructure flakes automatically, without pinging teams (just like we do with cluster creation failures and SSH errors). Luckily, apt flakes are fairly easy to identify: the process exists with code 100.

Interestingly, the Jepsen tests already try to detect this scenario [1], but the check only happens when running apt-get install; the failures linked in the first paragraph happened during apt-get upgrade. This highlights the need for a more general solution that doesn't involve every test adding their own logic to handle these flakes.

Related issues (created when this behavior was noted in the past):

• roachprod: add --retry option to run command #71929: although I'm unsure retrying will be of any help in this case. These "outages" typically last more than a few seconds.
• roachprod: configure apt-get on cluster hosts to retry getting files on failures #71934: another retry-based approach. Once again, I don't believe retrying, at the roachprod level, will be the right solution.

Perhaps we should close the issues above if handling flakes in roachtest is deemed sufficient.

[1]

cockroach/pkg/cmd/roachtest/tests/jepsen.go

Lines 168 to 170 in 1ceb218

	if result.RemoteExitStatus == 100 {
	t.Skip("apt-get failure (#31944)", result.Stdout+result.Stderr)
	}

Jira issue: CRDB-27941

[blathers-crl]

cc @cockroachdb/test-eng

[rickystewart]

Using pre-baked images instead of using apt to setup nodes would be an interesting alternative (though not an easy one to implement).

[srosenberg]

Using pre-baked images instead of using apt to setup nodes would be an interesting alternative (though not an easy one to implement).

Indeed, we even have a story for it [1]. Complexity-wise, it's definitely easier to execute than getting rid of public ips, but the effort/lift trade-off is fuzzy. We don't see a high frequency of apt-related flakes; otherwise, we would have most likely prioritized using pre-baked images.

[1] #77644