roachtest: add a retry loop around network activity

[BramGruneir]

This is a first attempt to try to address #31127 and #31298 by putting a retry
loop around anything that does any downloading. We've seen a number of flakes
because of iffy networking and this should hopefully solve the problem.

If this doesn't fix the issues, the next step it to add some alternative
sources to sources.list.d.

Release note: None

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

This change is 

[petermattis]

Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/cmd/roachtest/hibernate.go, line 108 at r1 (raw file):

			// Build hibernate, this step involves some downloading, so it gets added
			// to the retry loop.

Starting the retry loop from scratch at this point seems unfortunate. The earlier parts seem reasonable to keep in the same loop, but I'm thinking this should be pulled out to a separate loop.

[BramGruneir]

Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/cmd/roachtest/hibernate.go, line 108 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Starting the retry loop from scratch at this point seems unfortunate. The earlier parts seem reasonable to keep in the same loop, but I'm thinking this should be pulled out to a separate loop.

Done.

[petermattis]

Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/cmd/roachtest/hibernate.go, line 111 at r2 (raw file):

		t.Status("building hibernate (without tests)")
		attempt = 0
		for r := retry.StartWithCtx(ctx, opts); r.Next(); {

I believe this can be for attempt, r := 0, retry.StartWithCtx(...); r.Next(); { ... }. Ditto for the loop above.