roachprod: identify apt failures automatically and mark them as flakes

[vidit-bhat]

Previously, we used to get roachtest failures when the Ubuntu mirrors were down for a period of time.

This PR identifies these apt failures and marks them as a flake.

Epic: none
Fixes: #103316
Release note: None

[cockroach-teamcity]

This change is 

[DarrylWong]

A quick search shows it's not a bad assumption to say exit code 100 = apt failure.

I did see one case where exit code 100 wasn't a transient error though, but I suppose the same could be true for SSH. Maybe we just hope someone will notice if the same test keeps flaking every night 🤷

SGTM but maybe someone else has a different opinion

[vidit-bhat]

bors r+

[craig]

Build succeeded:

• linux_arm64_build
• local_roachtest_fips
• macos_amd64_build
• acceptance
• linux_amd64_fips_build
• macos_arm64_build
• check_generated_code
• local_roachtest
• lint
• windows_build
• examples_orms
• docker_image_amd64
• unit_tests
• linux_amd64_build

[renatolabs]

blathers backport 24.1 23.2

[vidit-bhat]

blathers backport 23.1

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 488551c to blathers/backport-release-23.1-123725: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1 failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}