CI step "Set toolchain version" is flaky due to network timeouts

[joshlf]

Update: #296 doesn't seem to have done the trick. A few more possible approaches:

• Retry the entire job in order to give more time for the network issue to resolve itself
• Maybe part of the issue is that the connection is bad, and the retry is reusing the bad connection? Maybe we could implement our own manual retry logic?
• Maybe we could somehow cache the needed crates so the jobs usually don't have to re-download?

Old text

Over the past few weeks, most full CI runs (currently 149 jobs) have had at least one failure due to Cargo failing to download a package even after 3 retries (the default value for the net.retry configuration option). Hopefully increasing the number of retries will make this less common.

Note that this was previously encountered in #225. #289 made the errors more obvious, but didn't make them any less common.

[zoo868e]

Hello @joshlf, I'm interested in implementing the retry logic. Additionally, is the cargo metadata necessary? It appears that the flow only requires retrieving the version and specifying the cargo version. I believe the version can be obtained from the Cargo.toml file.

[joshlf]

Awesome, thanks! If I had to make a recommendation, I'd suggest starting with the caching approach; I suspect it's the one that's most likely to be successful. But it may turn out that there's no good way to cache these artifacts across runs of the build.

As for cargo metadata, I left a comment on the PR you put up - its use is necessary in other build steps, so we unfortunately can't get rid of it.

[zoo868e]

Sure, I'll try to cache it.

[zoo868e]

Hi @joshlf, I tested the cache on my forked repository, and it worked. However, it doesn't seem to be functioning correctly on this main repository.

[joshlf]

Hi @joshlf, I tested the cache on my forked repository, and it worked. However, it doesn't seem to be functioning correctly on this main repository.

How so?

[zoo868e]

I reviewed the logs of the action job, which initially indicated that the cache was not found. Subsequently, an attempt to save it failed due to the cache's preexisting existence. This failure appears to stem from a delay in GitHub's cache write-back process.

[joshlf]

Huh interesting. Does the current version of the PR work despite that issue, or is that a bug we need to fix first before the PR is ready to merge?

[zoo868e]

I have addressed this in the pull request. 😄