Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate making the buildbots network-access-free #35002

Closed
arielb1 opened this issue Jul 23, 2016 · 6 comments
Closed

Investigate making the buildbots network-access-free #35002

arielb1 opened this issue Jul 23, 2016 · 6 comments
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason)

Comments

@arielb1
Copy link
Contributor

arielb1 commented Jul 23, 2016

Currently, the Rust buildbots (bors) all clone LLVM and other libs from Github. This seems to occasionally hang and cause mysterious spurious failures, possibly due to rate-limiting.

It could be better if we would do the git submodule update once and push the updated directory to the buildbots. We need to figure out how this interacts with lazy LLVM recompilation, but it should be possible.

cc @rust-lang/tools

@arielb1 arielb1 added the A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) label Jul 23, 2016
@hanna-kruppe
Copy link
Contributor

This is not only messing with PRs, it also killed the last three nightlies.

@alexcrichton
Copy link
Member

I'd love to do this! I also have no idea how to do this easily though :(

I have no idea why LLVM clones are timing out so much, I sent mail to GitHub but haven't gotten much of a response...

@arielb1
Copy link
Contributor Author

arielb1 commented Jul 25, 2016

You can clone and git submodule update on a dedicated machine and push the result directory to the buildbots.

@alexcrichton
Copy link
Member

Buildbots are almost gone now, so closing.

@arielb1
Copy link
Contributor Author

arielb1 commented Jan 4, 2017

Still an issue with travis.

@alexcrichton
Copy link
Member

Fundamentally I believe this is unfortunately just impossible for us. We will always be required to perform network requests on Travis/AppVeyor and I don't see an end in sight to that.

That being said there are mitigation strategies for problems caused by this:

  • Caching - we can leverage Travis/AppVeyor caching where possible as that's typically a more reliable connection
  • Retries - we can just retry network requests on error
  • Mirror in reliable locations - some servers are unreliable over time but S3 is pretty reliable.

We've implemented a lot of this (mirroring on S3, retrying everything, etc), so I'm going to close this. Other general spurious network errors should be tracked at #40474 and we can investigate fixing those as they come up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason)
Projects
None yet
Development

No branches or pull requests

3 participants