Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry downloads on transient network errors (e.g. connection reset) #1717

Closed
bshi opened this issue Sep 2, 2016 · 0 comments
Closed

Retry downloads on transient network errors (e.g. connection reset) #1717

bshi opened this issue Sep 2, 2016 · 0 comments
Labels
P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) type: feature request

Comments

@bshi
Copy link

bshi commented Sep 2, 2016 •
edited

Somewhat related to #1194

We hit a "Connection reset" error while attempting to download the Go toolchain

2016-09-02 02:04:31.543330500 ____Loading package: vendor/google.golang.org/api/googleapi
2016-09-02 02:04:31.589733500 ____Downloading from https://storage.googleapis.com/golang/go1.7.linux-amd64.tar.gz: 0B
2016-09-02 02:04:32.467964500 ERROR: /cache/bazel/test_tmp/_bazel_root/8c069df52082beee3c95ca17836fb8e2/external/io_bazel_rules_go/go/toolchain/BUILD:33:1: no such package '@golang_linux_amd64//': Error downloading from https://storage.googleapis.com/golang/go1.7.linux-amd64.tar.gz to /cache/bazel/test_tmp/_bazel_root/8c069df52082beee3c95ca17836fb8e2/external/golang_linux_amd64: Error downloading https://storage.googleapis.com/golang/go1.7.linux-amd64.tar.gz to /cache/bazel/test_tmp/_bazel_root/8c069df52082beee3c95ca17836fb8e2/external/golang_linux_amd64/go1.7.linux-amd64.tar.gz: Connection reset and referenced by '@io_bazel_rules_go//go/toolchain:go_include'.

It would be helpful to retry on certain classes of errors (particularly network errors).

CC @dvanderb
@kchodorow kchodorow added type: feature request P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) category: extensibility > external repositories labels Sep 6, 2016
laszlocsomor pushed a commit that referenced this issue Nov 30, 2016
@jart @laszlocsomor
Support multiple mirror URLs for external repos
92887a0 
This change improves upon 4c67807.

- A urls attribute has been added to the native workspace rules, with
  the exception of maven_jar and git_repository. The Skylark repository
  API also supports multiple URLs now.

- The earlier mirrors in the list are preferred. Failover will happen
  automatically in parallel.

- The first 32kB of data is checked before choosing a mirror in order
  to evade captive portals.

- If one's Internet goes down or a download times out, then the
  download will resume automatically where it left off, provided the
  server supports RFC7233 for that particular file. Please note that
  GitHub does not support this for archive snapshots. Files should
  always be mirrored to a CDN, e.g. GCS, because they support this.

- A semaphore is now used on downloads so only 8 can happen at once.

Fixes #1814
Fixes #2131
Fixes #2008
Fixes #1968
Fixes #1717
Fixes #943
Wont fix #1194
Fixes tensorflow/tensorflow#5933
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5432
See #1607
See #821
See tensorflow/tensorflow#5080
See tensorflow/tensorflow#5029
See tensorflow/tensorflow#4583
See tensorflow/tensorflow#4058

RELNOTES: A urls attribute has been added to repository rules to support multiple mirror URLs for reliably downloading files.

--
MOS_MIGRATED_REVID=140495736
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bshi @kchodorow