Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download error for new_http_archive url #821

Closed
markusdr opened this issue Jan 27, 2016 · 4 comments
Closed

Download error for new_http_archive url #821

markusdr opened this issue Jan 27, 2016 · 4 comments

Comments

@markusdr
Copy link

I'm getting the following bazel error with tensorflow:

$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
ERROR:  tensorflow/tensorflow/core/platform/default/build_config/BUILD:36:1: 
no such package '@jpeg_archive//': 
Error downloading from http://www.ijg.org/files/jpegsrc.v9a.tar.gz

It turns out that bazel cannot download directly from the ijg.org in my network, due to security restrictions. However, I am able to download the file by hand.

I tried just putting it into the required .cache/bazel location by hand and run the bazel command again. This used to work, but it won't work in the latest version, and the same error comes up again.

Would it be possible to enable bazel to pick up a locally downloaded jpegsrc.v9a.tar.gz file? Allowing a local file:// url in new_http_archive should solve the issue.

@ahumesky
Copy link
Contributor

Hi Markus,
You could convert the new_http_archive rule in tensorflow's workspace file
to a new_local_repository rule:
http://bazel.io/docs/be/workspace.html#new_local_repository

Unpack jpegsrc.v9a.tar.gz somewhere and point the new_local_repository to
that path.

On Wed, Jan 27, 2016 at 6:46 PM Markus Dreyer notifications@github.com
wrote:

I'm getting the following bazel error with tensorflow:

$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
ERROR: tensorflow/tensorflow/core/platform/default/build_config/BUILD:36:1:
no such package '@jpeg_archive//':
Error downloading from http://wwwijgorg/files/jpegsrcv9atargz

It turns out that bazel cannot download directly from the ijgorg in my
network, due to security restrictions However, I am able to download the
file by hand

I tried just putting it into the required cache/bazel location by hand
and run the bazel command again This used to work, but it won't work in the
latest version, and the same error comes up again

Would it be possible to enable bazel to pick up a locally downloaded
jpegsrcv9atargz file? Allowing a local file:// url in new_http_archive
should solve the issue


Reply to this email directly or view it on GitHub
#821.

@markusdr
Copy link
Author

For some reason, when I convert to new_local_repository it gives another error:

$ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
ERROR: /home/local/ANT/mddreyer/.cache/bazel/_bazel_mddreyer/276918196e903f82b3401cd1a0adce36/external/jpeg_archive/BUILD:67:1: in genrule rule @jpeg_archive//:configure: cycle in dependency graph:
    //tensorflow/cc:tutorials_example_trainer
    //tensorflow/core:tensorflow
    //tensorflow/core:tensorflow_opensource
    //tensorflow/core:core
    //tensorflow/core:core_cpu
    //tensorflow/core:core_cpu_internal
    //tensorflow/core:framework
    //tensorflow/core:framework_internal
    //tensorflow/core:lib_internal
    //tensorflow/core/platform/default/build_config:platformlib
    @jpeg_archive//:jpeg
  * @jpeg_archive//:configure [self-edge].
ERROR: Loading of target '//tensorflow/cc:tutorials_example_trainer' failed; build aborted.

@markusdr
Copy link
Author

I figured out a fix for that issue, so new_local_repository is working now. Thanks!

@ahumesky
Copy link
Contributor

Great, glad it's working for you now

bazel-io pushed a commit that referenced this issue Nov 29, 2016
This change improves upon 4c67807.

- A urls attribute has been added to the native workspace rules, with
  the exception of maven_jar and git_repository. The Skylark repository
  API also supports multiple URLs now.

- The earlier mirrors in the list are preferred. Failover will happen
  automatically in parallel.

- The first 32kB of data is checked before choosing a mirror in order
  to evade captive portals.

- If one's Internet goes down or a download times out, then the
  download will resume automatically where it left off, provided the
  server supports RFC7233 for that particular file. Please note that
  GitHub does not support this for archive snapshots. Files should
  always be mirrored to a CDN, e.g. GCS, because they support this.

- A semaphore is now used on downloads so only 8 can happen at once.

Fixes #1814
Fixes #2131
Fixes #2008
Fixes #1968
Fixes #1717
Fixes #943
Wont fix #1194
Fixes tensorflow/tensorflow#5933
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5432
See #1607
See #821
See tensorflow/tensorflow#5080
See tensorflow/tensorflow#5029
See tensorflow/tensorflow#4583
See tensorflow/tensorflow#4058

RELNOTES: A urls attribute has been added to repository rules to support multiple mirror URLs for reliably downloading files.

--
MOS_MIGRATED_REVID=140495736
laszlocsomor pushed a commit that referenced this issue Nov 30, 2016
This change improves upon 4c67807.

- A urls attribute has been added to the native workspace rules, with
  the exception of maven_jar and git_repository. The Skylark repository
  API also supports multiple URLs now.

- The earlier mirrors in the list are preferred. Failover will happen
  automatically in parallel.

- The first 32kB of data is checked before choosing a mirror in order
  to evade captive portals.

- If one's Internet goes down or a download times out, then the
  download will resume automatically where it left off, provided the
  server supports RFC7233 for that particular file. Please note that
  GitHub does not support this for archive snapshots. Files should
  always be mirrored to a CDN, e.g. GCS, because they support this.

- A semaphore is now used on downloads so only 8 can happen at once.

Fixes #1814
Fixes #2131
Fixes #2008
Fixes #1968
Fixes #1717
Fixes #943
Wont fix #1194
Fixes tensorflow/tensorflow#5933
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5924
Fixes tensorflow/tensorflow#5432
See #1607
See #821
See tensorflow/tensorflow#5080
See tensorflow/tensorflow#5029
See tensorflow/tensorflow#4583
See tensorflow/tensorflow#4058

RELNOTES: A urls attribute has been added to repository rules to support multiple mirror URLs for reliably downloading files.

--
MOS_MIGRATED_REVID=140495736
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants