tensorflow building issue on ubuntu 14.04: Accessing remote repository through a proxy

[lacartelacarte]

As (tensorflow/tensorflow#67) is declared as Bazel issue I'm writing here. So I'm having the same issue with the original post. As mentioned in the original post, by manually wget and placing the jpeg library the error message about jpeg has gone. However there are still many error messages, and by trying multiple times the error message seems to change but always fails with downloading something. These are the variations.

Variation 1:
ERROR: /home/name/work/tensorflow/tensorflow/core/platform/default/build_config/BUILD:36:1: no such package '@re2//': https://github.com/google/re2.git: cannot open git-upload-pack and referenced by '//tensorflow/core/platform/default/build_config:platformlib'.

Variation 2:
ERROR: /home/name/work/tensorflow/tensorflow/core/platform/default/build_config/BUILD:36:1: no such package '@png_archive//': Error downloading from https://storage.googleapis.com/libpng-public-archive/libpng-1.2.53.tar.gz to /home/name/.cache/bazel/_bazel_name/894ee6ce07893c5c665fe600d592867f/external/png_archive: Error downloading https://storage.googleapis.com/libpng-public-archive/libpng-1.2.53.tar.gz to /home/name/.cache/bazel/_bazel_name/894ee6ce07893c5c665fe600d592867f/external/png_archive/libpng-1.2.53.tar.gz: storage.googleapis.com and referenced by '//tensorflow/core/platform/default/build_config:platformlib'.

Variation 3:
ERROR: /home/name/work/tensorflow/tensorflow/tensorflow.bzl:224:3: no such package '@gemmlowp//': https://github.com/google/gemmlowp.git: cannot open git-upload-pack and referenced by '//tensorflow/core:kernels'.

[damienmg]

Wow, do you have special network configuration? It seems like it cannot fetch remote dependencies.

@kchodorow @davidzchen might have more idea on the problem source.

[lacartelacarte]

@damienmg Thank you for reply. Yes it seems to be network problem. My working machine is behind a proxy but I think I have correct setting to pass it. At least I could manually wget the jpeg library without problem. I'll check with each of the libraries mentioned in the error message when I have access to the machine tomorrow, but I think there won't be much problem by manual wget. Does Bazel need special setting for proxy?

[damienmg]

Oops assigning to Kristina then

[jgbos]

FYI, I'm having the same issue behind a proxy set as http://site.com:port (same for https).

• I'm using Centos 6.7
• Proxy settings for both HTTP_PROXY and HTTPS_PROXY are set (and lowercase versions)
• If I download the files for new_http_archive the same issue appears for git_repository
• Global config for proxy is set for git

[kchodorow]

Unfortunately, there is no support at the moment for fetching external resources through a proxy. It looks like we'd want to add a mechanism to specify local settings, which I don't think Bazel has at the moment.

[lacartelacarte]

Just confirmed I can git clone and wget those resources without problem, but fails with Bazel. Are there any work around to let Bazel use the system's proxy setting?

[kchodorow]

No (although we should add that!), but you should be able to change your WORKSPACE file to local_repository() or new_local_repository() once they're downloaded to your local machine (see http://bazel.io/docs/be/workspace.html#local_repository, http://bazel.io/docs/be/workspace.html#new_local_repository).

[lacartelacarte]

Got it. Thank you for the information. So the current best practice would be:

1. try building
2. read error message for missing library
3. download it and keep it somewhere (as local repository)
4. goto 1 until no error appears
   I'll try this later maybe.

[ulfjack]

Does java.net.URL use the proxy settings?

[philwo]

Java seems to support HTTP(S) proxies transparently: https://docs.oracle.com/javase/7/docs/api/java/net/doc-files/net-properties.html

We should make sure that our fetching code works with them (I haven't tried whether it does) and optionally on server start-up check whether these variables are empty, but the usual environment variables are set (HTTP_PROXY, HTTPS_PROXY) and then set the Java properties correctly from that.

[srsaharoy]

Hi,
After using local_repository() for libpng, jpegsrc, re2 and gemmlowp packages, the earlier proxy issue went away, but the build is now failing due to 'cycle in dependency graph', as shown below:

$ bazel build -c opt //tensorflow/cc:tutorials_example_trainer
Extracting Bazel installation...
.......
ERROR: /home/ssaharo/.cache/bazel/_bazel_ssaharo/64bf5fcc928610d1fc32c86de697b420/external/png_archive/BUILD:23:1: in genrule rule @png_archive//:configure: cycle in dependency graph:
//tensorflow/cc:tutorials_example_trainer
//tensorflow/core:tensorflow
//tensorflow/core:tensorflow_opensource
//tensorflow/core:core
//tensorflow/core:core_cpu
//tensorflow/core:framework
//tensorflow/core:lib
//tensorflow/core/platform/default/build_config:platformlib
@png_archive//:png

• @png_archive//:configure [self-edge].
  ERROR: Loading of target '//tensorflow/cc:tutorials_example_trainer' failed; build aborted.
  ERROR: Loading failed; build aborted.

[zhanmusi1207]

met the same problem as @srsaharoy ...

[srsaharoy]

If I change the 'genrule' parameters in png.BUILD and jpeg.BUILD to specifically include "." as follows

from:

genrule(
name = "configure",
srcs = glob(

["*/"],

to:

genrule(
name = "configure",
srcs = glob(

["/."],

then the build no longer fails with error "cycle in dependency graph"

However, now it fails because it can't find a source file (which does exist!) in either libpng-1.2.53 or jpeg-9a directory path. New error message is below which says that it could not find "libpng-1.2.53/pngpread.c". I also do not understand how or who is
generating signal 15 (sigterm) - is it generated because one thread of build process failed due to missing file? I did execute "bazel clean --expunge" before starting the build...

$ bazel build -c opt //tensorflow/cc:tutorials_example_trainer
......
INFO: Found 1 target...
ERROR: missing input file '@png_archive//:libpng-1.2.53/pngpread.c'.
ERROR: /home/ssaharo/tensorflow/tensorflow/cc/BUILD:28:1: Creating runfiles tree bazel-out/host/bin/tensorflow/cc/ops/array_ops_gen_cc.runfiles [for host] failed: build-runfiles failed: error executing command /home/ssaharo/.cache/bazel/_bazel_ssaharo/64bf5fcc928610d1fc32c86de697b420/tensorflow/_bin/build-runfiles bazel-out/host/bin/tensorflow/cc/ops/array_ops_gen_cc.runfiles_manifest ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.AbnormalTerminationException: Process terminated by signal 15; also encountered an error while attempting to retrieve output.
ERROR: /home/ssaharo/.cache/bazel/_bazel_ssaharo/64bf5fcc928610d1fc32c86de697b420/external/png_archive/BUILD:33:1: @png_archive//:png: missing input file '@png_archive//:libpng-1.2.53/pngpread.c'.
Target //tensorflow/cc:tutorials_example_trainer failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/ssaharo/.cache/bazel/_bazel_ssaharo/64bf5fcc928610d1fc32c86de697b420/external/png_archive/BUILD:33:1 1 input file(s) do not exist.

[srsaharoy]

sorry, somehow made a typo while posting:
the change was from "/*" to "/."

[lacartelacarte]

I tried the 'read error -> download lib -> set local repository' loop.
After fixing these reference issues I could build tensorflow with bazel without problem.
My protobuf's version was old and this became a problem when I tried to import tensorflow in python, but by replacing protobuf with >= 3.0 this problem is fixed and so finally I can test some network with tensorflow.

[uriv]

could anyone who succeeded to overcome this problem post their changes to the WORKSPACE file?

[damienmg]

See issue #623 srsaharoy posted a WORKSPACE file (which only trouble is the circular dependencies and the fix is in the bug report).

We should probably provide a WORKSPACE + shell script to work around that bug.

[uriv]

Thanks. I added a summary of the fix to that issue (#623 (comment)).

[kbrems]

Note, the posted WORKSPACE file only gets you past building the tensorflow tutorials_example_trainer. If you want to rebuild the whole package and install, for example to create a new TensorFlow user operator, there are no less than 74 external dependencies to manually download and edit in your WORKSPACE file...(mostly needed for TensorBoard). This is not very workable for those of us trying to build TensorFlow from behind a corporate firewall.

[hughperkins]

Hi @sunil3590 I really like your idea, and created a repo, with a script to facilitate slightly https://github.com/hughperkins/tf_cached_build

[damienmg]

Do you know that you could simply does a `bazel fetch --experimental_repository_cache /path/to/cache //...` and then ship the `/path/to/cache` directory and `bazel build --experimental_repository_cache /path/to/cache //...` would use that cache?

…

On Sun, May 28, 2017 at 8:17 AM Hugh Perkins ***@***.***> wrote: Hi @sunil3590 <https://github.com/sunil3590> I really like your idea, and created a repo, with a script to facilitate slightly https://github.com/hughperkins/tf_cached_build — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#587 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf8P5rSFL3dK5-dfjfZkXevDLyiYhks5r-ZAlgaJpZM4GgBZt> .

[hughperkins]

@damienmg I did not. Extremely useful information :-)

[damienmg]

:)

…

On Sun, May 28, 2017 at 8:36 AM Hugh Perkins ***@***.***> wrote: @damienmg <https://github.com/damienmg> I did not. Extremely useful information :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#587 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf3oSdH1P-J4O5djngZK5YwGA_L1tks5r-ZSHgaJpZM4GgBZt> .

[hughperkins]

By the way, @damienmg what version is required for this fuctionality to work?

[hughperkins]

Hi @damienmg I tried upgrading to bazel 0.5.0, but doesnt seem to work for me for tensorflow?

(basically what I did was:

• modify ./configure to use the --experimental_repository_cache command, in the fetch
• with wifi on, run ./configure, wait for it to finish
• weirdly, the cache was only 17MB at this point. I was expecting it to be much larger?
• then I started running a build for a few minutes
• stopped the build, turned off wifi, redid ./configure
• => screenshot above

[damienmg]

Can you give the reproduction steps?

…

On Tue, May 30, 2017 at 12:50 AM Hugh Perkins ***@***.***> wrote: Hi @damienmg <https://github.com/damienmg> I tried upgrading to bazel 0.5.0, but doesnt seem to work for me for tensorflow? [image: screen shot 2017-05-30 at 8 48 17 am] <https://cloud.githubusercontent.com/assets/123560/26573285/0909ea7e-4515-11e7-8f86-44d0a02d5fa5.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#587 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf_SB-OuA9xTCbsU0LorzaSSpVNgFks5r-8pegaJpZM4GgBZt> .

[mcelhennyi]

I need this to build tensor flow for the native_client in mozilla's DeepSpeech project.

I am behind an authorized proxy, where I need a "domain\user:password@my.proxy.com:9090" as my proxy

Even after using (where my $http_host, $https_host, and $prox_port are correct):
bazel --host_jvm_args=-Dhttp.proxyHost=$http_host --host_jvm_args=-Dhttp.proxyPort=$prox_port --host_jvm_args=-Dhttps.proxyHost=$https_host --host_jvm_args=-Dhttps.proxyPort=$prox_port build -c opt --copt=-march=native --copt=-mtune=native --copt=-O3 //tensorflow:libtensorflow_cc.so //native_client:*

I still cannot get past the error:
WARNING: ignoring http_proxy in environment.
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '@io_bazel_rules_closure//closure': Error downloading [http://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/dec425a4ff3faf09a56c85d082e4eed05d8ce38f.tar.gz, https://github.com/bazelbuild/rules_closure/archive/dec425a4ff3faf09a56c85d082e4eed05d8ce38f.tar.gz] to /home/adc-lab/.cache/bazel/_bazel_adc-lab/4534105d4045448773f96d944be86b59/external/io_bazel_rules_closure/dec425a4ff3faf09a56c85d082e4eed05d8ce38f.tar.gz: All mirrors are down: [java.lang.NullPointerException]
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '@io_bazel_rules_closure//closure': Error downloading [http://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/dec425a4ff3faf09a56c85d082e4eed05d8ce38f.tar.gz, https://github.com/bazelbuild/rules_closure/archive/dec425a4ff3faf09a56c85d082e4eed05d8ce38f.tar.gz] to /home/adc-lab/.cache/bazel/_bazel_adc-lab/4534105d4045448773f96d944be86b59/external/io_bazel_rules_closure/dec425a4ff3faf09a56c85d082e4eed05d8ce38f.tar.gz: All mirrors are down: [java.lang.NullPointerException]
INFO: Elapsed time: 2.146s
FAILED: Build did NOT complete successfully (0 packages loaded)

Does Auth support domain as well in user name?

[developer-mayuan]

@mcelhennyi I had the same issue as yours.

[ghost]

@sunil3590 I tried your method on a CentOS 6 cluster but it isn't working. I downloaded the file on my login node of cluster and created a http server. I tested the server using wget and it is workiing fine. But when I provide the same url to bazel, it cannot download the file.
Note: I am running it using the option --output_user_root=/tmp/bazel since my cluster uses Lustre filesystem which doesn't support bazel.
The command I am running is bazel --output_user_root=/tmp/bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package. I changed the urls in io_bazel_rules_closure in tensorflow workspace.bzl as you mentioned.

[hliang]

Just report that I don't have this issue.
I'm using Centos 7.2, + bazel 0.13.0. Proxy settings for both HTTP_PROXY and HTTPS_PROXY are set (uppercase). Bazel is able to download the packages, although it said "WARNING: ignoring http_proxy in environment." (I read somewhere that the variables are ignored by client, not server - which is doing the downloading).

[JonnoFTW]

I got this working on CentOS, you need to unset http(s)_proxy, after starting a http proxy on host_running_proxy:

ssh -f -N -n -L9999:localhost:9999 host_running_proxy
unset https_proxy
unset http_proxy
export HTTPS_PROXY=http://locahost:9999
export HTTP_PROXY=http://localhost:9999

[chaitanyabandiapps]

Hi everyone, I tried all the options above to try my problem get going, but I'm unable to make it work. I'm using windows 10 on an Azure Deep Learning DSVM, probably the latest build. I'm trying to build the summarize_graph tool using tensorflow's bazel commands.

bazel build graph_transforms:summarize_graph

I'm getting the following error :
tensorflow/core/platform/default/build_config/BUILD:153:1: no such package '@png_archive//'

I'm using Bazel Version "0.15.0".

I tried removing the proxy setting too both using cmd and manually in settings, but that seemed not to work. Could anyone please help me with this ? Please let me know if you need any more information.

[b-ak]

@JonnoFTW That works. Expect that I didn't any proxy server
To elaborate a little.. Here's how it looks like

username@hostname:~$ unset http_proxy 
username@hostname:~$ unset https_proxy 
username@hostname:~$ export HTTPS_PROXY=http://me:mypassword@myproxyserver.domain.com:8000
username@hostname:~$ export HTTPS_PROXY=http://me:mypassword@myproxyserver.domain.com:8000

[ackbar03]

I have this issue as well and its really starting to get annoying. It seems that Tensorflow serve with gpu support can only be built with this Bazel thing and the problems existed for 3 years without a solution... They really couldn't have built it some other way : ( , spent my entire day on this.

[ackbar03]

ok so i've finally managed to find a hacky way around this. I'm behind the great firewall of chinois, my vpn doesn't really work anymore and basically my connection keeps getting cut off when ever its trying to fetch something as part of the build process and I have to start again everytime, might be a different problem for those behind a corporate firewall.

basically i run

echo build --experimental_repository_cache /.bazel_downloads >>/.bazelrc

and then instead of running bazel build I run bazel fetch on the target which sort of prefetches it and caches it so you just keep doing that and cursing under your breath until most of the the packages are fetched, and then you run bazel build and hopefully it works.

I'm not actually sure if you need to run echo build --experimental_repository_cache /.bazel_downloads >>/.bazelrc
or it the fetch caches it automatically but I can't be bothered to figure it out. God this was so frustrating, I can't believe theres no easier way to do this

just posting in case it could save someone else the frustration. hhrrrRUAAGHH

[ackbar03]

Does bazel have a mirror or something one can use to download the dependencies from? I had such a bad experience with this last time and I have to do it again tomorrow and it's bringing up some ptsd