local build error #23

zh794390558 · 2019-03-01T11:46:53Z

ERROR: error loading package 'lingvo': Encountered error while reading extension file 'subpar.bzl': no such package '@subpar//': Traceback (most recent call last):
        File "/home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/bazel_tools/tools/build_defs/repo/git.bzl", line 166
                _clone_or_update(ctx)
        File "/home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/bazel_tools/tools/build_defs/repo/git.bzl", line 72, in _clone_or_update
                fail(("error cloning %s:\n%s" % (ctx....)))
error cloning subpar:

The text was updated successfully, but these errors were encountered:

drpngx · 2019-03-01T23:11:17Z

What bazel version are you using? Can you try 0.17.2?

jonathanasdf · 2019-03-02T00:26:10Z

Can you please run this script

https://github.com/tensorflow/lingvo/blob/master/tf_env_collect.sh

zh794390558 · 2019-03-03T01:24:25Z

solved by updating git version to 2.18.0, thanks.

zh794390558 · 2019-03-03T01:32:58Z


== cat /etc/issue ===============================================
Linux luban-351 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="7 (Core)"
VERSION_ID="7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

== are we in docker =============================================
Yes

== compiler =====================================================
c++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


== bazel =====================================================
Build label: 0.17.2
Build time: Fri Sep 21 10:31:42 2018 (1537525902)
Build timestamp: 1537525902
Build timestamp as int: 1537525902

== uname -a =====================================================
Linux luban-351 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
numpy               1.15.4    
protobuf            3.6.1     
tensorflow          1.12.0    

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.12.0
tf.GIT_VERSION = b'unknown'
tf.COMPILER_VERSION = b'unknown'
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH /nfs/project/tools/anaconda3/pkgs/cudnn-7.2.1-cuda9.2_0/lib:/nfs/project/tools/anaconda3/pkgs/cudatoolkit-9.2-0/lib:/usr/local/nccl_2.3.7-1+cuda10.0_x86_64/lib/:/usr/local/cuda-9.0/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Sun Mar  3 09:29:55 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.72       Driver Version: 410.72       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P40           Off  | 00000000:02:00.0 Off |                  N/A |
| N/A   40C    P0    67W / 250W |  16827MiB / 22919MiB |     29%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           Off  | 00000000:03:00.0 Off |                  N/A |
| N/A   41C    P0    69W / 250W |  16827MiB / 22919MiB |     30%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           Off  | 00000000:83:00.0 Off |                  N/A |
| N/A   23C    P8    10W / 250W |     10MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P40           Off  | 00000000:84:00.0 Off |                  N/A |
| N/A   23C    P8    10W / 250W |     10MiB / 22919MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

== cuda libs  ===================================================
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart_static.a

drpngx · 2019-03-03T01:33:59Z

Good to know. We'll update the README.

zh794390558 · 2019-03-03T01:42:22Z

ERROR: /nfs/project/zhanghui/lingvo/lingvo/core/BUILD:339:1: no such package '@protobuf_protoc//': java.io.IOException: Error downloading [https://github.com/google/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip] to /home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/protobuf_protoc/protoc-3.6.1-linux-x86_64.zip: GET returned 403 Forbidden and referenced by '//lingvo/core:inference_graph_py_pb2_genpy'
ERROR: Analysis of target '//lingvo:trainer_test' failed; build aborted: no such package '@protobuf_protoc//': java.io.IOException: Error downloading [https://github.com/google/protobuf/releases/download/v3.6.1/protoc-3.6.1-linux-x86_64.zip] to /home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/external/protobuf_protoc/protoc-3.6.1-linux-x86_64.zip: GET returned 403 Forbidden
INFO: Elapsed time: 422.566s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded)
FAILED: Build did NOT complete successfully (2 packages loaded)

drpngx · 2019-03-03T01:44:17Z

I just tried and I'm able to download that file. Is it possible that you're under a firewall that blocks the download?

zh794390558 · 2019-03-03T01:46:22Z

Maybe, how can I install this offline, if i have this package.

drpngx · 2019-03-03T01:48:03Z

Currently there's no option to use a local package. If you have a local mirror, then the location is specified in repo.bzl

…

On Sat, Mar 2, 2019, 5:46 PM Bayberry Z. ***@***.***> wrote: Maybe, how can I install this offline, if i have this package. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AT_SbTIsZ4hHpMeUzXBL7Alxbk2ViXf7ks5vSylvgaJpZM4bY2UW> .

zh794390558 · 2019-03-03T02:22:40Z

http_archive will download zip without check if the target zip existing. How can I change the code to check local exists or just extracting the zip directly?

def _http_archive_impl(ctx):
    """Implementation of the http_archive rule."""
    if not ctx.attr.url and not ctx.attr.urls:
        fail("At least one of url and urls must be provided")
    if ctx.attr.build_file and ctx.attr.build_file_content:
        fail("Only one of build_file and build_file_content can be provided.")

    all_urls = []
    if ctx.attr.urls:
        all_urls = ctx.attr.urls
    if ctx.attr.url:
        all_urls = [ctx.attr.url] + all_urls

    ctx.download_and_extract(
        all_urls,
        "",
        ctx.attr.sha256,
        ctx.attr.type,
        ctx.attr.strip_prefix,
    )
    patch(ctx)
    workspace_and_buildfile(ctx)

drpngx · 2019-03-03T03:05:00Z

I am loath to introduce a way to pick up local binaries, since it circumvents the idea of hermetic builds. That would make debugging harder. You can download the file by any means necessary and then specify file:// as the url.

zh794390558 · 2019-03-03T03:50:26Z

thanks, work for me.

https://github.com/tensorflow/lingvo/blob/master/lingvo/repo.bzl#L5
https://github.com/tensorflow/lingvo/blob/master/lingvo/repo.bzl#L19

These functions not work when I using virtual env for tensorflow.
fix this by return corresponding dirs.

zh794390558 · 2019-03-03T03:51:35Z

INFO: Build options have changed, discarding analysis cache.
INFO: Analysed target //lingvo:trainer_test (31 packages loaded).
INFO: Found 1 test target...
ERROR: /nfs/project/zhanghui/lingvo/lingvo/tools/BUILD:98:1: Linking of rule '//lingvo/tools:generate_proto_def' failed (Exit 1)
bazel-out/host/bin/lingvo/tools/_objs/generate_proto_def/generate_proto_def.o:generate_proto_def.cc:function (anonymous namespace)::WriteDotProto(google::protobuf::FileDescriptor const*, char const*): error: undefined reference to 'google::protobuf::FileDescriptor::DebugString() const'
collect2: error: ld returned 1 exit status
Target //lingvo:trainer_test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 18.259s, Critical Path: 2.09s
INFO: 2 processes: 2 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

FAILED: Build did NOT complete successfully

verbose output

INFO: Analysed target //lingvo:trainer_test (0 packages loaded).
INFO: Found 1 test target...
ERROR: /nfs/project/zhanghui/lingvo/lingvo/tools/BUILD:98:1: Linking of rule '//lingvo/tools:generate_proto_def' failed (Exit 1): gcc failed: error executing command 
  (cd /home/luban/.cache/bazel/_bazel_luban/b5ef85f1c360696308ba7ab9000cfd03/execroot/__main__ && \
  exec env - \
    LD_LIBRARY_PATH=/nfs/project/tools/anaconda3/pkgs/cudnn-7.2.1-cuda9.2_0/lib:/nfs/project/tools/anaconda3/pkgs/cudatoolkit-9.2-0/lib:/usr/local/nccl_2.3.7-1+cuda10.0_x86_64/lib/:/usr/local/cuda-9.0/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64: \
    PATH=/nfs/project/tools/openfst1.6.2/bin/:/nfs/project/tools/packages/kaldi-master/src/bin:/nfs/project/tools/packages/kaldi-master/src/fstbin/:/nfs/project/tools/packages/kaldi-master/src/gmmbin/:/nfs/project/tools/packages/kaldi-master/src/featbin/:/nfs/project/tools/packages/kaldi-master/src/lm/:/nfs/project/tools/packages/kaldi-master/src/sgmmbin/:/nfs/project/tools/packages/kaldi-master/src/sgmm2bin/:/nfs/project/tools/packages/kaldi-master/src/fgmmbin/:/nfs/project/tools/packages/kaldi-master/src/latbin/:/nfs/project/tools/packages/kaldi-master/src/nnetbin:/nfs/project/tools/packages/kaldi-master/src/nnet2bin/:/nfs/project/tools/packages/kaldi-master/src/kwsbin:/nfs/project/tools/packages/kaldi-master/tools/sph2pipe_v2.5:/nfs/project/tools/packages/kaldi-master/src/ivectorbin:/tools/kaldi-io/build/bin:/nfs/project/tools/anaconda3/envs/tf1.12_py3.5/bin:/nfs/project/tools/anaconda3/bin:/home/luban/miniconda3/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/home/luban/miniconda3/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/luban/.local/bin:/home/luban/bin:/home/luban/.local/bin:/home/luban/bin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -o bazel-out/host/bin/lingvo/tools/generate_proto_def '-Wl,-rpath,$ORIGIN/../../_solib_k8/_U@tensorflow_Usolib_S_S_Cframework_Ulib___Uexternal_Stensorflow_Usolib_Stensorflow_Usolib' -Lbazel-out/host/bin/_solib_k8/_U@tensorflow_Usolib_S_S_Cframework_Ulib___Uexternal_Stensorflow_Usolib_Stensorflow_Usolib '-fuse-ld=gold' -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -B/usr/bin -pass-exit-codes -Wl,--gc-sections -Wl,-S -Wl,@bazel-out/host/bin/lingvo/tools/generate_proto_def-2.params)

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/host/bin/lingvo/tools/_objs/generate_proto_def/generate_proto_def.o:generate_proto_def.cc:function (anonymous namespace)::WriteDotProto(google::protobuf::FileDescriptor const*, char const*): error: undefined reference to 'google::protobuf::FileDescriptor::DebugString() const'
collect2: error: ld returned 1 exit status
Target //lingvo:trainer_test failed to build
INFO: Elapsed time: 11.045s, Critical Path: 0.40s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

FAILED: Build did NOT complete successfully

drpngx · 2019-03-09T18:07:03Z

This is a duplicate of #32 -- let's continue the discussion there.

zh794390558 mentioned this issue Mar 6, 2019

Undefined reference to google::protobuf::FileDescriptor::DebugString() #32

Closed

drpngx closed this as completed Mar 9, 2019

drpngx added the bug Something isn't working label Mar 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

local build error #23

local build error #23

zh794390558 commented Mar 1, 2019

drpngx commented Mar 1, 2019

jonathanasdf commented Mar 2, 2019

zh794390558 commented Mar 3, 2019

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019 via email

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019

zh794390558 commented Mar 3, 2019 •

edited

Loading

zh794390558 commented Mar 3, 2019 •

edited

Loading

drpngx commented Mar 9, 2019

local build error #23

local build error #23

Comments

zh794390558 commented Mar 1, 2019

drpngx commented Mar 1, 2019

jonathanasdf commented Mar 2, 2019

zh794390558 commented Mar 3, 2019

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019 via email

zh794390558 commented Mar 3, 2019

drpngx commented Mar 3, 2019

zh794390558 commented Mar 3, 2019 • edited Loading

zh794390558 commented Mar 3, 2019 • edited Loading

drpngx commented Mar 9, 2019

zh794390558 commented Mar 3, 2019 •

edited

Loading

zh794390558 commented Mar 3, 2019 •

edited

Loading