Tensorflow build is broken in Bazel Nightly and in 0.10

[dslomov]

Failure:
https://ci.bazel.build/blue/organizations/jenkins/Global%2FTensorFlow/detail/TensorFlow/375/pipeline/
Smaller repro:

/bazel build //tensorflow/contrib/fused_conv:fused_conv2d_bias_activation_op_pygenrule
.........
INFO: Analysed target //tensorflow/contrib/fused_conv:fused_conv2d_bias_activation_op_pygenrule (46 packages loaded).
INFO: Found 1 target...
ERROR: /usr/local/google/home/dslomov/work/tensorflow/tensorflow/contrib/fused_conv/BUILD:105:1: Executing genrule //tensorflow/contrib/fused_conv:fused_conv2d_bias_activation_op_pygenrule failed (Aborted): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
*** Error in `bazel-out/host/bin/tensorflow/contrib/fused_conv/gen_fused_conv2d_bias_activation_op_py_wrappers_cc': munmap_chunk(): invalid pointer: 0x0000000000480320 ***
/bin/bash: line 1: 186348 Aborted                 (core dumped) bazel-out/host/bin/tensorflow/contrib/fused_conv/gen_fused_conv2d_bias_activation_op_py_wrappers_cc , '' 0 0 > bazel-out/k8-opt/genfiles/tensorflow/contrib/fused_conv/ops/gen_fused_conv2d_bias_activation_op.py
Target //tensorflow/contrib/fused_conv:fused_conv2d_bias_activation_op_pygenrule failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 5.692s, Critical Path: 1.64s
FAILED: Build did NOT complete successfully

[dslomov]

I bisected this to 2aeaeba @jmillikin-stripe

[dslomov]

I'll be rollbacking 2aeaeba

[dslomov]

Talked with lberki@ offline, assigning to him for investigation

[lberki]

Paging @jmillikin-stripe and @htuch

This change appears to have been a mistake according to my admittedly limited knowledge. The point of mostly static linking is that every library is linked statically except for the runtime libraries, which are linked dynamically, so why would one link runtime libraries statically in that mode?

I'll send out a rollback to be cherry-picked into the 0.10.0 release tomorrow morning CEST unless a strong argument against it and a workaround for this issue comes up.

@iirina : How come this comes to light now (as opposed to right after having submitted that change?)

[htuch]

I think the idea is that libstdc++ is a language/compiler library, not an environment specific runtime library. @mattklein123 for Envoy perspective on what a mostly static link should be.

[jmillikin-stripe]

Yes, libstdc++ is supposed to be safe to link statically. The primary goal of "mostly-static" (as I understand it from GOOG days) is to statically link everything except glibc, which depends on dynamic linking for core functionality.

I don't know much about Tensorflow or CUDA, but I notice https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/fused_conv/BUILD is referencing shared objects and https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tensorflow.bzl is setting alwayslink=1, linkstatic=1. Is it possible that the new Bazel flags are exposing a pre-existing bug in the compilation of gen_fused_conv2d_bias_activation_op_py_wrappers_cc?

If you remove linkstatic=1 from the tf_cc_binary() call in tf_gen_op_wrapper_py (https://github.com/tensorflow/tensorflow/blob/2c5667cad1eccf599b8834557a89453222f29e0b/tensorflow/tensorflow.bzl#L508), does that fix the error?

[htuch]

@lberki can we hold back on this for a short period; I'll reach out to you on IM.

[mattklein123]

On the high level topic, libstdc++ should definitely be statically linked in "mostly static" builds. libstdc++ is a compiler/application library, not a system library like glibc. Thus, an application mostly statically linked to everything other than glibc can be dropped on a system that only has a compatible glibc version. This is very important.

[gunan]

@jhseu and @allenlavoie to check if he has any ideas on this.

[allenlavoie]

Happy to help debug, but I haven't been able to reproduce the failure.

I tried with Bazel nightly #198 and TensorFlow at head (tensorflow/tensorflow@4595f1c), and I'm getting a successful build.

> bazel version
Build label: 
Build target: bazel-out/k8-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jul 13 02:35:31 +50017 (1516233609331)
Build timestamp: 1516233609331
Build timestamp as int: 1516233609331
> yes "" | ./configure
> bazel build //tensorflow/contrib/fused_conv:fused_conv2d_bias_activation_op_pygenrule
[...]
Target //tensorflow/contrib/fused_conv:fused_conv2d_bias_activation_op_pygenrule up-to-date:
  bazel-genfiles/tensorflow/contrib/fused_conv/ops/gen_fused_conv2d_bias_activation_op.py
INFO: Elapsed time: 141.657s, Critical Path: 34.78s
INFO: Build completed successfully, 1011 total actions

Could be related to the version of libstdc++? I'm using GLIBCXX_3.4.22

[dslomov]

Ok, interesting, I can reliably reproduce on my machine, and CI machines repro this too. We need to look more.

[dslomov]

Ok, the debug yielded that:

• //tensorflow:libtensorflow_framework.so is linked against a dynamic version of libstdc++ (-lstdc++ im params)
• binaries dependent on it (e.g. //tensorflow/contrib/fused_conv:gen_fused_conv2d_bias_activation_op_py_wrappers_cc) is linked against a static version of libstfc++ (-l:libstdc++.a)

This is problematic at least for gcc 4.8.4 (libstdc++.so.6.0.19)

[dslomov]

Machines that have no issues use gcc 6.3.0.

[lberki]

Let's roll back.

This is a case of linking libstdc++ twice, but unfortunately, there is no easy workaround for TensorFlow and I'm not convinced the offending patch does the right thing: it makes dynamic libraries link libstdc++ dynamically but binaries link it statically, thus making dynamic libraries built with Bazel and binaries built with Bazel incompatible. That doesn't sound good.

Now, one could make the case that TensorFlow is doing something weird (namely, linking dynamic libraries into a binary built in mostly-static mode), but I'd like to make that decision without time pressure.

[lberki]

@dslomov , my hunch is that it's just so happens that the standard library of gcc 6.3.0 is accidentally more resilient to issues like this.

[iirina]

TF is now green on our CI, but I see no rollback. Was this fixed on TF side?

[dslomov]

I think some machines have gcc 6.3.0 and some have gcc 4.8.4, that is why we haven't detected this earlier.

[dslomov]

and also why CI is now green

[iirina]

Is there a reason why we have machines with different gcc versions for our CI?

[iirina]

@dslomov I don't think that explains it. The rollback probably fixed this for now, but I still do not understand why the CI did not catch this.

The job is green on our CI because it uses the latest bazel release (0.9.0) to build TF and that release doesn't have this bug.

About machines with different gcc versions: Jenkins runs the job both on Ubuntu 16 and on Ubuntu 14. I checked and each of the Ubuntu 14 machines uses gcc 4.8.4, and each of the Ubuntu 16 machines uses gcc 5.4.0. So this should have been detected earlier.

[iirina]

All the Tensorflow jobs that used bazel built @ HEAD have been red for a while

https://ci.bazel.build/job/Global/job/TensorFlow/

I don't think anyone was checking on them.

[lberki]

Put then why did we have a green build recently? Are our CI machines not uniform enough?

[Kaju-Bubanja]

I was wondering what combination of bazel and gcc is now needed to successfully built tensorflow? It looks a bit confusing and I'm unsure of what versions to try and what specific commits of bazel to try. I currently have 0.16 and get this error but the bug is filled for 0.10 is there a version in between where this was fixed or do i need bazel < 0.10?