Possibly wrong link order / devtoolset-8 libstdc++ unknown symbols

[GregBowyer]

You are probably not going to like this one that much.

It seems that bazel + binutils + gcc + redhats devtoolset-8 gets angry regarding -lstdc++, this appears to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1570853

Recording of the bug

The following is a (long) recording of the bug, and workaround, the following documents it outright

Bazel issues with devtoolset-8 regarding libstdc++

It appears that bazel is adding -lstdc++ into the linker arguments early in the command line for gcc.
This freaks out oddball environments (such as the one provided by the software collections library).

SCL from RedHat provides a linker script in place of libstdc++ which does some trickery to fake out having an old libstdc++ installed on the system, but needing newer symbols for newer C++ features, versions and ABIs. This is seemingly done to allow for cross ABI compatibility

Reproducing

I have tested this in the environment we must compile for at $DAYJOB, I have not tested using stock RHEL or Centos but I suspect it might be an issue. It could also be a more general problem with link ordering.

Any bazel build using C++ will break including bootstrapping bazel itself

Setup a test env with docker

The following should give a test env that can reproduce the issue

docker run -it oraclelinux:slim-6

# Inside this env
yum install -y yum-security
yum updateinfo -y --verbose

yum-config-manager --enable -y ol6_latest
yum-config-manager --enable -y ol6_software_collections
yum updateinfo -y --verbose
yum install -y scl-utils

# Install some basics
yum install which curl wget unzip zip bzip2 patch xz

# Grab a compiler
yum install -y devtoolset-8-gcc devtoolset-8-gcc-c++ devtoolset-8-binutils devtoolset-8-toolchain

# Grab a python
yum install -y rh-python36-python

# Grab a jvm
rpm --import http://repos.azulsystems.com/RPM-GPG-KEY-azulsystems
curl -o /etc/yum.repos.d/zulu.repo http://repos.azulsystems.com/rhel/zulu.repo
yum install -y zulu-8

# Setup a environment faker to force the use of devtoolkit-8 as a compiler
cat > /opt/scl_enable <<EOF
#!/bin/bash
unset BASH_ENV PROMPT_COMMAND ENV
source scl_source enable devtoolset-8
source scl_source enable python27
EOF

chmod 755 /opt/scl_enable

# Grab bazel
mkdir /root/bazel_bootstrap
cd /root/bazel_bootstrap
wget https://github.com/bazelbuild/bazel/releases/download/1.2.1/bazel-1.2.1-dist.zip
unzip bazel-1.2.1-dist.zip

# The magical env is to force into SCL / devtoolkit
# It can be sourced, but that has historically proven to be unrealiable in our CI env
BASH_ENV=/opt/scl_enable ENV=/opt/scl_enable PROMPT_COMMAND=". /opt/scl_enable" ./compile.sh

... After a while bazel will fail to compile with some C++ linker error, normally complaining about some undefined symbol. The undef symbol will be something deep in libstdc++

What I think causes this

Looking at a failed link, we find that bazel produces linker args something like so

-o bazel-out/host/bin/external/com_google_protobuf/protoc
-Wl,-S
-fuse-ld=gold
-Wl,-no-as-needed 
-Wl,-z,relro,-z,now
-B/opt/rh/devtoolset-8/root/usr/bin
-pass-exit-codes
-lstdc++
-lm
-Wl,--gc-sections
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc/main.o                                                                     -Wl,--start-lib
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/code_generator.o                                                       
SNIP SNIP %<  ...
-Wl,--end-lib
-lpthread
-lm
-lpthread
-lm
-lpthread
-lm

Looking at the file libstdc++.so under devtoolkit-8 it looks like so

/* GNU ld script Use the shared library, but some functions are only in 
the static library, so try that secondarily.  */                       

OUTPUT_FORMAT(elf64-x86-64)
INPUT ( /usr/lib64/libstdc++.so.6 -lstdc++_nonshared )

Which is trickery to, as mentioned above, use ABI hacks, this confuses ld and I think removes the special handling for libstdc++

Moving libstdc++ to the bottom like so

-o bazel-out/host/bin/external/com_google_protobuf/protoc                                                                                  -Wl,-S
-fuse-ld=gold
-Wl,-no-as-needed
-Wl,-z,relro,-z,now
-B/opt/rh/devtoolset-8/root/usr/bin
-pass-exit-codes
-Wl,--gc-sections
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc/main.o
-Wl,--start-lib
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/code_generator.o
SNIP SNIP %<  ...
-Wl,--end-lib
-lpthread
-lm
-lpthread
-lm
-lpthread
-lm
-lstdc++
-lm

Allows for linking

Workaround

A workaround we have found is to export the following

BAZEL_LINKOPTS=-static-libstdc++ BAZEL_LINKLIBS=-l%:libstdc++.a

... technically this statically links libstdc++ and is probably not what some people want

[irengrig]

/cc @hlopko @oquenchil

[dalance]

The following Dockerfile seems to reproduce the same issue.

FROM centos:7

ADD https://copr.fedorainfracloud.org/coprs/vbatts/bazel/repo/epel-7/vbatts-bazel-epel-7.repo /etc/yum.repos.d

RUN yum install -y centos-release-scl
RUN yum install -y \
        devtoolset-8 \
        git \
        wget \
        bazel

SHELL [ "scl", "enable", "devtoolset-8"]

RUN git clone https://github.com/google/verible

RUN gcc --version; cd verible; bazel build --cxxopt='-std=c++17' //...

The following workaround is effective.

 BAZEL_LINKOPTS="-static-libstdc++:-lm" BAZEL_LINKLIBS="-l%:libstdc++.a" bazel build --cxxopt='-std=c++17' //...

[hlopko]

Making P2 since there is a workaround.

[oquenchil]

This change 906f5f1 introduces the fix so that the workaround is no longer needed.

The tracking issue for the flag flip is in #10905

[nehaljwani]

There is one more work around which doesn't require statically linking libstdc++. Use a linker script.

$ cat > /tmp/linker_hack_for_bazel <<EOF
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
INPUT ( -lstdc++ )
EOF

$ export BAZEL_LINKLIBS="/tmp/linker_hack_for_bazel"

This makes sure that -lstdc++ goes at the end of the link line.

[nukes]

solve this problem with:
BAZEL_LINKOPTS="" BAZEL_LINKLIBS="-lstdc++" bazel build .....

[stellaraccident]

I do not believe this is fixed: With devtoolset-9, which has the same libstdc++ linker script trickery, it is impossible to build bazel itself, and a look at the link params file for the failing invocation reveals that libstdc++ is still in the wrong order.

Steps to repro:

• docker run --rm -it quay.io/pypa/manylinux2014_x86_64 /bin/bash
• Download bazel dist and attempt to bootstrap bazel.
• Note linker errors for std::__throw_out_of_range_fmt and others, which were introduced in the dev toolset after release (and come in by way of the static library in the linker script).

I just verified this with bazel 3.7.2.

[stellaraccident]

Another egregious hack memorialized: https://github.com/stellaraccident/manylinux-bazel

[SuperBo]

@stellaraccident : thank you, you saved my day :D