fully_static_link is broken

[iceboy233]

Description of the problem / feature request:

fully_static_link is broken.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

WORKSPACE

BUILD

cc_binary(
    name = "a",
    srcs = ["a.cc"],
)

a.cc

int main() { new int; }

Build command and output:

% bazel build --features=fully_static_link :a
INFO: Analyzed target //:a (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /usr/local/google/home/iceboy/tmp/hellobazel/BUILD:1:1: Linking of rule '//:a' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc @bazel-out/k8-fastbuild/bin/a-2.params

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/k8-fastbuild/bin/_objs/a/a.pic.o:a.cc:function main: error: undefined reference to 'operator new(unsigned long)'
collect2: error: ld returned 1 exit status
Target //:a failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.190s, Critical Path: 0.10s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

release 0.27.0

Any other information, logs, or outputs that you want to share?

Could you also share the state-of-the-art way of doing mostly static build? That is, linking libc and libm dynamically while linking other libraries (including libstdc++) statically?

[meisterT]

cc @hlopko

[cblichmann]

@hlopko We're also having this problem. We need fully static binaries because of sandbox policies.
Bazel seems to use the C-compiler to link. If I run g++ @bazel-out/k8-fastbuild/bin/a-2.params, I get the expected, fully-static binary.

My repro case, slightly modified BUILD file:

cc_binary(
    name = "a",
    srcs = ["a.cc"],
    features = [
        "-pie",
        "fully_static_link",
    ],
    linkstatic = 1,
)

[cblichmann]

One way to link this with the C compiler is to wrap libstdc++ with the -Wl,--start-group and -Wl,--end-group options.
This leaks the underlying compiler in the build file, though, as for Clang, this would be different.

[hlopko]

gcc is not a problem, you'll get your libstdc++ linked statically when you pass -static-libstdc++ and -l:libstdc++.a.

The problem is how to put these flags in the right places on the command line. With Bazel 0.28 you'll be able to use BAZEL_LINKLIBS env var to set flags that will go after default link flags, after libraries to link, and after user link flags. That will allow you to link libstdc++ statically for the entire build.

If you need to decide on a per cc_binary basis whether to link libstdc++ statically, we still need to fix fully_static_link. Ftr, it never worked, so it's not a regression :P Fixing it is not hard, it's just work. We will probably fix it only after we move cc_configure to bazelbuild/rules_cc (which should happen before 1.0).

[cblichmann]

gcc is not a problem, you'll get your libstdc++ linked statically when you pass -static-libstdc++ and -l:libstdc++.a.

Ok, but then I can't easily switch compilers to Clang, for example.

The problem is how to put these flags in the right places on the command line. With Bazel 0.28 you'll be able to use BAZEL_LINKLIBS env var to set flags that will go after default link flags, after libraries to link, and after user link flags. That will allow you to link libstdc++ statically for the entire build.

This is actually not what I'm after. I only need a few targets to be fully statically linked.

If you need to decide on a per cc_binary basis whether to link libstdc++ statically, we still need to fix fully_static_link. Ftr, it never worked, so it's not a regression :P Fixing it is not hard, it's just work. We will probably fix it only after we move cc_configure to bazelbuild/rules_cc (which should happen before 1.0).

How so? It did seem to work (at least for us) in earlier releases (0.25.0 IIRC). Are there examples for using cc_configure even if it's still experimental?

[hlopko]

1. you can achieve similar thing with clang
2. ack
3. Are you sure? All the changes that I'm aware of related to fully_static_link are backwards compatible (they were just moving logic from place to place, but not changing the logic). Because of that I extrapolate it never worked. Re cc_configure, you're already using it, that the thing that autoconfigures your C++ toolchain when you don't specify a custom C++ toolchain. It's currently bundled with Bazel, and we have plans to move it out to rules_cc repository.

[cblichmann]

Re 3., kinda. Our build broke, once I updated our Docker container to the latest Bazel :)

Ok, so for now I'll work around the issue. I just really want people to be able to clone our repo (Sandboxed API) and just do a bazel build ... without additional options (at least at first).

[hlopko]

Interesting. And are you sure libstdc++ was linked statically before?

[cblichmann]

Yes. Full example:

touch WORKSPACE
cat <<EOF> BUILD.bazel
[cc_binary(
    name = "with" + ("out" if not opt else "") + "_linkopts",
    srcs = ["a.cc"],
    features = [
        "-pie",
        "fully_static_link",  # link libc statically
    ],
    linkopts = ["-l:libstdc++.a"] if not opt else [],
    linkstatic = 1,  # prefer static libraries
) for opt in [
    True,
    False,
]]
EOF
printf '#include <cstdio>\nint main() { new int; printf("hi!\\n"); }\n' > a.cc

Then, to reproduce with Bazelisk:

for ver in 0.25.0 0.26.0 0.27.0; do
  echo "$ver ========================="
  USE_BAZEL_VERSION=$ver bazelisk clean
  USE_BAZEL_VERSION=$ver bazelisk build :all
  ldd bazel-bin/with{_,out_}linkopts
done

Output is:

0.25.0 =========================
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed 2 targets (10 packages loaded, 69 targets configured).
INFO: Found 2 targets...
INFO: Elapsed time: 1.571s, Critical Path: 0.23s
INFO: 4 processes: 4 linux-sandbox.
INFO: Build completed successfully, 11 total actions
bazel-bin/with_linkopts:
        not a dynamic executable
bazel-bin/without_linkopts:
        not a dynamic executable
0.26.0 =========================
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed 2 targets (10 packages loaded, 69 targets configured).
INFO: Found 2 targets...
INFO: Elapsed time: 1.574s, Critical Path: 0.22s
INFO: 4 processes: 4 linux-sandbox.
INFO: Build completed successfully, 11 total actions
bazel-bin/with_linkopts:
        not a dynamic executable
bazel-bin/without_linkopts:
        not a dynamic executable
0.27.0 =========================
Starting local Bazel server and connecting to it...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed 2 targets (11 packages loaded, 67 targets configured).
INFO: Found 2 targets...
INFO: Deleting stale sandbox base /usr/local/google/home/cblichmann/.cache/bazel/_bazel_cblichmann/6064a55415da04bdabd2d61e522140c5/sandbox
ERROR: /usr/local/google/home/cblichmann/__bazel/BUILD.bazel:1:2: Linking of rule '//:with_linkopts' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc @bazel-out/k8-fastbuild/bin/with_linkopts-2.params

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/k8-fastbuild/bin/_objs/with_linkopts/a.pic.o:a.cc:function main: error: undefined reference to 'operator new(unsigned long)'
collect2: error: ld returned 1 exit status
INFO: Elapsed time: 1.647s, Critical Path: 0.21s
INFO: 3 processes: 3 linux-sandbox.
FAILED: Build did NOT complete successfully
bazel-bin/with_linkopts:
ldd: bazel-bin/with_linkopts: No such file or directory
bazel-bin/without_linkopts:
        not a dynamic executable

[cblichmann]

Does this example help?

[hlopko]

I didn't have time to look into it, and I'll be out of office for a month soon. So unless @scentini or @oquenchil won't investigate in the meantime our of their own interest, I'll only take a look after the leave. I'm sorry for that.

Since you already have a nice repro test, would you mind bisecting Bazel commits to find the culprit? That would speed up fixing this tremendously.

[karamach]

Just wondering what the status of this issue. Is there a plan to fix it.

[iceboy233]

For who has the same problem, I put the following flags in .bazelrc to statically build binaries:

build --action_env=BAZEL_LINKLIBS=-l%:libstdc++.a:-lm
build --action_env=BAZEL_LINKOPTS=-static-libgcc

This will statically link libc++ and libgcc, dynamically link libc and libm, and be ABI compatible with most linux distros.

[ulfjack]

It seems odd that this is a P3 when there is a significant number of projects affected by this, including Envoy (they have a local workaround).

I was able to build statically with this:

    linkopts = ["-l:libstdc++.a"],
    features = ["fully_static_link"],

It should be straightforward to fix cc_configure to add -l:libstdc++.a to the end of the link command, right?

[ulfjack]

(To be clear, my proprietary project is also affected. We have a case where we'd like to link statically and I just spent ~an hour to figure out how to do it.)

[benjaminp]

I thought this was fixed by --incompatible_linkopts_to_linklibs.

[ulfjack]

Cool! Yeah, that seems to work. Rollout bug is #10905, which seems to be scheduled for 4.0.

[oquenchil]

If this is still an issue, please re-open.