Bazel dependency check for asan_blacklist.txt with clang-10

[asraa]

Description of the problem / feature request:

Envoy (and other bazel projects) are failing asan builds with clang-10 due to a dependency check. The exact error message:

Step #4: ERROR: /builder/home/.cache/bazel/_bazel_root/4e9824db8e7d11820cfa25090ed4ed10/external/boringssl/BUILD:131:1: undeclared inclusion(s) in rule '@boringssl//:crypto':
Step #4: this rule is missing dependency declarations for the following files included by 'external/boringssl/src/crypto/dh/params.c':
Step #4:   '/usr/local/lib/clang/10.0.0/share/asan_blacklist.txt'

Tracking issue on OSS-Fuzz : google/oss-fuzz#3093

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Install clang-10 and build a bazel project (eg Envoy) with asan. OSS-Fuzz's docker image runs with this. The clang-10 installation in the docker environment was reverted in google/oss-fuzz#3093, building before this commit will reproduce the error.

The issue was solved by using Clang-9 release version.

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

2.0.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

Replace this line with your answer.

Have you found anything relevant by searching the web?

The patch that may have caused this issue is llvm/llvm-project@03b84e4#diff-e2fe4c5daa679eec2fe3bad655f14dc0

Comment from Pete Collingbourne:

The change made it so that /usr/local/lib/clang/10.0.0/share/asan_blacklist.txt would appear in the depfile (-MF output) when passing -MD. Presumably bazel implements some sort of check that the files mentioned in the depfile are a subset of those in the rule's dependencies clause. Maybe the solution is to teach bazel that asan_blacklist.txt is a valid dependency of any asan compilation (or maybe all compilations, whichever is easier).

[m-gupta]

This is breaking Chrome OS fuzzer builds when building tensorflow (https://bugs.chromium.org/p/chromium/issues/detail?id=1042874)
Do we have a workaround in bazel?

[m-gupta]

Note any sanitizer blacklist, not just asan must be handled:
e.g. '/usr/lib64/clang/10.0.0/share/asan_blacklist.txt'
'/usr/lib64/clang/10.0.0/share/msan_blacklist.txt'
'/usr/lib64/clang/10.0.0/share/ubsan_blacklist.txt'

[asraa]

@dslomov Hey Dmitry, quick ping on this. It would really help us to unblock our fuzzer builds and view our coverage. Thank you!

[dslomov]

ping @oquenchil @lberki

[oquenchil]

I will take a look at this on Monday.

[oquenchil]

I tried to reproduce by building the image here https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-clang/Dockerfile.

I tried to build Bazel within the container but I keep bumping into missing tools that aren't in the image. Can you please provide me with an image that makes it easy for me to reproduce which also has the version of clang that causes the issue?

[oquenchil]

We have this image where we can build Bazel.

gcr.io/bazel-untrusted/ubuntu1404:java8 /

If you tell me how to install clang-10 there and which target I should try to build from which repo in order to reproduce, that would be helpful.

[asraa]

I think you were missing tools because we build bazel and Envoy in the base-builder, which is built on top of base-clang (bazel is built here https://github.com/asraa/oss-fuzz/blob/a155cb25879ce825299fad3eb8b189a7c332e7e5/projects/envoy/Dockerfile). I just reproduced in OSS-Fuzz directory by running python infra/helper.py build_image envoy; python infra/helper.py build_fuzzers --sanitizer address envoy in the oss-fuzz directory with the diff

diff --git a/projects/envoy/Dockerfile b/projects/envoy/Dockerfile
index 0be16774..d13eb9ca 100644
--- a/projects/envoy/Dockerfile
+++ b/projects/envoy/Dockerfile
@@ -16,7 +16,7 @@
 
 # TODO(https://github.com/google/oss-fuzz/issues/3093): Stop specifying the
 # image SHA once the bug is fixed.
-FROM gcr.io/oss-fuzz-base/base-builder@sha256:276813aef0ce5972db43c0230f96162003994fa742fb1b2f4e66c67498575c65
+FROM gcr.io/oss-fuzz-base/base-builder
 MAINTAINER htuch@google.com
 
 RUN apt-get update && apt-get -y install  \

@Dor1s

[oquenchil]

Thank you Asra. I can reproduce now.

[oquenchil]

So that is an error because we see it in the .d file because the file path doesn't match any of the declared include dirs. We get the include dirs from clang -E -xc++ - -v here

bazel/tools/cpp/unix_cc_configure.bzl

Line 125 in 469d855

def get_escaped_cxx_inc_directories(repository_ctx, cc, lang_flag, additional_flags = []):

This is where the check happens:

bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppCompileAction.java

Line 1080 in 469d855

private static boolean isDeclaredIn(

This will require clang adding a section in the output of clang -v, so that we can figure out what that directory is. Something like DataDir.

[m-gupta]

The sanitizer files are part of clang resource directory in the

/share location.

$ clang -print-resource-dir
/usr/lib64/clang/10.0.0

[oquenchil]

So just to test that this would fix the problem, I tried adding the following lines to

bazel/tools/cpp/unix_cc_configure.bzl

Line 143 in 469d855

return [

:

    m_list = [
        _prepare_include_path(repository_ctx, _cxx_inc_convert(p))
        for p in inc_dirs.split("\n")
    ]
    m_list.append(_prepare_include_path(repository_ctx, RESOURCE_DIR))
    return m_list

where RESOURCE_DIR is whatever clang --print-resource-dir prints and the blacklist.txt error disappeared. I then started getting errors with virtual include hdrs from protobuf not being found:

  'bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_includes/any_proto/google/protobuf/any.pb.h'
  'bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_includes/descriptor_proto/google/protobuf/descriptor.pb.h'
  'bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_includes/empty_proto/google/protobuf/empty.pb.h'
  'bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_includes/struct_proto/google/protobuf/struct.pb.h'
  'bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_includes/wrappers_proto/google/protobuf/wrappers.pb.h

And later after restarting the container but without changing anything, the error changed to:

  /usr/local/bin/clang @bazel-out/k8-fastbuild/bin/test/common/common/hash_fuzz_test_driverless-2.params)
Execution platform: @local_config_platform//:host
/usr/bin/ld.gold: error: cannot find -lFuzzingEngine
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_extra_spill_area'
/usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../x86_64-linux-gnu/crt1.o:function _start: error: undefined reference to 'main'

I have been trying to debug this for a while and I cannot get rid of those errors which seem unrelated. If you could modify the unix_cc_configure.bzl file adding the resource_dir, build Bazel from head and tell me if that fixes your problem, that would be great.

Also ideally if you know why I was getting those other errors I can try again myself.

[Dor1s]

/usr/bin/ld.gold: error: cannot find -lFuzzingEngine

-lFuzzingEngine is available inside OSS-Fuzz images, but isn't available anywhere else. The fuzzing engine has the main function, which is also not found as per the last log line.

[oquenchil]

I'm running the following command from within the container created by python infra/helper.py build_image envoy; python infra/helper.py build_fuzzers --sanitizer address envoy:

bazel build --verbose_failures --dynamic_mode=off --spawn_strategy=standalone --genrule_strategy=standalone --strip=never --copt=-fno-sanitize=vptr --linkopt=-fno-sanitize=vptr --define tcmalloc=disabled --define signal_trace=disabled --define ENVOY_CONFIG_ASAN=1 --copt -D__SANITIZE_ADDRESS__ --define force_libcpp=enabled --build_tag_filters=-no_asan --linkopt=-lc++ --linkopt=-pthread --conlyopt=-O1 --linkopt=-O1 --conlyopt=-fno-omit-frame-pointer --linkopt=-fno-omit-frame-pointer --conlyopt=-gline-tables-only --linkopt=-gline-tables-only --conlyopt=-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION --linkopt=-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION --conlyopt=-fsanitize=address --linkopt=-fsanitize=address --conlyopt=-fsanitize-address-use-after-scope --linkopt=-fsanitize-address-use-after-scope --conlyopt=-fsanitize=fuzzer-no-link --linkopt=-fsanitize=fuzzer-no-link --cxxopt=-O1 --linkopt=-O1 --cxxopt=-fno-omit-frame-pointer --linkopt=-fno-omit-frame-pointer --cxxopt=-gline-tables-only --linkopt=-gline-tables-only --cxxopt=-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION --linkopt=-DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION --cxxopt=-fsanitize=address --linkopt=-fsanitize=address --cxxopt=-fsanitize-address-use-after-scope --linkopt=-fsanitize-address-use-after-scope --cxxopt=-fsanitize=fuzzer-no-link --linkopt=-fsanitize=fuzzer-no-link --cxxopt=-stdlib=libc++ --linkopt=-stdlib=libc++ //test/extensions/filters/common/expr:evaluator_fuzz_test_driverless

I copied that Bazel command from the error I get when running the Python scripts. When running the Bazel command myself inside the OSS-Fuzz container, it says that it cannot find -lFuzzingEngine. I see OSS-Fuzz related files all over the container but not the library. No libFuzzingEngine.a or libFuzzingEngine.so anywhere.

Also isn't it weird that the command line has -lFuzzingEngine and at the same time --linkopt=-fsanitize=fuzzer-no-link?

[asraa]

Hm few things. Did you run python infra/helper.py build_image envoy; python infra/helper.py build_fuzzers --sanitizer address envoy in the master OSS-Fuzz branch? If so, that should produce no error since clang-9 is used in the base image. Running with the diff from the previous comment will reproduce the error by just running that command.

The error should come up with just a --config asan build, and without any of the fuzzing related infrastructure. I will try this locally when I run build_iamge with the diff.

The reason why there's a -fsanitize-no-link is because we compile without linking first, and then link to the whichever engine we want in a later stage.

[asraa]

Alright, double checked. This is how I repro'd without any fuzz related configurations:

1. I applied the diff above in OSS-Fuzz directory:

diff --git a/projects/envoy/Dockerfile b/projects/envoy/Dockerfile
index 0be16774..d13eb9ca 100644
--- a/projects/envoy/Dockerfile
+++ b/projects/envoy/Dockerfile
@@ -16,7 +16,7 @@
 
 # TODO(https://github.com/google/oss-fuzz/issues/3093): Stop specifying the
 # image SHA once the bug is fixed.
-FROM gcr.io/oss-fuzz-base/base-builder@sha256:276813aef0ce5972db43c0230f96162003994fa742fb1b2f4e66c67498575c65
+FROM gcr.io/oss-fuzz-base/base-builder
 MAINTAINER htuch@google.com

2. I ran python infra/helper.py build_image envoy
3. I ran docker run -it gcr.io/oss-fuzz/envoy:latest /bin/bash
4. I ran bazel build test/common/compressor:compressor_fuzz_test --config asan

I outputted:

Starting local Bazel server and connecting to it...
INFO: Analyzed target //test/common/compressor:compressor_fuzz_test (232 packages loaded, 11755 targets configured).
INFO: Found 1 target...
ERROR: /root/.cache/bazel/_bazel_root/4e9824db8e7d11820cfa25090ed4ed10/external/boringssl/BUILD:131:1: undeclared inclusion(s) in rule '@boringssl//:crypto':
this rule is missing dependency declarations for the following files included by 'external/boringssl/src/crypto/cipher_extra/e_rc4.c':
  '/usr/local/lib/clang/10.0.0/share/asan_blacklist.txt

[oquenchil]

Thank you again for the repro instructions Asra.

I got past the black.txt error in #10510 (comment).

I was trying to get the whole build to complete. First I got some protobuf headers not found errors, then the FuzzingEngine error and now (with just --config=asan), I'm getting this:

bazel-out/k8-fastbuild/bin/include/envoy/upstream/_virtual_includes/locality_lib/envoy/upstream/locality.h:17:12: error: no matching function for call to object of type 'absl::Hash<LocalityTuple>' (aka 'Hash<std::tuple<const std::__cxx11::basic_string<char> &, const std::__cxx11::basic_string<char> &, const std::__cxx11::basic_string<char> &> >')
    return absl::Hash<LocalityTuple>()({locality.region(), locality.zone(), locality.sub_zone()});
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_absl/absl/hash/internal/hash.h:924:10: note: candidate function not viable: cannot convert initializer list argument to 'const std::tuple<const std::__cxx11::basic_string<char> &, const std::__cxx11::basic_string<char> &, const std::__cxx11::basic_string<char> &>'
  size_t operator()(const T& value) const { return CityHashState::hash(value); }
         ^
1 error generated.

In any case, the blacklist.txt error is gone with my patched version of Bazel. I tested it by creating a custom example in your docker container with clang-10 using just a single cc_library and passing the flag --config=asan.

I would like to see this all the way through to make sure that Envoy builds properly but I'm going to need the right set of flags to avoid errors. I know that --config=asan is not enough because I built the image without your diff and that bazel command (which with release bazel doesn't give the blacklist.txt error) gives the same error in locality.h.

So ideally, without using your diff, you can start the docker container and find a bazel command that won't give any errors at all. After that I will try the same command in the container with your diff and using my patched bazel.

[asraa]

You're right, I was able to repro your error with the working image. I think there's something a little different about the OSS-Fuzz environment, I'm testing a working command in the clang-9 container for you to test. Will update

[asraa]

in retrospect, I should have realized this. The hash related errors are from --config libc++ missing, and the missing main and -lFuzzingEngine came up from building _driverless targets.

I again went to the shell that had the working version of clang, and this succeeds all the way through:
bazel build --verbose_failures --dynamic_mode=off --spawn_strategy=standalone --genrule_strategy=standalone --strip=never --define tcmalloc=disabled --config=asan --define force_libcpp=enabled --linkopt=-lc++ --linkopt=-pthread --cxxopt=-stdlib=libc++ --linkopt=-stdlib=libc++ //test/extensions/filters/common/expr:evaluator_fuzz_test

[iphydf]

Is there a workaround for bazel 2.2.0 until the next version is released? Also, when is the next version going to be released?

[iphydf]

The documentation at https://docs.bazel.build/versions/2.2.0/bazel-container.html is currently broken because of this. "Build the project with sanitizers by adding the –config=<asan/tsan/msan> build flag" does not work.

[oquenchil]

I can't think of a workaround. The built in include directories get written to a file, there are no environment variables to write additional includes to that file. I also can't think of a command line flag that will skip these checks.