Building Envoy with Bazel
To build Envoy with Bazel in a production environment, where the Envoy dependencies are typically independently sourced, the following steps should be followed:
- Install the latest version of Bazel in your environment.
- Configure, build and/or install the Envoy dependencies.
- Configure a Bazel WORKSPACE to point Bazel at the Envoy dependencies. An example is provided in the CI Docker image WORKSPACE and corresponding BUILD files.
bazel build --package_path %workspace%:<path to Envoy source tree> //source/exe:envoy-staticfrom the directory containing your WORKSPACE.
Quick start Bazel build for developers
As a developer convenience, a WORKSPACE and rules for building a recent version of the various Envoy dependencies are provided. These are provided as is, they are only suitable for development and testing purposes. The specific versions of the Envoy dependencies used in this build may not be up-to-date with the latest security patches. See this doc for how to update or override dependencies.
- Install the latest version of Bazel in your environment.
- Install external dependencies libtool, cmake, ninja, realpath and curl libraries separately. On Ubuntu, run the following command:
sudo apt-get install \ libtool \ cmake \ realpath \ clang-format-7 \ automake \ ninja-build \ curl
On Fedora (maybe also other red hat distros), run the following:
dnf install cmake libtool libstdc++
On OS X, you'll need to install several dependencies. This can be accomplished via Homebrew:
brew install coreutils # for realpath brew install wget brew install cmake brew install libtool brew install go brew install bazel brew install automake brew install ninja
Envoy compiles and passes tests with the version of clang installed by XCode 9.3.0: Apple LLVM version 9.1.0 (clang-902.0.30).
- Install Golang on your machine. This is required as part of building BoringSSL and also for Buildifer which is used for formatting bazel BUILD files.
go get github.com/bazelbuild/buildtools/buildifierto install buildifier
bazel build //source/exe:envoy-staticfrom the Envoy source directory.
Building Bazel with the CI Docker image
Bazel can also be built with the Docker image used for CI, by installing Docker and executing:
./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev'
See also the documentation for developer use of the CI Docker image.
Using a compiler toolchain in a non-standard location
By setting the
LD_LIBRARY_PATH in the environment that Bazel executes from as
appropriate, an arbitrary compiler toolchain and standard library location can be specified. One
slight caveat is that (at the time of writing), Bazel expects the binutils in
$(dirname $CC) to be
as instead of
Supported compiler versions
Though Envoy has been run in production compiled with GCC 4.9 extensively, we now require GCC >= 5 due to known issues with std::string thread safety and C++14 support. Clang >= 4.0 is also known to work.
Clang STL debug symbols
By default Clang drops some debug symbols that are required for pretty printing to work correctly.
More information can be found here. The easy solution
is to set
--copt=-fno-limit-debug-info on the CLI or in your .bazelrc file.
Testing Envoy with Bazel
All the Envoy tests can be built and run with:
bazel test //test/...
bazel test //test/common/http:async_client_impl_test
To observe more verbose test output:
bazel test --test_output=streamed //test/common/http:async_client_impl_test
It's also possible to pass into an Envoy test additional command-line args via
example, for extremely verbose test debugging:
bazel test --test_output=streamed //test/common/http:async_client_impl_test --test_arg="-l trace"
By default, testing exercises both IPv4 and IPv6 address connections. In IPv4 or IPv6 only environments, set the environment variable ENVOY_IP_TEST_VERSIONS to "v4only" or "v6only", respectively.
bazel test //test/... --test_env=ENVOY_IP_TEST_VERSIONS=v4only bazel test //test/... --test_env=ENVOY_IP_TEST_VERSIONS=v6only
By default, tests are run with the gperftools heap checker enabled in "normal" mode to detect leaks. For other mode options, see the gperftools heap checker documentation. To disable the heap checker or change the mode, set the HEAPCHECK environment variable:
# Disables the heap checker bazel test //test/... --test_env=HEAPCHECK= # Changes the heap checker to "minimal" mode bazel test //test/... --test_env=HEAPCHECK=minimal
Bazel will by default cache successful test results. To force it to rerun tests:
bazel test //test/common/http:async_client_impl_test --cache_test_results=no
Bazel will by default run all tests inside a sandbox, which disallows access to the
local filesystem. If you need to break out of the sandbox (for example to run under a
local script or tool with
you can run the test with
bazel test //test/common/http:async_client_impl_test --strategy=TestRunner=standalone --run_under=/some/path/foobar.sh
Stack trace symbol resolution
Envoy can produce backtraces on demand and from assertions and other fatal
actions like segfaults. Where supported, stack traces will contain resolved
symbols, though not include line numbers. On systems where absl::Symbolization is
not supported, the stack traces written in the log or to stderr contain addresses rather
than resolved symbols. The
tools/stack_decode.py script exists to process the output
and do symbol resolution including line numbers, to make the stack traces useful.
Any log lines not relevant to the backtrace capability
are passed through the script unchanged (it acts like a filter).
The script runs in one of two modes. If passed no arguments it anticipates Envoy (or test) output on stdin. You can postprocess a log or pipe the output of an Envoy process. If passed some arguments it runs the arguments as a child process. This enables you to run a test with backtrace post processing. Bazel sandboxing must be disabled by specifying standalone execution. Example command line:
bazel test -c dbg //test/server:backtrace_test --run_under=`pwd`/tools/stack_decode.py --strategy=TestRunner=standalone --cache_test_results=no --test_output=all
You will need to use either a
dbg build type or the
opt build type to get symbol
information in the binaries.
By default main.cc will install signal handlers to print backtraces at the
location where a fatal signal occurred. The signal handler will re-raise the
fatal signal with the default handler so a core file will still be dumped after
the stack trace is logged. To inhibit this behavior use
--define=signal_trace=disabled on the Bazel command line. No signal handlers will
Running a single Bazel test under GDB
tools/bazel-test-gdb //test/common/http:async_client_impl_test -c dbg
-c dbg Bazel option at the end of the command line the test
binaries will not include debugging symbols and GDB will not be very useful.
Additional Envoy build and test options
In general, there are 3 compilation modes that Bazel supports:
-O0, aimed at developer speed (default).
-O2 -DNDEBUG -ggdb3, for production builds and performance benchmarking.
-O0 -ggdb3, no optimization and debug symbols.
You can use the
-c <compilation_mode> flag to control this, e.g.
bazel build -c opt //source/exe:envoy-static
bazel test -c dbg --config=asan //test/...
The ASAN failure stack traces include line numbers as a result of running ASAN with a
dbg build above. If the
stack trace is not symbolized, try setting the ASAN_SYMBOLIZER_PATH environment variable to point to the
llvm-symbolizer binary (or make sure the llvm-symbolizer is in your $PATH).
If you have clang-5.0 or newer, additional checks are provided with:
bazel test -c dbg --config=clang-asan //test/...
Similarly, for thread sanitizer (TSAN) testing:
bazel test -c dbg --config=clang-tsan //test/...
Log verbosity is controlled at runtime in all builds.
Disabling optional features
The following optional features can be disabled on the Bazel build command-line:
- Hot restart with
- Google C++ gRPC client with
- Backtracing on signals with
Enabling optional features
The following optional features can be enabled on the Bazel build command-line:
- Exported symbols during linking with
--define exported_symbols=enabled. This is useful in cases where you have a lua script that loads shared object libraries, such as those installed via luarocks.
- Perf annotation with
--define perf_annotation=enabled(see source/common/common/perf_annotation.h for details).
Envoy uses a modular build which allows extensions to be removed if they are not needed or desired. Extensions that can be removed are contained in extensions_build_config.bzl. Use the following procedure to customize the extensions for your build:
- The Envoy build assumes that a Bazel repository named
@envoy_build_configexists which contains the file
@envoy_build_config//:extensions_build_config.bzl. In the default build, a synthetic repository is created containing extensions_build_config.bzl. Thus, the default build has all extensions.
- Start by creating a new Bazel workspace somewhere in the filesystem that your build can access.
This workspace should contain:
- Empty WORKSPACE file.
- Empty BUILD file.
- A copy of extensions_build_config.bzl.
- Comment out any extensions that you don't want to build in your file copy.
To have your local build use your overridden configuration repository there are two options:
- Use the
--override_repositoryCLI option to override the
- Use the following snippet in your WORKSPACE before you load the Envoy repository. E.g.,
workspace(name = "envoy") local_repository( name = "envoy_build_config", # Relative paths are also supported. path = "/somewhere/on/filesystem/envoy_build_config", ) local_repository( name = "envoy", # Relative paths are also supported. path = "/somewhere/on/filesystem/envoy", ) ...
The default maximum number of stats in shared memory, and the default
maximum length of a cluster/route config/listener name, can be
overridden at compile-time by defining
ENVOY_DEFAULT_MAX_OBJ_NAME_LENGTH, respectively, to the desired
value. For example:
bazel build --copts=-DENVOY_DEFAULT_MAX_STATS=32768 --copts=-DENVOY_DEFAULT_MAX_OBJ_NAME_LENGTH=150 //source/exe:envoy-static
Release builds should be built in
opt mode, processed with
strip and have a
.note.gnu.build-id section with the Git SHA1 at which the build took place.
They should also ignore any local
.bazelrc for reproducibility. This can be
bazel --bazelrc=/dev/null build -c opt //source/exe:envoy-static.stripped.stamped
One caveat to note is that the Git SHA1 is truncated to 16 bytes today as a result of the workaround in place for https://github.com/bazelbuild/bazel/issues/2805.
To generate coverage results, make sure you have
gcovr 3.3 in your
PATH (or set
point at it) and are using a GCC toolchain (clang does not work currently, see
https://github.com/envoyproxy/envoy/issues/1000). Then run:
The summary results are printed to the standard output and the full coverage
report is available in
Coverage for every PR is available in Circle in the "artifacts" tab of the coverage job. You will need to navigate down and open "coverage.html" but then you can navigate per normal. NOTE: We have seen some issues with seeing the artifacts tab. If you can't see it, log out of Circle, and then log back in and it should start working.
The latest coverage report for master is available here.
It's also possible to specialize the coverage build to a single test target. This is useful
when doing things like exploring the coverage of a fuzzer over its corpus. This can be done with
VALIDATE_COVERAGE environment variables, e.g.:
COVERAGE_TARGET=//test/common/common:base64_fuzz_test VALIDATE_COVERAGE=false test/run_envoy_bazel_coverage.sh
Cleaning the build and test artifacts
bazel clean will nuke all the build/test artifacts from the Bazel cache for
Envoy proper. To remove the artifacts for the external dependencies run
bazel clean --expunge.
If something goes really wrong and none of the above work to resolve a stale build issue, you can
always remove your Bazel cache completely. It is likely located in
Adding or maintaining Envoy build rules
Bazel performance on (virtual) machines with low resources
If the (virtual) machine that is performing the build is low on memory or CPU
resources, you can override Bazel's default job parallelism determination with
--jobs=N to restrict the build to at most
N simultaneous jobs, e.g.:
bazel build --jobs=2 //source/...
Debugging the Bazel build
When trying to understand what Bazel is doing, the
are useful. To have Bazel provide verbose output on which commands it is executing:
bazel build -s //source/...
To have Bazel emit to a text file the rationale for rebuilding a target:
bazel build --explain=file.txt //source/...
To get more verbose explanations:
bazel build --explain=file.txt --verbose_explanations //source/...
Resolving paths in bazel build output
Sometimes it's useful to see real system paths in bazel error message output (vs. symbolic links).
tools/path_fix.sh is provided to help with this. See the comments in that file.
tools/gen_compilation_database.py to generate
a JSON Compilation Database. This could be used
with any tools (e.g. clang-tidy) compatible with the format.