Add support for OpenSSL?

[tmshort]

Right now, envoyproxy uses BoringSSL, and does not have a mechanism to substitute another TLS stack. Specifically, I am thinking of OpenSSL. Is there any interest in adding support to compile envoy with OpenSSL?

A follow-on question is whether dynamic linking of BoringSSL/OpenSSL would be supported.

[mattklein123]

The answer is almost definitely not, as we are slowly migrating the code to use BoringSSL specific constructs and features.

[mattklein123]

cc @PiotrSikora @ggreenway @davidben

@tmshort can you also provide some info on why you don't want to use BoringSSL?

[tmshort]

• We have standardized on OpenSSL, and has an internal distribution with additional changes; we don't want to have to maintain those changes in multiple different repositories.
• Lack of FIPS support in BoringSSL (I know about BoringSSL FIPS), it claims to be in progress, but I don't know how old that is. The focus of OpenSSL's next release is to provide a FIPS compliant version.

[mattklein123]

@tmshort thx for the info. I will let @davidben and @PiotrSikora comment on BoringSSL FIPS support, but in general would love to understand other missing features so we can make everyone happy with BoringSSL.

[agl]

The current version of the BoringSSL FIPS document lists the CMVP certificate number and links to the security policy. Since BoringSSL is open-source, anyone can build and use it as defined in the security policy. (BoringSSL does not need to be dynamically linked to be FIPS compliant.)

Some things that are missing in BoringSSL are because we've not had a use for them and could be supported. Some things were affirmatively removed and are unlikely to be supported. But we can give a breakdown if you have a list of your differences.

[PiotrSikora]

Since I don't believe Akamai will switch to BoringSSL, let me answer this from another perspective.

We're slowly moving to BoringSSL-specific APIs, so default Envoy's TLS stack is going to use BoringSSL and I don't envision OpenSSL being able to be used as a drop-in replacement for that.

However, Envoy has pluggable TransportSocket, which means that there is nothing stopping you from writing standalone OpenSSL-based transport that can be plugged into configuration, similarly to how Envoy-based Istio Proxy supports ALTS.

[mattklein123]

@PiotrSikora that's a good point. I suppose we could support an OpenSSL transport socket implementation in the repo, though, I'm not sure if there will be enough community interest to justify adding to the main repo (and the confusion it might cause for what we recommend for most users).

[ggreenway]

I'm not sure that you could compile both OpenSSL and BoringSSL into the same executable. I think they would have symbol collisions.

[PiotrSikora]

@ggreenway good point, perhaps we could offer an option to build Envoy without BoringSSL, similarly to how other dependencies be elided.

@mattklein123 outside of a few companies that are invested in OpenSSL, I doubt people are going to care enough to have 2 officially supported TLS stacks, and like you said, it's only going to lead to confusion, so I'd suggest that we keep it outside of the main repo (but maybe still under envoyproxy org on GitHub).

[PiotrSikora]

(assuming there is strong enough interest for someone to create and maintain OpenSSL-based TransportSocket in the first place).

[agl]

I'm not sure that you could compile both OpenSSL and BoringSSL into the same executable. I think they would have symbol collisions.

If you're using OpenSSL as a shared object (which you must for their FIPS build, I believe) then you could make it work with some proxy symbols and a linker script to hide the other symbols from the dynamic exports table. One would need to be careful to use the correct headers in the correct place, however.

[tmshort]

Thanks, @agl, we were unaware of the current FIPS status. It's fairly easy for us to maintain a local BoringSSL repo if needed, and since Envoy won't use the our features (we are slowing pushing them upstream, so as not to overwhelm), we likely won't attempt to maintain them in BoringSSL.

[mattklein123]

I'm going to go ahead and close this out as answered. @tmshort feel free to reopen if you want to discuss more about this or have other questions.

[knrc]

FYI Red Hat are working on switching out BoringSSL for OpenSSL for the same reasons, we are not allowed to ship BoringSSL because of our requirements for FIPS compliance. We would like to have a conversation with any other interested parties and come up with a solution which enables the switch to be made but without forcing the community to switch back to OpenSSL.

If you are interested then please reach out to me.

[gdimitrov-vmware]

@tmshort @knrc I am evaluating use of Envoy at Vmware and we have the exact same issues with BoringSSL's certification and would prefer to use OpenSSL.

Here are some caveats with the Certified version of BoringSSL:

• BoringSSL didn't include DH or ECDH; ECDH(E) is preferred to RSA nowadays for TLS connections.
• technically only BoringCrypto built from that exact Git hash is certified. This will make for a complicated build if Envoy has moved to a different git hash
• BorinSSL certification only covers Ubuntu so using on any other OS is not certified.

[agl]

@gdimitrov-vmware There are certainly valid reasons for wanting to use OpenSSL here, but I don't think any that you listed are correct:

• BoringSSL certainly supports ECDHE. ECDH is the, extremely obscure, fixed-DH mode of TLS; I'm not sure that OpenSSL supports that. It is the case that BoringSSL doesn't support DHE in TLS, but it's rarely used these days.
• You have to follow the security policy and build the exact version for any FIPS module, OpenSSL included.
• BoringSSL's certification covers any Linux 3.x or Linux 4.x kernel on x86-64, POWER8, or POWER9. (Although note that the next version will drop Linux 3.x and POWER8.)

[tmshort]

ECDH is not recommended, certainly. I think the concern may have been ECDHE in FIPS mode (or am I misreading)?

This is BoringSSL's current tested configurations (the link is in an @agl post above) :

• Ubuntu Linux 14.04 LTS running on Intel Xeon E5 without PAA (clang Compiler Version 4.0.0)
• Ubuntu Linux 15.04 running on POWER8 without PAA (clang Compiler Version 4.0.0)
• Ubuntu Linux 16.04 running on Intel Xeon E5 with PAA (clang Compiler Version 4.0.0)
• Ubuntu Linux 17.04 running on POWER8 with PAA (clang Compiler Version 4.0.0)
• Ubuntu Linux 17.04 running on POWER9 with PAA (clang Compiler Version 4.0.0) (single-user mode)

I'm not sure you can claim any 3.x or 4.x kernels based on this, but only those kernel versions in the given Ubuntu release (e.g. 4.4.0 for Ubuntu 16.04) and also that it only qualifies on the Xeon E5 processor (for those interested in x86_64). My reading of FIPS certifications is that it is a stickler when it comes to tested configurations, and it likely means that any user of Envoy that doesn't match the exact FIPS configuration (of either BoringSSL or OpenSSL) would need their own certification.

Given that I might need to get my own certification, I would prefer to do it for one software package on a given OS/processor/compiler combination, rather than two software packages.

[kscguru]

I work with @gdimitrov-vmware ...

• Some OS modification is allowed but nobody really knows how much. Some users would be okay with the risk, some wouldn't and would contract with a lab and re-certify to alleviate the risk. OpenSSL has a project goal of making "private" validations easy, whereas BoringCrypto ... does not.
• Exact sources are indeed a requirement. One would just have to be clever enough to build Envoy while ensuring BoringCrypto sources match a specific git commit.
• ECDHE is much harder. (Agree DH/DHE/ECDH are irrelevant in practice). In theory, ECDHE requires some sort of SP800-56A validation (most would pick SP800-56Arev2 vendor affirmation). The BoringCrypto CMVP certificate / security policy is missing it, whereas OpenSSL has it. It would read something like "EC DH Key Agreement (NNN bits of cryptographic strength)"; BoringCrypto says that for RSA but not EC DH. BoringCrypto certainly implements ECDHE - it's just not validated via FIPS.

I doubt anyone short of BoringCrypto's FIPS submitter could answer whether BoringCrypto is intended to cover ECDHE. I would also guess that if it doesn't, it's unintentional and maybe we'd see some action from BoringCrypto eventually. But the omission of "EC DH Key Agreement" from the FIPS certificate is hard to interpret differently.

[agl]

Some OS modification is allowed but nobody really knows how much. Some users would be okay with the risk, some wouldn't and would contract with a lab and re-certify to alleviate the risk. OpenSSL has a project goal of making "private" validations easy, whereas BoringCrypto ... does not.

It sounds like you really want to use OpenSSL, and that's ok! But if you want to do a validation of BoringSSL on a different platform then I can give you the contacts for it. We're getting pretty good at it now and, for an already evaluated source-code revision, it should be a doable in a day's work. (At least on the external side, I'm sure the lab has to do a chunk of work too.)

As for rendering opinions about operational environments, I can point you to section one of the approved security policy but you would have to contact a CMVP lab for a meaningful opinion for your particular context.

BoringCrypto says that for RSA but not EC DH. BoringCrypto certainly implements ECDHE - it's just not validated via FIPS.

Our customers who drove the FIPS validation had different requirements. However, the 2018 validation (which has been submitted to NIST already) covers “KAS ECC” key agreement in the FIPS module. So that'll be covered by the updated BoringSSL module when that certificate is issued if required.

(The KAS ECC was added for a non-FIPS customer requirement that happened to be satisfied by that CAVP certificate. Since we then had the testing wired up anyway, we included it in the 2018 CMVP testing.)

[kscguru]

However, the 2018 validation (which has been submitted to NIST already) covers “KAS ECC” key agreement in the FIPS module.

Brilliant. My "FIPS ECDHE eventually" guess became "FIPS ECDHE already done, the paperwork just isn't approved yet". Made my day, my complements.
(@agl, NIST treats hypervisors as a different platform than physical hardware. Given my / @gdimitrov-vmware 's employer... we always re-cert.)

The complexity of a fixed-hash build and dealing with "other" platforms are substantial, but are operational and not technical. The important detail here is that FIPS adds a lot of complexity that should be irrelevant to Envoy itself.

[agl]

we always re-cert

(I don't want to put it on a public GitHub issue, but I can point you at the right person at a lab and give you our playbook for running all the required tests as smoothly as possible, if you're interested.)

[PiotrSikora]

FWIW, Envoy tracks chromium-stable branch of BoringSSL that follows Chrome stable releases, which means that we update BoringSSL every 6 weeks, so ~9x per year, in case someone wants to re-certify each version of BoringSSL used in Envoy.

[knrc]

To give you all a status update we (Red Hat) have been working on this for some time and are at the point where we believe we have a version of envoy using OpenSSL ready for testing, we should be starting this internally next week. Once we are happy with this we are going to move the discussion to the upstream envoy community and work there to determine the best solution for supporting it, we suspect it will be through an agreed abstraction where the support for OpenSSL would be through a separate open source project and not necessarily supported directly by the upstream envoy community but obviously this will be a decision for the community.

[nickolaev]

@knrc is there any progress update you can share?

[georgi-d]

@knrc Could you also share some details on the approach you are using for adding OpenSSL support?

Thanks

[knrc]

@georgi-d You can take a look at the work Bill has been doing, he's tidying up a few final things but it is working and passing the tests.

https://github.com/bdecoste/envoy/tree/openssl

https://github.com/bdecoste/proxy/tree/openssl

This work is a PoC to investigate the scope of the changes, we still need to discuss this with the community and @bdecoste will be driving that.

[mattklein123]

@knrc @bdecoste can you email me at mklein@lyft.com so that we can discuss this and I can loop in various folks? I would like to get a conversation going sooner rather than later before we get to the larger community conversation. Thank you!

[thorsten-avaya]

I have built a FIPS compliant version of BoringSSL with a statically linked BoringCrypto. My goal is to build Istio with an Envoy proxy making use of that FIPS compliant version of BoringSSL.

My next step is to build Envoy. I did a git clone https://github.com/envoyproxy/envoy.git, followed by a bazel build --package_path %workspace%:/usr/thorsten/ENVOY/MASTER/envoy/ //source/exe:envoy. That resulted in:

WARNING: Output base '/home/thorsten/.cache/bazel/_bazel_thorsten/5f642dcc1f744c8d2d608658cf3d5214' is on NFS. This may lead to surprising failures and undetermined behavior.
External dependency cache directory /home/thorsten/.cache/bazel/_bazel_thorsten/5f642dcc1f744c8d2d608658cf3d5214/external/envoy_deps_cache_2c744dffd279d7e9e0910ce594eb4f4f
make: Entering directory `/home/thorsten/.cache/bazel/_bazel_thorsten/5f642dcc1f744c8d2d608658cf3d5214/external/envoy_deps'
ERROR: Analysis of target '//source/exe:envoy' failed; build aborted: no such package '@com_github_golang_protobuf//proto': BUILD file not found on package path

---

bazel version: Build label: 0.19.0- (@non-git) / bazel-0.19.0-2.el7.x86_64

1. What am I doing wrong - causing the above error?
2. What configuration do I need to change to point and link in the FIPS compliant boringssl/build ?
3. Do you know of any instructions to build istio with a custom Envoy proxy?

Thanks,
Thorsten

[PiotrSikora]

1. What am I doing wrong - causing the above error?

You're invoking Bazel in a bit weird way, you should be able to simply call bazel build //source/exe:envoy from Envoy's directory.

But if the error persists, then perhaps, as the error suggests, the issue is with ~/.cache/bazel being on NFS, so you might try to put your cache elsewhere, e.g. keep it in /tmp with:

rm -rf ~/.cache/bazel/
mkdir -p /tmp/bazel-cache
ln -s /tmp/bazel-cache ~/.cache/bazel

2. What configuration do I need to change to point and link in the FIPS compliant boringssl/build ?

BoringSSL is build and statically linked during Envoy's build process, so you'd need to modify build rules to select FIPS-tagged commit of BoringSSL and compile it with -DBORINGSSL_FIPS=1.

Unfortunately, this is a bit more complex than it should be, because Bazel rules in BoringSSL don't support FIPS. Perhaps we should migrate back to using CMake for the time being? cc @agl

3. Do you know of any instructions to build istio with a custom Envoy proxy?

You need to build custom Istio Proxy Docker image against your custom Envoy repository (see: https://github.com/istio/proxy/blob/master/istio.deps), then use it in your Istio clusters.

[thorsten-avaya]

Thank you @PiotrSikora for your quick response. I removed the cache and linked it to the local fs, as you suggested. Unfortunately the same problem persists (Good news - I no longer see the NFS issue):

git clone https://github.com/istio/envoy
cd envoy
bazel build //source/exe:envoy
...
INFO: Repository rule 'com_github_golang_protobuf' returned: {"remote": "https://github.com/golang/protobuf", "commit": "31e0d063dd98c052257e5b69eeb006818133f45c", "shallow_since": "2018-10-03", "init_submodules": False, "verbose": False, "strip_prefix": "", "patches": [Label("@io_bazel_rules_go//third_party:com_github_golang_protobuf-gazelle.patch"), Label("@io_bazel_rules_go//third_party:com_github_golang_protobuf-extras.patch")], "patch_tool": "patch", "patch_args": ["-p1"], "patch_cmds": [], "name": "com_github_golang_protobuf"}
ERROR: Analysis of target '//source/exe:envoy' failed; build aborted: no such package '@com_github_golang_protobuf//proto': BUILD file not found on package path
INFO: Elapsed time: 53.172s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (216 packages loaded, 7682 targets configured)

Can you think of another reason for the above error? Note that many other targets build before this error.

Based on the answer to my 2nd question, it sounds like you describe modifying the Envoy build process to point to a FIPS tagged version of BoringSSL and building BoringSSL as part of that process. This appears to be cumbersome due to the reasons you discuss above. Is there a way to avoid the actual building of BoringSSL and merely link in libraries from an already built FIPS compliant version of BoringSSL (I've built BoringSSL under ../boringssl/build)? Could this be a workaround to "Bazel rules in BoringSSL don't support FIPS." This may be a silly question, but is there a way to build Envoy using bazel, and having bazel build BoringSSL using cmake/make internally?

My ultimate goal is to build Istio with Envoy using a FIPS compliant version of BoringSSL. Let me ask this: how would you go about achieving such goal?

Thanks again for your help,
Thorsten

[PiotrSikora]

@thorsten-avaya but you can now build Envoy with BoringSSL FIPS.

[tmshort]

BoringSSL does not support post-handshake authentication for TLSv1.3; last I knew. Although using PHA would require changes in how Envoy handles certificate authentication.

[davidben]

Note TLS-level post-handshake authentication is incompatible with both HTTP/2 and QUIC. It's a weird HTTP/1.1-only hack.

[tmshort]

URL for building Envoy with BoringSSL FIPS has changed.

[tmshort]

@davidben Regardless, neither stack offers TLSv1.3 + FIPS, yet.

Please note that the FIPS-compliant build is based on an older version of BoringSSL than the non-FIPS build, and it predates the final version of TLS 1.3.

[daganibhanu]

I'm trying to build envoy docker image for which I'm creating a binary first from source code, with Boring SSL FIPS.

I'm following the approach given in https://github.com/envoyproxy/envoy/tree/master/ci

Since boring SSL is built and linked during envoy built process, I'm passing the required flag --define boringssl=fips in the build options as below

IMAGE_NAME=envoyproxy/envoy-build-ubuntu http_proxy=http://proxy.foo.com:8080 https_proxy=http://proxy.bar.com:8080 BAZEL_BUILD_OPTIONS='--define boringssl=fips' ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'

the binary got generated successfully at a defined location.

as mentioned in the link the correctness of the FIPS build can be verified by checking the presence of BoringSSL-FIPS in the --version output.

I have check the version, but it is still saying only Boring SSL(with out FIPS) as below:

e95ef6b/1.10.0/Clean/RELEASE/BoringSSL

Am I missing anything in making this FIPS compliant? please suggest.

[PiotrSikora]

IMAGE_NAME=envoyproxy/envoy-build-ubuntu http_proxy=http://proxy.foo.com:8080 https_proxy=http://proxy.bar.com:8080 BAZEL_BUILD_OPTIONS='--define boringssl=fips' ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'

Try with BAZEL_BUILD_OPTIONS passed to the command executed inside Docker:

IMAGE_NAME=envoyproxy/envoy-build-ubuntu http_proxy=http://proxy.foo.com:8080 https_proxy=http://proxy.bar.com:8080 ./ci/run_envoy_docker.sh 'BAZEL_BUILD_OPTIONS=--define=boringssl=fips ./ci/do_ci.sh bazel.release.server_only'

[daganibhanu]

@PiotrSikora Yeah, Thank you.

[daganibhanu]

Looks like the base image which we are taking for compilation envoyproxy/envoy-build-ubuntu or envoyproxy/envoy-build doe not have the latest clang which is required for boringssl-fips compilation, hence resulting in the below error.

ERROR: /build/tmp/_bazel_bazel/436badd4919a15958fa3800a4e21074a/external/boringssl_fips/BUILD.bazel:27:1: Executing genrule @boringssl_fips

tar: clang+llvm-6.0.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now.

The build was successful when the option --define boringssl=fips was not passed.

Is my assumption correct? please suggest.

[daganibhanu]

The above issue is solved by setting the proxies inside boringssl_fips.genrule_cmd file. Don't know why proxies passed through command were ignored.

[skyrocknroll]

IMAGE_NAME=envoyproxy/envoy-build-ubuntu ./ci/run_envoy_docker.sh 'BAZEL_BUILD_EXTRA_OPTIONS=--define=boringssl=fips ./ci/do_ci.sh bazel.release.server_only' This one worked for me

[shane-yuan]

I get the following error when building with the latest SHA (without any local change) via IMAGE_NAME=envoyproxy/envoy-build-ubuntu ./ci/run_envoy_docker.sh 'BAZEL_BUILD_EXTRA_OPTIONS=--define=boringssl=fips ./ci/do_ci.sh bazel.release.server_only'. Is it a known issue?

INFO: From CcCmakeMakeRule bazel/foreign_cc/zlib/include:

ERROR: /source/source/extensions/filters/network/kafka/BUILD:137:17: C++ compilation of rule '//source/extensions/filters/network/kafka:kafka_response_parser_lib' failed (Killed): clang failed: error executing command 
  (cd /build/tmp/_bazel_envoybuild/b570b5ccd0454dc9af9f65ab1833764d/sandbox/processwrapper-sandbox/69/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a:-lm \
    BAZEL_LINKOPTS=-lm \
    CC=clang \cc: Cleaning temp directories
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
  /opt/llvm/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-stdlib=libc++' -MD -MF bazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_objs/kafka_response_parser_lib/kafka_response_resolver.d '-frandom-seed=bazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_objs/kafka_response_parser_lib/kafka_response_resolver.o' -gsplit-dwarf -DFMT_HEADER_ONLY -DSPDLOG_FMT_EXTERNAL -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/com_github_fmtlib_fmt -iquote bazel-out/k8-opt/bin/external/com_github_fmtlib_fmt -iquote external/com_github_gabime_spdlog -iquote bazel-out/k8-opt/bin/external/com_github_gabime_spdlog -iquote external/com_github_cyan4973_xxhash -iquote bazel-out/k8-opt/bin/external/com_github_cyan4973_xxhash -Ibazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_virtual_includes/kafka_response_parser_lib -Ibazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_virtual_includes/kafka_response_lib -Ibazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_virtual_includes/serialization_lib -Ibazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_virtual_includes/kafka_types_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/macros -Ibazel-out/k8-opt/bin/include/envoy/common/_virtual_includes/base_includes -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/fmt_lib -Ibazel-out/k8-opt/bin/include/envoy/buffer/_virtual_includes/buffer_interface -Ibazel-out/k8-opt/bin/include/envoy/api/_virtual_includes/os_sys_calls_interface -Ibazel-out/k8-opt/bin/include/envoy/network/_virtual_includes/io_handle_interface -Ibazel-out/k8-opt/bin/include/envoy/network/_virtual_includes/address_interface -Ibazel-out/k8-opt/bin/include/envoy/api/_virtual_includes/io_error_interface -Ibazel-out/k8-opt/bin/include/envoy/event/_virtual_includes/file_event_interface -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/assert_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/minimal_logger_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/base_logger_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/lock_guard_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/thread_annotations -Ibazel-out/k8-opt/bin/include/envoy/thread/_virtual_includes/thread_interface -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/non_copyable -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/logger_impl_lib_standard -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/byte_order_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/utility_lib -Ibazel-out/k8-opt/bin/source/common/common/_virtual_includes/hash_lib -Ibazel-out/k8-opt/bin/include/envoy/common/_virtual_includes/interval_set_interface -Ibazel-out/k8-opt/bin/include/envoy/common/_virtual_includes/time_interface -Ibazel-out/k8-opt/bin/source/common/singleton/_virtual_includes/const_singleton -Ibazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_virtual_includes/tagged_fields_lib -Ibazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_virtual_includes/parser_lib -isystem external/com_github_fmtlib_fmt/include -isystem bazel-out/k8-opt/bin/external/com_github_fmtlib_fmt/include -isystem external/com_github_gabime_spdlog/include -isystem bazel-out/k8-opt/bin/external/com_github_gabime_spdlog/include -fPIC '-std=c++17' -Wall -Wextra -Werror -Wnon-virtual-dtor -Woverloaded-virtual -Wold-style-cast -Wformat -Wformat-security -Wvla -ggdb3 -gsplit-dwarf -fno-limit-debug-info -Wgnu-conditional-omitted-operand -Wc++2a-extensions -DTCMALLOC -DENVOY_HANDLE_SIGNALS -DENVOY_OBJECT_TRACE_ON_DUMP -DENVOY_HOT_RESTART -DENVOY_GOOGLE_GRPC -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c bazel-out/k8-opt/bin/source/extensions/filters/network/kafka/external/kafka_response_resolver.cc -o bazel-out/k8-opt/bin/source/extensions/filters/network/kafka/_objs/kafka_response_parser_lib/kafka_response_resolver.o)
Execution platform: @local_config_platform//:host

[shane-yuan]

It turns out that there are some issues with my MacBook and my colleague is able to build FIPS envoy with the latest SHA on his MacBook. Sorry for the confusion.