Aws lc fips build fixes by kpramesh2212 · Pull Request #43953 · envoyproxy/envoy

kpramesh2212 · 2026-03-13T17:55:50Z

Use of generative AI

I used generative AI to help fix the AWS-LC FIPS build. I am not a C++ developer, so this PR should be reviewed carefully before merging. I was able to compile Envoy successfully with the changes below, but I may not have fully understood all implications. Please review thoroughly.

Additional Description

This PR fixes several build failures encountered when building Envoy with --config=aws-lc-fips, particularly on aarch64 and in container/CI environments. These fixes address the build infrastructure issues that prevent the AWS-LC FIPS build from completing.

Note for native aarch64 builds: The build also requires BAZEL_USE_HOST_SYSROOT=True (set as an environment variable before building). This is an existing Envoy option documented in bazel/repo.bzl and is not changed in this PR.

Risk Level

Low – changes are limited to build configuration and external dependency setup, not runtime code.

Testing

Production build (successful):

bazel build -c opt //source/exe:envoy-static \
  --config=aws-lc-fips \
  --config=sizeopt \
  --define=no_debug_info=1 \
  --copt=-ffunction-sections \
  --copt=-fdata-sections \
  --linkopt=-Wl,--gc-sections \
  --linkopt=-Wl,-s \
  --strip=always \
  --fission=no \
  --jobs=63

Debug build (successful):

bazel build //source/exe:envoy-static --config=aws-lc-fips --spawn_strategy=standalone

Environment: Native aarch64 Linux build with BAZEL_USE_HOST_SYSROOT=True exported.

Docs Changes

None. Consider documenting BAZEL_USE_HOST_SYSROOT=True for native aarch64 AWS-LC FIPS builds in a follow-up.

Release Notes

build: Fix AWS-LC FIPS build for aarch64 and container environments (Go deps, tar extraction, sandbox)

Platform Specific Features

Linux aarch64 (native build)
Linux x86_64 (expected to work; not tested)

Fixes #43904

BUILD machine

NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.3.20240312"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"

EXPLANATION OF CHANGES (for review comments or PR description)

Go dependency resolution (bazel/dependency_imports.bzl)
- Problem: com_github_spf13_afero and com_github_lyft_protoc_gen_star_v2
  import golang.org/x/text and golang.org/x/tools, but Gazelle didn't resolve
  them, causing "missing strict dependencies" errors.
- Change: Added build_directives to tell Gazelle how to resolve these
  imports to @org_golang_x_text and @org_golang_x_tools. Also added
  org_golang_x_tools go_repository if it wasn't present.
Tar ownership (bazel/external/aws_lc.genrule_cmd)
- Problem: The AWS-LC FIPS build downloads and extracts Clang, Go, Ninja,
  and CMake tarballs. These archives contain ownership metadata (uid 11827,
  gid 9000). In containers or sandboxes, those uid/gid don't exist, so tar
  fails with "Cannot change ownership: Invalid argument".
- Change: Added --no-same-owner to all 5 tar commands so extracted files
  use the current user's uid/gid. Safe for build toolchains and works in
  restricted environments.
AWS-LC genrule sandbox (bazel/external/aws_lc.BUILD)
- Problem: The AWS-LC genrule runs CMake, which writes files during
  configure. Bazel's sandbox makes the execroot read-only for some actions,
  causing "Read-only file system" when CMake runs configure_file.
- Change: Added tags = ["no-sandbox"] to the AWS-LC genrule so it runs
  outside the sandbox. Only this genrule is affected; other build actions
  remain sandboxed.
Host sysroot (environment variable - not a code change)
- Problem: On native aarch64, the hermetic sysroot may not have the
  expected libc layout, causing linker errors like "cannot find
  /lib/aarch64-linux-gnu/libc.so.6".
- Change: Use BAZEL_USE_HOST_SYSROOT=True (existing Envoy option) to use
  the host's libc. Document this for native aarch64 builds.

SUMMARY TABLE

File	Change	Purpose
bazel/dependency_imports.bzl	build_directives	Fix Go "missing strict dependencies"
bazel/external/aws_lc.genrule_cmd	--no-same-owner on tar	Fix tar ownership errors in containers
bazel/external/aws_lc.BUILD	tags = ["no-sandbox"]	Fix CMake "Read-only file system"
Environment	BAZEL_USE_HOST_SYSROOT=True	Use host libc on native aarch64 (document)

================================================================================

repokitteh-read-only · 2026-03-13T17:55:56Z

Hi @kpramesh2212, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #43953 was opened by kpramesh2212.

see: more, trace.

repokitteh-read-only · 2026-03-13T17:56:00Z

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).
envoyproxy/dependency-shepherds assignee is @RyanTheOptimist

🐱

Caused by: #43953 was opened by kpramesh2212.

see: more, trace.

paul-r-gall · 2026-03-17T13:01:15Z

@kpramesh2212 please apply DCO and merge main.

See

envoy/CONTRIBUTING.md

Line 412 in 491b0af

## Fixing DCO

for intstructions on DCO.

phlax

thanks for working on this @kpramesh2212

phlax · 2026-03-17T13:06:04Z

bazel/external/aws_lc.BUILD

    ],
    cmd = "$(location {}) $(location crypto/libcrypto.a) $(location ssl/libssl.a)".format("@envoy//bazel/external:aws_lc.genrule_cmd"),
    tools = ["@envoy//bazel/external:aws_lc.genrule_cmd"],
+    tags = ["no-sandbox"],


i think this is wrong/unnecessary

@phlax I am working on this will update the PR once I fix it.

@phlax I have removed that tag and changed the AWS_LC cmd to get it compiling

yanavlasov

/wait

yanavlasov · 2026-03-18T18:52:07Z

source/common/tls/cert_validator/default_validator.cc

      continue;
    }

+    #ifdef OPENSSL_IS_BORINGSSL


I do not think you need this ifdef, since compiler will have no issue convert non const type to const. But not the other way around. I would just ad a comment that aws+lc needs non const input to X509_NAME_dup

@yanavlasov i have removed that block

yanavlasov · 2026-03-18T18:54:19Z

source/common/tls/context_impl.cc

    switch (policy.value()) {
      using ProtoPolicy = envoy::extensions::transport_sockets::tls::v3::TlsParameters;
    case ProtoPolicy::FIPS_202205:
+    #ifdef OPENSSL_IS_BORINGSSL


Does this change also fix a build error, or is it a runtime error? If it is a runtime error, can you move it into a separate PR to keep unrelated changes separate, please?

@yanavlasov Yes without that if block the build is failing

The preferred way to fix these types of build errors is to define something in aws_lc_compat.h for whatever is missing/different in aws-lc.

@ggreenway I have fixed this

- Upgrade AWS-LC from 1.66.2 to 1.70.0 - Replace runtime tool downloads with Bazel external repos for Clang, Go, Ninja, and CMake (per-arch: x86_64, aarch64, ppc64le) - Fix sandbox: build in writable temp dir with GOCACHE instead of execroot - Add X509_NAME_dup const_cast for AWS-LC/OpenSSL API compatibility - Reject FIPS_202205 policy when not using BoringSSL - Add Gazelle resolve directives for golang.org/x/text and x/tools Signed-off-by: Ramesh KP <ramesh_kp.kurichi_ponnuswamy@genesys.com>

phlax · 2026-03-19T13:25:12Z

bazel/external/aws_lc.BUILD

+    name = "cmake_bin",
+    srcs = [
+        "@aws_lc_cmake_linux_ppc64le//:all",
+        "@aws_lc_cmake_linux_ppc64le//:bootstrap",


this should be agnostic to arch - aws can be used eg on arm also if a fips build is required

also wondering if all this duplication is necessary

if this works for builds other than ppce, i think this repo is just misnamed - its not ppce-specific

iiuc - it should probably be called fips_cmake_src or similar to be consistent with how we have existing fips repos

@phlax I have addressed this comment

kpramesh2212 · 2026-03-19T13:45:18Z

I have build envoy in both x86 and arm machines with AWS_LC and build is completing fine

RyanTheOptimist · 2026-03-19T23:46:49Z

Assigning to @phlax who understands this better and is already reviewing :)

ggreenway

/wait

ggreenway · 2026-03-20T00:19:55Z

source/common/tls/cert_validator/default_validator.cc


-    bssl::UniquePtr<X509_NAME> name_dup(X509_NAME_dup(name));
+    // AWS-LC/OpenSSL require non-const input to X509_NAME_dup; BoringSSL accepts const.
+    bssl::UniquePtr<X509_NAME> name_dup(X509_NAME_dup(const_cast<X509_NAME*>(name)));


Don't const-cast; just remove the the const on line 554

ggreenway · 2026-03-20T00:20:50Z

source/common/tls/context_impl.cc

    switch (policy.value()) {
      using ProtoPolicy = envoy::extensions::transport_sockets::tls::v3::TlsParameters;
    case ProtoPolicy::FIPS_202205:
+    #ifdef OPENSSL_IS_BORINGSSL


The preferred way to fix these types of build errors is to define something in aws_lc_compat.h for whatever is missing/different in aws-lc.

ggreenway · 2026-03-20T00:21:31Z

source/extensions/transport_sockets/tls/cert_validator/spiffe/spiffe_validator.cc

      continue;
    }

+    #ifdef OPENSSL_IS_BORINGSSL


remove const_cast same as above

fixed all of these

kpramesh2212 · 2026-03-23T09:43:11Z

@phlax @ggreenway I have fixed all the review comments could you please take a look into this PR please

kpramesh2212 · 2026-03-25T11:47:31Z

@phlax @ggreenway @yanavlasov

Hi! Just following up on this PR. I've incorporated all the requested changes from the previous review.

Whenever you get a chance, I'd appreciate a re-review. Thanks a lot for your help!

phlax

the main issue for us reviewing this is that its not supported, and therefore not tested

i think we can probably accomodate this but you will need to reduce the duplication and deps to do so

wrt duplication - also wondering if this cant reuse more existing code - but lets address above first and then consider that possibility

phlax · 2026-03-25T17:16:02Z

bazel/external/aws_lc.BUILD

+./bootstrap --parallel=$$(nproc)
+make -j$$(nproc)
+cp bin/cmake $@
+""",


Suggested change

""",

""",

target_compatible_with = select({

"@platforms//cpu:x86_64": ["@platforms//:incompatible"],

"@platforms//cpu:aarch64": ["@platforms//:incompatible"],

"//conditions:default": [],

}),

as these to platforms have already a prebuilt - i think this should be marked as incompatible with them

phlax · 2026-03-25T17:17:03Z

bazel/external/aws_lc.BUILD

+    ] + select({
+        "@platforms//cpu:x86_64": [
+            "@aws_lc_clang_x86_64//:bin/clang",
+            "@aws_lc_clang_x86_64//:bin/clang++",


this is not correct - for x86 it shoudl use the toolchains_llvm platform - same for aarch64 below

phlax · 2026-03-25T17:20:03Z

bazel/external/aws_lc.genrule_cmd

-OS=`uname`
-ARCH=`uname -m`
+OS=$(uname)
+ARCH=$(uname -m)


this is definitely an improvement - i think this file still doesnt pass shellcheck tho (annoyingly its not currently tested by ci)

could you run

$ shellcheck -x bazel/external/aws_lc.genrule_cmd

and at least ensure that no new issues are added

phlax · 2026-03-25T17:21:28Z

bazel/external/aws_lc.genrule_cmd

-    echo "PATH: $PATH"
-    echo "PLATFORM: $PLATFORM"
-    echo "ERROR: CMake version doesn't match. Expected: ${VERSION}, Got: $(cmake --version | head -n1)"
+VERSION=4.2.3


fwiw we have removed these version checks from the boringssl fips build as we have adopted a slightly different policy for managing fips (see bazel/SSL.md)

phlax · 2026-03-25T17:23:06Z

bazel/repositories.bzl

+        build_file_content = GO_BUILD_CONTENT,
+    )
+    external_http_archive(
+        name = "aws_lc_go_aarch64",


are these go binaries that aws_lc produces - if not they dont want the aws_lc prefix - also wondering why you need to add go binaries for x86/arm when we already have them

phlax · 2026-03-25T17:23:48Z

bazel/repository_locations.bzl

        urls = ["https://dl.google.com/go/go{version}.linux-arm64.tar.gz"],
    ),
+
+    aws_lc_clang_x86_64 = dict(


as commented above we dont want these deps - its duplicating the toolchain

phlax · 2026-03-25T17:24:30Z

bazel/repository_locations.bzl

+        urls = ["https://github.com/llvm/llvm-project/releases/download/llvmorg-{version}/clang+llvm-{version}-powerpc64le-linux-ubuntu-18.04.tar.xz"],
+    ),
+
+    aws_lc_go_x86_64 = dict(


same with these deps - they are unnecessary

phlax · 2026-03-25T17:24:56Z

bazel/repository_locations.bzl

+        strip_prefix = "go",
+        urls = ["https://dl.google.com/go/go{version}.linux-arm64.tar.gz"],
+    ),
+    aws_lc_go_ppc64le = dict(


and these next two also should nto be prefixed with aws_lc

kpramesh2212 · 2026-03-26T17:05:38Z

@phlax I have fixed all the comments

kpramesh2212 · 2026-03-30T07:55:25Z

@phlax @ggreenway @yanavlasov I have fixed all the comments could you please take a look into this PR once you get a chance

Medik-GH · 2026-04-01T13:06:55Z

Thanks @kpramesh2212 for these changes! We have been wanting this functionality on our end for a while now. Is there anything I can do to help move this PR along? It seems like FIPS-140-2 is not going to be compliant in September this year so this change is urgently needed.

@phlax @ggreenway @yanavlasov

phlax

lgtm, thanks for iterating @kpramesh2212

will also need signoff from @ggreenway

ggreenway

/wait

bazel/repository_locations.bzl

ggreenway

Source changes LGTM

Signed-off-by: Ramesh KP <ramesh_kp.kurichi_ponnuswamy@genesys.com>

kpramesh2212 · 2026-04-02T09:10:01Z

@phlax @ggreenway

Sorry the CI was failing I had to fix it could you please take on look at this change please https://github.com/envoyproxy/envoy/compare/dfd6fc3288ec93e2ef0dd31465c26dd991f700d0..7fbfa4e4a1d63d96d718033cbc1f29422d244d80

kpramesh2212 requested review from botengyao, mathetake and tyxia as code owners March 13, 2026 17:55

kpramesh2212 had a problem deploying to external-contributors March 13, 2026 17:55 — with GitHub Actions Error

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label Mar 13, 2026

repokitteh-read-only bot assigned RyanTheOptimist Mar 13, 2026

phlax reviewed Mar 17, 2026

View reviewed changes

yanavlasov reviewed Mar 18, 2026

View reviewed changes

repokitteh-read-only bot added the waiting label Mar 18, 2026

Ramesh KP added 2 commits March 19, 2026 12:59

Merge branch 'main' into aws-lc-fips-build-fixes

71f464b

kpramesh2212 force-pushed the aws-lc-fips-build-fixes branch from 022f67a to 71f464b Compare March 19, 2026 13:18

repokitteh-read-only bot removed the waiting label Mar 19, 2026

kpramesh2212 had a problem deploying to external-contributors March 19, 2026 13:19 — with GitHub Actions Error

phlax reviewed Mar 19, 2026

View reviewed changes

RyanTheOptimist assigned phlax and unassigned RyanTheOptimist Mar 19, 2026

ggreenway requested changes Mar 20, 2026

View reviewed changes

repokitteh-read-only bot added waiting and removed waiting labels Mar 20, 2026

kpramesh2212 had a problem deploying to external-contributors March 20, 2026 15:30 — with GitHub Actions Error

phlax reviewed Mar 25, 2026

View reviewed changes

kpramesh2212 force-pushed the aws-lc-fips-build-fixes branch from 04922dd to dfd6fc3 Compare March 26, 2026 17:03

kpramesh2212 temporarily deployed to external-contributors March 26, 2026 17:03 — with GitHub Actions Inactive

phlax previously approved these changes Apr 1, 2026

View reviewed changes

repokitteh-read-only bot removed the deps Approval required for changes to Envoy's external dependencies label Apr 1, 2026

phlax assigned ggreenway Apr 1, 2026

ggreenway requested changes Apr 1, 2026

View reviewed changes

bazel/repository_locations.bzl Show resolved Hide resolved

repokitteh-read-only bot added the waiting label Apr 1, 2026

ggreenway previously approved these changes Apr 1, 2026

View reviewed changes

Address review comments for aws-lc-fips-build-fixes

7fbfa4e

Signed-off-by: Ramesh KP <ramesh_kp.kurichi_ponnuswamy@genesys.com>

kpramesh2212 dismissed stale reviews from ggreenway and phlax via 7fbfa4e April 2, 2026 09:01

kpramesh2212 force-pushed the aws-lc-fips-build-fixes branch from dfd6fc3 to 7fbfa4e Compare April 2, 2026 09:01

kpramesh2212 requested a deployment to external-contributors April 2, 2026 09:01 — with GitHub Actions Waiting

repokitteh-read-only bot added deps Approval required for changes to Envoy's external dependencies and removed waiting labels Apr 2, 2026

kpramesh2212 requested review from ggreenway, phlax and yanavlasov April 2, 2026 09:05

-""",
+""",
+    target_compatible_with = select({
+        "@platforms//cpu:x86_64": ["@platforms//:incompatible"],
+        "@platforms//cpu:aarch64": ["@platforms//:incompatible"],
+        "//conditions:default": [],
+    }),

Conversation

kpramesh2212 commented Mar 13, 2026

Use of generative AI

Additional Description

Risk Level

Testing

Docs Changes

Release Notes

Platform Specific Features

Fixes #43904

BUILD machine

EXPLANATION OF CHANGES (for review comments or PR description)

SUMMARY TABLE

Uh oh!

repokitteh-read-only bot commented Mar 13, 2026

Uh oh!

repokitteh-read-only bot commented Mar 13, 2026

Uh oh!

paul-r-gall commented Mar 17, 2026

Uh oh!

phlax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yanavlasov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kpramesh2212 commented Mar 19, 2026

Uh oh!

RyanTheOptimist commented Mar 19, 2026

Uh oh!

ggreenway left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kpramesh2212 commented Mar 23, 2026

Uh oh!

kpramesh2212 commented Mar 25, 2026

Uh oh!

phlax left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Medik-GH commented Apr 1, 2026 •

edited

Loading