Performance: Over a third of CPU usage is unaccelerated cryptographic operations on arm64

[asheldon]

The BoringSSL dependency used by Envoy does not use CPU intrinsics for cryptographic acceleration, falling back to unaccelerated mode.

Description

We profiled Envoy on both AWS Graviton2 machines and AWS c5 instances on the same workload at the same time. We found that on Graviton2, we are spending over a third of our time in BoringSSL's nohw methods and virtually none in corresponding methods on our amd64 instances. For example, we spend 0.0091% of our CPU time in CRYPTO_gcm128_encrypt_ctr32 on amd64, but a whopping 12.22% of our CPU time in that method on Graviton2.

On Graviton2, we spend a net of over a third of our CPU time in nohw BoringSSL methods.

We believe the issue is that the fork of BoringSSL that Envoy relies on lacks direct support for the arm64 architecture.

pprof output

Top Methods, amd64

__nss_passwd_lookup
tcmalloc::SLL_TryPop
Envoy::Http::HeaderMapImpl::StaticLookupTable::lookup
operator delete

Top Methods, arm64

gcm_mul64_nohw
aes_nohw_xor
gcm_polyval_nohw
aes_nohw_sub_bytes

Flame Graphs

AWS c5

AWS Graviton2

AWS Graviton2, filtered to nohw

Repro steps:

• Build Envoy for arm64 and run traffic through it on Graviton2
  • In our case, we compiled with -march=armv8.2-a+crypto
• Profile Envoy
• Observe significant CPU spent in unaccelerated cryptographic operations

[jensengrey]

Thanks @asheldon, folks on the Envoy team are currently testing this patch

#16104

[PiotrSikora]

@asheldon if you could build Envoy from #16104 and verify that the performance improved, that would be great!

[asheldon]

Do you have a branch based on Envoy 1.17.2 I can build? We aren't on 1.18 yet, but this patch looks straight-forward enough to backport and verify.

[PiotrSikora]

@asheldon I don't, but you can simply download updated bazel/boringssl_static.patch.

[asheldon]

I think this patch works.

I haven't put this into production, so I can't share a comparable profile or comment on performance improvements yet, but I can confirm that my non-production profiles now have no methods with nohw in them and methods like aes_hw_ctr32_encrypt_blocks now exist.