-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance: Over a third of CPU usage is unaccelerated cryptographic operations on arm64 #16105
Comments
Do you have a branch based on Envoy 1.17.2 I can build? We aren't on 1.18 yet, but this patch looks straight-forward enough to backport and verify. |
@asheldon I don't, but you can simply download updated bazel/boringssl_static.patch. |
I think this patch works. I haven't put this into production, so I can't share a comparable profile or comment on performance improvements yet, but I can confirm that my non-production profiles now have no methods with |
Fixes #16105. Signed-off-by: Piotr Sikora <piotrsikora@google.com>
Fixes envoyproxy#16105. Signed-off-by: Piotr Sikora <piotrsikora@google.com> Signed-off-by: Gokul Nair <gnair@twitter.com>
The BoringSSL dependency used by Envoy does not use CPU intrinsics for cryptographic acceleration, falling back to unaccelerated mode.
Description
We profiled Envoy on both AWS Graviton2 machines and AWS c5 instances on the same workload at the same time. We found that on Graviton2, we are spending over a third of our time in BoringSSL's
nohw
methods and virtually none in corresponding methods on our amd64 instances. For example, we spend 0.0091% of our CPU time inCRYPTO_gcm128_encrypt_ctr32
on amd64, but a whopping 12.22% of our CPU time in that method on Graviton2.On Graviton2, we spend a net of over a third of our CPU time in
nohw
BoringSSL methods.We believe the issue is that the fork of BoringSSL that Envoy relies on lacks direct support for the arm64 architecture.
pprof output
Top Methods, amd64
__nss_passwd_lookup
tcmalloc::SLL_TryPop
Envoy::Http::HeaderMapImpl::StaticLookupTable::lookup
operator delete
Top Methods, arm64
gcm_mul64_nohw
aes_nohw_xor
gcm_polyval_nohw
aes_nohw_sub_bytes
Flame Graphs
AWS c5
AWS Graviton2
AWS Graviton2, filtered to nohw
Repro steps:
-march=armv8.2-a+crypto
The text was updated successfully, but these errors were encountered: