cold start performance: aws-lc-rs jitter entropy overhead

[PeterUlb]

Summary

The aws-lc-rs crate uses CPU jitter entropy by default, which incurs a one-time-per-process latency cost. On traditional servers this cost is negligible, but on AWS Lambda it has a significant impact on cold start times.
This is particularly unfortunate as it makes Rust appear significantly slower than other runtimes for Lambda, when the real culprit is a single build flag.
Any Lambda using a crate that relies on aws-lc-rs for cryptography is affected, which in practice means virtually all Rust Lambda functions using the AWS SDK.

Measured Impact

I did a micro-benchmark for cold starts of a Lambda calling DDB, averaging multiple iterations (20) to reduce network latency / DDB warmup impact.

Memory	Default (jitter on)	AWS_LC_SYS_NO_JITTER_ENTROPY=1
128MB	1414.5ms	351.8ms
256MB	770.9ms	231.5ms
512MB	449.6ms	189.0ms
1024MB	303.0ms	178.7ms
2048MB	249.2ms	175.4ms

Setting AWS_LC_SYS_NO_JITTER_ENTROPY=1 at build time reduces cold starts by ~4x at 128MB and still ~30% even at 2GB. aws-lc-rs must be an explicitly declared dependency for this to work.

Discussion

I'm not sure if cargo-lambda can do much about this, as everyone using it has to decide if the second source of randomness is a security requirement for them. It seems some environments and frameworks disable it (see linked references).
Maybe documentation is the right answer here, but I'm curious if anyone has ideas for something more proactive (a build info? a template flag?). If it's documentation, where would it best fit?

References

• https://aws.github.io/aws-lc-rs/resources.html#entropy-configuration
• Large performance regression 1.14.0 -> 1.14.1 aws/aws-lc-rs#899

[calavera]

The awslabs/aws-lambda-rust-runtime repository might be a better place to discuss this since it affects the runtime more than cargo lambda.

[calavera]

I would not mind adding this as a default to all builds, but I don't have enough information about it to make an informed decision. The Runtime maintainers have direct access to the AWS-Lc-RS maintainers, so they might be able to provide better guidance.