LambdaSharp.Benchmark

This module is used to benchmark .NET AWS Lambda functions with different compilation and deployment options. The measurements are then collected into a CSV file for analysis.

Presentation

Links from Presentation

• When is the Lambda Init Phase Free, and when is it Billed?
• Profiling functions with AWS Lambda Power Tuning
• AWS Lambda Memory Vs CPU configuration
• Benchmark .NET Lambda Projects with LambdaSharp.Benchmark
• Explore the results from the benchmarks in this public Google Sheet (LambdaSharp.Benchmark Explorer)

Benchmarked Projects

1. AwsNewtonsoftJson: using AWS .NET SDK and Newtonsoft JSON.NET (Results)
2. AwsSdk: using AWS .NET SDK (Results)
3. Minimal: minimal baseline project (Results)
4. MinimalTopLevel: minimal baseline project using top-level statements (Results)
5. NewtonsoftJson: using Newtonsoft JSON.NET (Results)
6. SampleAwsNewtonsoftTopLevel: using AWS .NET SDK, Newtonsoft JSON.NET, and Top-Level Statements (Results)
7. SampleAwsSystemTextJsonTopLevel: using AWS .NET SDK, System.Text.Json, and Top-Level Statements (Results)
8. SampleMinimalApi: using ASP.NET Core Minimal API (Results)
9. SourceGeneratorJson: using .NET 6 source generators for JSON parsing (Results)
10. SystemTextJson: using System.Text.Json (Results)

Using LambdaSharp.Benchmark

The measurements were gathered using the LambdaSharp.Benchmark module. Read more about the methodology in the this document.

Module Deployment

LambdaSharp.Benchmark uses the LambdaSharp Tool for deployment. Once LambdaSharp is set up, deploy LambdaSharp.Benchmark directly from its published module. Alternatively, the module can also be deployed from a cloned repository. Deploying the module takes about 3 minutes.

lash deploy LambdaSharp.Benchmark@lambdasharp

Make note of the S3 bucket name shown at the end of the deployment. It is used to upload sample projects to benchmark and holds the measurements at the end.

$ lash deploy LambdaSharp.Benchmark@lambdasharp
LambdaSharp CLI (v0.8.4.0) - Deploy LambdaSharp module
Readying module for deployment tier <DEFAULT>
Resolving module reference: LambdaSharp.Benchmark@lambdasharp
=> Validating module for deployment tier

Deploying stack: LambdaSharp-Benchmark [LambdaSharp.Benchmark:1.0@lambdasharp]
=> Stack create initiated for LambdaSharp-Benchmark [CAPABILITY_IAM]
CREATE_COMPLETE    AWS::CloudFormation::Stack             LambdaSharp-Benchmark (2m 57.71s)
CREATE_COMPLETE    AWS::S3::Bucket                        BuildBucket (22.74s)
CREATE_COMPLETE    LambdaSharp::Registration::Module      Module::Registration (3.21s)
CREATE_COMPLETE    AWS::IAM::Role                         Module::Role (21.36s)
CREATE_COMPLETE    LambdaSharp::S3::EmptyBucket           EmptyBuildBucket (2.57s)
CREATE_COMPLETE    AWS::IAM::Role                         CodeBuild::Role (20.25s)
CREATE_COMPLETE    AWS::CodeBuild::Project                CodeBuild::Project (2.96s)
CREATE_COMPLETE    AWS::IAM::Policy                       Module::Role::DeadLetterQueuePolicy (18.51s)
CREATE_COMPLETE    AWS::Lambda::Function                  CombineMeasurementsFunction (9.69s)
CREATE_COMPLETE    AWS::Lambda::Function                  ListArtifactsFunction (8.77s)
CREATE_COMPLETE    AWS::Lambda::Function                  MeasureFunction (8.49s)
CREATE_COMPLETE    LambdaSharp::Registration::Function    ListArtifactsFunction::Registration (2.9s)
CREATE_COMPLETE    AWS::Logs::LogGroup                    MeasureFunction::LogGroup (2.42s)
CREATE_COMPLETE    LambdaSharp::Registration::Function    MeasureFunction::Registration (1.93s)
CREATE_COMPLETE    LambdaSharp::Registration::Function    CombineMeasurementsFunction::Registration (2.66s)
CREATE_COMPLETE    AWS::Logs::LogGroup                    ListArtifactsFunction::LogGroup (2.49s)
CREATE_COMPLETE    AWS::Logs::LogGroup                    CombineMeasurementsFunction::LogGroup (2.46s)
CREATE_COMPLETE    AWS::IAM::Role                         TestWorkflow::StepFunctionRole (19.57s)
CREATE_COMPLETE    AWS::Logs::SubscriptionFilter          MeasureFunction::LogGroupSubscription (0.7s)
CREATE_COMPLETE    AWS::Logs::SubscriptionFilter          ListArtifactsFunction::LogGroupSubscription (1.02s)
CREATE_COMPLETE    AWS::Logs::SubscriptionFilter          CombineMeasurementsFunction::LogGroupSubscription (0.83s)
CREATE_COMPLETE    AWS::StepFunctions::StateMachine       TestWorkflow::StepFunction (2.79s)
CREATE_COMPLETE    AWS::IAM::Policy                       WatchBucketFunctionPermission (18.58s)
CREATE_COMPLETE    AWS::Lambda::Function                  WatchBucketFunction (8.99s)
CREATE_COMPLETE    AWS::Lambda::Permission                WatchBucketFunction::Source1Permission (10.77s)
CREATE_COMPLETE    LambdaSharp::Registration::Function    WatchBucketFunction::Registration (2.94s)
CREATE_COMPLETE    AWS::Logs::LogGroup                    WatchBucketFunction::LogGroup (3.34s)
CREATE_COMPLETE    AWS::Logs::SubscriptionFilter          WatchBucketFunction::LogGroupSubscription (1.03s)
CREATE_COMPLETE    LambdaSharp::S3::Subscription          WatchBucketFunction::Source1Subscription (2.93s)
=> Stack create finished
Stack output values:
=> BuildBucketName = lambdasharp-benchmark-buildbucket-q9bvmyfoiizu

Benchmarking

Identify a project to measure. Now add a RunSpec.json file to the project folder. This file contains the AWS Lambda entry point name and the JSON payload to use for the Lambda invocation.

{
    "Handler": "SAMPLE-PROJECT::LAMBDA-FUNCTION-CLASS-NAME::METHOD-NAME",
    "Payload": "{\"message\":\"Hello World!\"}"
}

Second zip the contents of the folder, omitting any build files:

zip -9 -r SAMPLE-PROJECT.zip /MY-PROJECTS/SAMPLE-PROJECT -x "**/bin/*" -x "**/obj/*"

Then upload the zip file to the S3 bucket using the AWS CLI:

aws s3 cp "$ZIP_FILE" "s3://S3-BUCKET-NAME/Projects/SAMPLE-PROJECT.zip"

This will automatically kick-off the a step-function to build and benchmark the code. Once completed, the resulting CSV can be find in Reports/ folder on the S3 bucket.

Release History

v1.2 (2022-08-29)

• Changed build name pattern to improve default sorting
• Added AwsNewtonsoftJson project to measure AWS SDK with Newtonsoft JSON.NET library using the class definition notation.
• Modified AwsSdk to initialize S3 client outside of request.
• Updated benchmark data set

v1.1 (2022-07-30)

• Increased cold start sample size to 100
• Increased warm start sample size to 100
• Added measurement of pre-JIT Lambda option
• Added command to delete test Lambdas and their logs in case the measurement failed
• Added command to convert raw JSON files to CSV for more flexible post-processing
• Added SNS topic for notifying that the measurement step function is done

v1.0 (2022-06-10)

• Initial release

License

Copyright (c) 2018-2022 LambdaSharp (λ#)

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.