Lambda Cold Start Benchmarker

Measures AWS Lambda cold start times across 6 runtimes, 2 architectures, and VPC/non-VPC configurations. Built to verify the claims in Cold Starts Are Dead.

What It Deploys

13 minimal Lambda functions via AWS SAM:

Runtime	arm64	x86_64	Notes
Python 3.13	✅	✅	+ VPC-connected variant
Node.js 22	✅	✅
Go	✅	✅	provided.al2023
Rust	✅	✅	provided.al2023 via cargo-lambda
Java 21	✅	✅
Java 21 + SnapStart	✅	✅	Published versions with alias

All functions are "hello world" handlers with no application dependencies. This isolates the platform's cold start overhead from application initialization time.

Prerequisites

• AWS SAM CLI
• AWS credentials configured
• Go 1.21+
• Maven + JDK 21+
• cargo-lambda (brew install cargo-lambda)
• Python 3.10+ with boto3 and rich (pip install boto3 rich)

Quick Start

# Build all functions locally (no containers needed)
sam build

# Deploy
sam deploy --guided

# Run the benchmark (50 iterations per function, ~2 hours)
cd orchestrator
python benchmark.py --stack-name <your-stack> --region us-west-2

# Quick smoke test (5 iterations, ~15 minutes)
python benchmark.py --stack-name <your-stack> --iterations 5 --region us-west-2

# Test specific runtimes
python benchmark.py --stack-name <your-stack> --iterations 20 \
  --functions python313-arm64,rust-arm64,go-arm64

# Use a named AWS profile
python benchmark.py --stack-name <your-stack> --profile my-profile --region us-west-2

How It Works

Forcing Cold Starts

For standard functions, the orchestrator updates the COLD_START_RUN_ID environment variable to a unique value before each invocation. Changing any function configuration forces Lambda to create a new execution environment, guaranteeing a cold start.

For SnapStart functions, the orchestrator publishes a new function version (triggering a fresh snapshot), updates the live alias, and invokes via the alias. This ensures each invocation goes through the full snapshot restore path.

Collecting Measurements

After each invocation, the orchestrator queries CloudWatch Logs using filter-log-events with the "Init Duration" filter pattern (or "Restore Duration" for SnapStart). It parses the millisecond value from the Lambda REPORT log line.

• Init Duration — time Lambda spends creating the execution environment and running initialization code, before the handler executes.
• Restore Duration — time to restore from a cached SnapStart snapshot.

Output

The orchestrator produces three reports in the terminal:

1. Per-function P50/P99 compared against blog claims (VERIFIED / FASTER / SLOWER)
2. arm64 vs x86_64 comparison with percentage difference
3. VPC vs non-VPC penalty measurement

Raw data is saved as JSON in results/.

Project Structure

├── template.yaml              # SAM template — all 13 Lambda functions + VPC
├── functions/
│   ├── python/                # Python 3.13 handler
│   ├── nodejs/                # Node.js 22 handler
│   ├── go/                    # Go handler (cross-compiled locally)
│   ├── rust/                  # Rust handler (built with cargo-lambda)
│   └── java/                  # Java 21 handler (Maven + shade plugin)
├── orchestrator/
│   ├── benchmark.py           # Main benchmark script
│   └── requirements.txt       # Python dependencies (boto3, rich)
└── results/                   # JSON output from benchmark runs (gitignored)

Build Notes

All functions build locally without containers. SAM uses BuildMethod: makefile for each function:

• Python / Node.js — Makefiles just copy the handler source. No compilation needed.
• Go — Cross-compiles with GOOS=linux GOARCH=arm64|amd64. Requires Go installed.
• Rust — Builds with cargo lambda build --release. Requires cargo-lambda installed.
• Java — Builds a shaded JAR with Maven and extracts classes into the artifacts directory. Requires Maven + JDK 21+.

If you prefer container builds, you can add --use-container to sam build, but Go and Rust will need their toolchains installed inside the container (see the Makefiles for the container-compatible approach).

Cleanup

sam delete --stack-name <your-stack>

This removes all Lambda functions, IAM roles, and the benchmark VPC.

Interpreting Results

Our minimal handlers measure the platform floor — the minimum cold start for each runtime with no application dependencies. The blog's claimed ranges reflect production workloads with real frameworks and libraries, so measured values are expected to be at or below the blog's numbers.

For SnapStart specifically, the blog's 90-140ms claim is for Spring Boot applications where SnapStart eliminates seconds of framework initialization. A minimal handler has almost no init code to snapshot, so the restore mechanism's own overhead (~670ms) dominates. This is expected and doesn't contradict the blog's claim for its stated context.