lambda-benchmarker

Note - Work-in-progress. Expect some bugs

A CLI tool that benchmarks AWS Lambda cold start performance across all compatible Python runtimes and memory tiers, then recommends the optimal configuration for your function.

Installation

pip install -e .

Requires Python 3.11+ and AWS credentials with the following permissions:

• lambda:CreateFunction, lambda:UpdateFunctionConfiguration, lambda:InvokeFunction, lambda:DeleteFunction, lambda:GetFunctionConfiguration
• iam:CreateRole, iam:AttachRolePolicy, iam:DetachRolePolicy, iam:DeleteRole, iam:GetRole (only if not providing --role-arn)

Quick Start

# Benchmark a single Lambda function
benchmarker run myLambda.py

# With a custom minimum improvement threshold
benchmarker run myLambda.py --min-improvement 15

# Compare two versions of a function
benchmarker run compare myLambdav1.py myLambdav2.py

# Compare with a custom threshold
benchmarker run compare myLambdav1.py myLambdav2.py --min-improvement 10

How It Works

1. Static Analysis

The tool parses your .py file before touching AWS to determine which Python runtimes are compatible:

• Import scanning — detects third-party packages (warned, not excluded — you handle layers/packaging)
• Syntax detection — identifies version-specific syntax:
  • match/case → requires Python 3.10+
  • walrus operator (:=) → requires Python 3.8+ (all supported runtimes qualify)
  • except* / exception groups → requires Python 3.11+
• Handler validation — confirms a Lambda handler function exists

Supported runtimes tested: python3.9, python3.10, python3.11, python3.12

2. Deployment

For each compatible runtime, a temporary Lambda function is created with:

• A benchmarker-tmp- prefix in the function name
• A minimal IAM execution role (auto-created if --role-arn is not provided)
• Only the source file packaged (no third-party dependencies — use layers for those)

3. Benchmarking with Adaptive Early Stopping

Memory tiers tested (in order): 128 → 256 → 512 → 1024 → 1769 → 3008 MB

For each memory tier:

1. The function's memory configuration is updated
2. For each of --iterations invocations (default: 10):
   • An environment variable (_BENCH_BUST) is incremented to force a fresh execution environment (genuine cold start)
   • The function is invoked
   • The REPORT log line from CloudWatch Logs (returned inline in LogResult) is parsed for:
     • Init Duration (cold start time)
     • Duration (execution time)
     • Max Memory Used

Early Stopping

After completing each tier (starting from the second), the tool checks whether the improvement in average cold start time justifies continuing:

improvement % = (prev_avg_cold_start - curr_avg_cold_start) / prev_avg_cold_start × 100

If improvement % < --min-improvement (default: 10%), all remaining higher-memory tiers are skipped for that runtime. The table marks these rows as Skipped and shows the reason.

Example: If moving from 128 MB to 256 MB only reduces cold start from 400 ms to 385 ms (3.75% improvement), and --min-improvement is 10, then 512/1024/1769/3008 MB will all be skipped. The curve has flattened — adding more memory is not cost-effective for cold start reduction.

4. Cleanup

All temporary Lambda functions and any IAM role the tool created are deleted after the run, even if the run fails partway through (try/finally). Use --keep to skip cleanup for debugging.

5. Output

┌─────────────────────────────────────────────────────────────────────────┐
│              Results by Runtime & Memory                                │
├────────────┬───────────┬────────────────┬────────────────┬──────────────┤
│ Runtime    │ Memory MB │ Avg Cold Start │ p95 Cold Start │ Avg Duration │
├────────────┼───────────┼────────────────┼────────────────┼──────────────┤
│ python3.9  │       128 │     423.1 ms   │     441.2 ms   │    5.2 ms    │
│            │       256 │     318.4 ms   │     329.0 ms   │    4.1 ms    │
│            │       512 │     289.2 ms   │     295.6 ms   │    3.8 ms    │
│            │      1024 │     skipped    │     skipped    │   skipped    │
│ python3.12 │       128 │     380.0 ms   │     392.1 ms   │    4.9 ms    │
│            │       256 │     271.3 ms   │     280.4 ms   │    3.7 ms    │
└────────────┴───────────┴────────────────┴────────────────┴──────────────┘

╭─────────────────────── Recommendation ───────────────────────╮
│ Best cold start:  python3.12 @ 256 MB → 271.3 ms             │
│ Best value:       python3.12 @ 256 MB (0.385 ms/MB)          │
╰──────────────────────────────────────────────────────────────╯

Best cold start — the single configuration with the lowest average Init Duration.

Best value — the configuration with the highest ms-of-cold-start-improvement per MB of memory added. This is the point where the curve bends: you're getting the most bang for your memory dollar before diminishing returns kick in.

CLI Reference

benchmarker run [OPTIONS] [SOURCE_FILE]
benchmarker run compare [OPTIONS] V1_FILE V2_FILE

Flag	Default	Description
--min-improvement	10.0	Min % cold start improvement required to continue to next memory tier
--region	env/config	AWS region
--role-arn	auto-created	IAM execution role ARN
--iterations	10	Invocations per runtime/memory combination
--keep	False	Leave Lambda functions deployed after the run (debug mode)

Project Structure

lambda-benchmarker/
├── benchmarker/
│   ├── cli.py        # Typer entry point, orchestration
│   ├── analyser.py   # AST-based static analysis and runtime compatibility
│   ├── deployer.py   # boto3: package, deploy, update, delete
│   ├── invoker.py    # Invoke, force cold starts, parse REPORT log lines
│   └── reporter.py   # Rich tables and recommendation logic
├── tests/
│   ├── test_analyser.py
│   ├── test_invoker.py
│   └── fixtures/
│       ├── sample_handler.py
│       ├── match_handler.py
│       └── third_party_handler.py
├── benchmark.yaml.example
└── pyproject.toml

Running Tests

pip install -e ".[dev]"
pytest

Tests cover static analysis (runtime compatibility, import detection, handler validation) and log parsing (REPORT line regex, early stopping logic). AWS calls are not mocked in unit tests — use --keep with a real AWS account for integration testing.

Limitations & Roadmap

• Python runtimes only (Node.js, Java etc. are out of scope)
• No VPC support
• No Lambda Layers support
• Third-party dependencies must be available via layers — the tool only packages the source file
• No persistent result storage (DynamoDB persistence is roadmapped)
• No HTML/web report output