Repository navigation
aws-bench-datasets v0.7.0 — Open Source Launch
aws-bench-datasets v0.7.0 — Open Source Launch 🚀
The first public release of aws-bench-datasets — the scenarios and tasks that power aws-bench, an open-source benchmark for evaluating AI agents on real-world AWS tasks.
This repo holds the benchmark content: containerized AWS environments (CDK stacks) and the tasks agents are scored on. The aws-bench framework provisions the environments, runs the agents, and scores the results.
What's in the benchmark
134 tasks across 8 scenarios, partitioned into three curated datasets:
| Dataset | Tasks | Scenarios | Description |
|---|---|---|---|
aws-bench-quickstart |
9 | 1 | Smoke-test to confirm your install and AWS environment are set up correctly. |
aws-bench-basic |
78 | 4 | Entry-level tasks across core AWS services — the fastest way to baseline an agent. |
aws-bench-advanced |
47 | 3 | Full-difficulty, multi-service tasks that challenge frontier models. |
The 8 scenarios: api-and-observability, compute-and-data, databases-and-storage, ec2-multiregion, reference-architectures, serverless-apps, streaming-and-iot, troubleshooting-multiservice.
Tasks come in two families:
- Introspection (99 tasks) — read-only diagnosis. The agent investigates the environment and writes an answer; an LLM judge scores it against a reference.
- Mutation (35 tasks) — create/modify. The agent changes real AWS resources and a programmatic boto3 verifier checks live state, then rolls the changes back.
Every scenario is also runnable as its own dataset, so you can work through the benchmark one environment at a time.
Getting started
Datasets are run through the framework — install aws-bench and reference a dataset by name:
uv run aws-bench run -d aws-bench-quickstart --env-name my-bench -a claude-code -m <model-id>Region: these scenarios expect
us-east-1. ExportAWS_REGION=us-east-1
(andAWS_DEFAULT_REGION=us-east-1) before running.
Datasets are versioned in the framework's registry.json; pin a version with -d <dataset>@<version>.
Authoring your own
The repo is structured so you can add scenarios and tasks in the extended Harbor format (tasks/ and scenarios/). See the Datasets Development Guide for the full contributor workflow.
Companion release
- Framework: aws-bench v0.7.0 —
aws-bench and aws-bench-datasets are versioned in lockstep; use matching versions.