v1.0.0

aws-bench-datasets v1.0.0

aws-bench-datasets v1.0.0 is the first stable release of the dataset: 134 tasks in 8 scenarios. It is released together with aws-bench v1.0.0. The registry in aws-bench v1.0.0 pins every dataset to the content of this release, each path at the commit that last changed it. Run a dataset with aws-bench run -d <dataset>.

Composition

Scenario	Tasks	Task type	Subset
api-and-observability	15	introspection	advanced
compute-and-data	27	mutation	basic
databases-and-storage	30	introspection	basic
ec2-multiregion	9	introspection	quickstart
reference-architectures	11	introspection	advanced
serverless-apps	13	introspection	basic
streaming-and-iot	8	mutation	basic
troubleshooting-multiservice	21	introspection	advanced
Total	134	99 introspection, 35 mutation	quickstart 9, basic 78, advanced 47

• Introspection tasks (99). The agent reads the account and writes a diagnosis. An LLM judge grades the diagnosis against a per-claim rubric (tests/judge.toml). The rubric is derived from the reference answer (tests/ground_truth.json). The agent runs with a mostly read-only role: AWS ReadOnlyAccess plus full access to Athena, Redshift and Bedrock.
• Mutation tasks (35). The agent creates or changes resources. A programmatic verifier (tests/check.py) checks the live state. The environment is reset afterwards. The agent runs with an administrator role.
• Every task ships a reference solution (solution/solve.sh). The framework's oracle agent replays it.

Each subset is published as a dataset: aws-bench-quickstart (9 tasks), aws-bench-basic (78) and aws-bench-advanced (47). The all dataset holds the 134 tasks. You can also run each scenario as its own dataset.

Changes since v0.7.0

Grading

• This release introduces per-claim judge rubrics for all 99 introspection tasks. They replace the single holistic "answers equivalent" criterion of v0.7.0. Each fact of the reference answer is a binary criterion, and a catch-all criterion penalizes extraneous or contradicting facts. Criteria have human-readable names (#68).
• The catch-all criterion does not penalize an extra finding that cites concrete evidence, such as a resource ID, an exact error message or an API response (#56, #60). In four region-scan tasks, it also does not penalize an answer that reports the regions an SCP blocks, which is the method the reference answers used (#52, #56).
• Rubric criteria are fixed. Before the release, regrading saved agent answers found these defects:
  • The CloudFront OAC task credits the missing bucket-policy grant as a blocking cause, in whatever order the answer ranks the causes (#65).
  • The WebSocket backend task credits any wording that establishes that flint-agent-professional.sh itself keeps a persistent connection. A verbatim restatement is not required (#65).
  • The OpenSearch request-status task: the access-log criterion states that the gap is in OpenSearch itself, not in the log types enabled on the domain. It credits an answer that shows that no OpenSearch-native log can supply the per-request status, and that an external component, such as a load balancer, API Gateway, a reverse proxy or the client's own captured response, would be needed (#65).
  • The Rekognition re-upload task no longer requires the DetectLabels parameter values. Its criterion asks whether Rekognition processed the new upload (#65).
  • The ASG secondary network interface task drops the criterion that the attachment fails for two issues (#65).
  • The Cognito account-recovery task: one criterion drops the absolute "never", one no longer chains four assertions, and a criterion about an app client that does not exist is removed (#78).
  • The Network Firewall routing task: the route-table criterion covers only the subnet on the instance's own traffic path, and states that no route leads to a firewall endpoint (#78).
• The Glue deployment-region rubric used the old name of a renamed scenario output, and now uses the new name. A lint check validates the judge placeholders (#74).
• The judge call is retried on transient Bedrock errors (#27). Verifiers call the system-installed rewardkit (#62). check.py uses the task's region scoping (#15).

Reference solutions

• Every task now ships solution/solve.sh. The six introspection scenarios gained their reference solutions in this release (#13, #14, #16, #19, #21, #26). As a result, the oracle agent validates every scenario and verifier end to end.

Tasks

• Ground truths are corrected:
  • The Step Functions job-poller task no longer cites execution history (#28).
  • The Glue table type is explicit (#29).
  • The CloudFormation UPDATE_FAILED ground truth is corrected (#30).
  • The EMR task asks for configuration risks (#31).
  • The SFTP alarm ground truth states the correct alarm state, the EventBridge ECS task claim is scoped to configuration, the unused-security-groups instruction asks for the total and the kind, and the ECR claim is scoped to current state (#65, #66).
• Tasks establish the state they describe on every trial, with task-local hooks:
  • The tasks for the Rekognition re-upload (#33), the Batch job history (#32) and the missing Step Functions state machine (#35) use pre- and post-invoke hooks.
  • The task for the failing Step Functions executions (#34) uses a pre-invoke hook only, because Step Functions has no delete-execution API.
• One intended fault per task:
  • The two ASG health-check tasks now have a single intended fault (#39, #46).
  • The SNS task excludes infrastructure automation in observable terms (#36).
  • The EC2 Image Builder task pins its build region (#37).
  • The Glue reference solution asserts the failed run (#38).
• Ten task.toml files with duplicate AWS_REGION keys are fixed, and CI parses every task.toml (#42). AWS service labels are updated (#17).

Scenarios

• To stop names and outputs from telling the agent that it runs in a benchmark, or where a planted fault is, these are renamed or removed (#58, #69, #72):
  • IAM role names and the role stack are renamed to neutral names. Introspection tasks run as ApplicationReadOnlyRole, and mutation tasks run as ApplicationAdminRole. Both roles come from the AppRoles stack.
  • Several resource names with benchmark terms, such as bench-* and EvalAdmin, are renamed.
  • Several CloudFormation outputs and stack descriptions that revealed a planted fault or an expected answer are removed or renamed.
• The MSK cluster uses Kafka 3.9.x, because AWS no longer creates Standard broker types on Kafka 3.6.0 (#79).
• The S3 Vectors read-only policy is auto-named, so the AppRoles stack deploys next to an older QARoles stack (#76).
• Reset hooks:
  • The reference-architectures reset hook restores the RabbitMQ consumer wiring (#67).
  • The streaming-and-iot hooks receive CLUSTER_NAME and delete orphan Cloud Map namespaces (#64).
• The compute-and-data and streaming-and-iot scenarios use AWS_REGION in their tasks, hooks, solutions and verifiers (#20).

Contributing

• This release adds task contribution guidelines and an intake template for new-task proposals:
  • CONTRIBUTING.md holds the task guidelines on difficulty, diversity and quality, with examples. It describes how to propose, build and submit tasks and scenarios (#49).
  • The new-task proposal issue template asks for the task title and description, the contribution type, the environment details, the proposed evaluation strategy and the task complexity (#45). It links to the task guidelines (#49).

Upgrade notes

• Environments deployed from v0.7.0 have the old role stack. A redeploy creates the AppRoles stack next to the old QARoles stack; you can delete the old stack separately (#76).
• Four introspection tasks now run in mutating concurrency mode, because their hooks create resources: their trials run one at a time, and the account is reset after each trial (#32, #33, #34, #35).

CHANGELOG.md lists the feature, fix and refactor changes.

---

1.0.0 (2026-10-20)

Features

• ec2-multiregion: ec2-multiregion oracle solutions (#13) (630fef3)
• judge: decomposed per-claim rubrics for all 99 introspection tasks (#68) (75ad8f7)
• Merge integration-testing into main (#72) (a45996f)
• oracle: solutions for api-and-observability (#19) (a9dc62e)
• oracle: solutions for databases-and-storage (#21) (269df70)
• oracle: solutions for reference-architectures (#14) (5267efc)
• oracle: solutions for serverless-apps ([#16](https://github.com/aws-bench/aws-bench-datasets/issue...

aws-bench-datasets v0.7.0 — Open Source Launch

aws-bench-datasets v0.7.0 — Open Source Launch 🚀

The first public release of aws-bench-datasets — the scenarios and tasks that power aws-bench, an open-source benchmark for evaluating AI agents on real-world AWS tasks.

This repo holds the benchmark content: containerized AWS environments (CDK stacks) and the tasks agents are scored on. The aws-bench framework provisions the environments, runs the agents, and scores the results.

What's in the benchmark

134 tasks across 8 scenarios, partitioned into three curated datasets:

Dataset	Tasks	Scenarios	Description
aws-bench-quickstart	9	1	Smoke-test to confirm your install and AWS environment are set up correctly.
aws-bench-basic	78	4	Entry-level tasks across core AWS services — the fastest way to baseline an agent.
aws-bench-advanced	47	3	Full-difficulty, multi-service tasks that challenge frontier models.

The 8 scenarios: api-and-observability, compute-and-data, databases-and-storage, ec2-multiregion, reference-architectures, serverless-apps, streaming-and-iot, troubleshooting-multiservice.

Tasks come in two families:

• Introspection (99 tasks) — read-only diagnosis. The agent investigates the environment and writes an answer; an LLM judge scores it against a reference.
• Mutation (35 tasks) — create/modify. The agent changes real AWS resources and a programmatic boto3 verifier checks live state, then rolls the changes back.

Every scenario is also runnable as its own dataset, so you can work through the benchmark one environment at a time.

Getting started

Datasets are run through the framework — install aws-bench and reference a dataset by name:

uv run aws-bench run -d aws-bench-quickstart --env-name my-bench -a claude-code -m <model-id>

Region: these scenarios expect us-east-1. Export AWS_REGION=us-east-1
(and AWS_DEFAULT_REGION=us-east-1) before running.

Datasets are versioned in the framework's registry.json; pin a version with -d <dataset>@<version>.

Authoring your own

The repo is structured so you can add scenarios and tasks in the extended Harbor format (tasks/ and scenarios/). See the Datasets Development Guide for the full contributor workflow.

Companion release

• Framework: aws-bench v0.7.0 —
  aws-bench and aws-bench-datasets are versioned in lockstep; use matching versions.

License

Apache-2.0. See LICENSE
and NOTICE.