Skip to content

v1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@nihal712 nihal712 released this 20 Oct 12:01
v1.0.0
73f814a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

aws-bench-datasets v1.0.0

aws-bench-datasets v1.0.0 is the first stable release of the dataset: 134 tasks in 8 scenarios. It is released together with aws-bench v1.0.0. The registry in aws-bench v1.0.0 pins every dataset to the content of this release, each path at the commit that last changed it. Run a dataset with aws-bench run -d <dataset>.

Composition

Scenario Tasks Task type Subset
api-and-observability 15 introspection advanced
compute-and-data 27 mutation basic
databases-and-storage 30 introspection basic
ec2-multiregion 9 introspection quickstart
reference-architectures 11 introspection advanced
serverless-apps 13 introspection basic
streaming-and-iot 8 mutation basic
troubleshooting-multiservice 21 introspection advanced
Total 134 99 introspection, 35 mutation quickstart 9, basic 78, advanced 47
  • Introspection tasks (99). The agent reads the account and writes a diagnosis. An LLM judge grades the diagnosis against a per-claim rubric (tests/judge.toml). The rubric is derived from the reference answer (tests/ground_truth.json). The agent runs with a mostly read-only role: AWS ReadOnlyAccess plus full access to Athena, Redshift and Bedrock.
  • Mutation tasks (35). The agent creates or changes resources. A programmatic verifier (tests/check.py) checks the live state. The environment is reset afterwards. The agent runs with an administrator role.
  • Every task ships a reference solution (solution/solve.sh). The framework's oracle agent replays it.

Each subset is published as a dataset: aws-bench-quickstart (9 tasks), aws-bench-basic (78) and aws-bench-advanced (47). The all dataset holds the 134 tasks. You can also run each scenario as its own dataset.

Changes since v0.7.0

Grading

  • This release introduces per-claim judge rubrics for all 99 introspection tasks. They replace the single holistic "answers equivalent" criterion of v0.7.0. Each fact of the reference answer is a binary criterion, and a catch-all criterion penalizes extraneous or contradicting facts. Criteria have human-readable names (#68).
  • The catch-all criterion does not penalize an extra finding that cites concrete evidence, such as a resource ID, an exact error message or an API response (#56, #60). In four region-scan tasks, it also does not penalize an answer that reports the regions an SCP blocks, which is the method the reference answers used (#52, #56).
  • Rubric criteria are fixed. Before the release, regrading saved agent answers found these defects:
    • The CloudFront OAC task credits the missing bucket-policy grant as a blocking cause, in whatever order the answer ranks the causes (#65).
    • The WebSocket backend task credits any wording that establishes that flint-agent-professional.sh itself keeps a persistent connection. A verbatim restatement is not required (#65).
    • The OpenSearch request-status task: the access-log criterion states that the gap is in OpenSearch itself, not in the log types enabled on the domain. It credits an answer that shows that no OpenSearch-native log can supply the per-request status, and that an external component, such as a load balancer, API Gateway, a reverse proxy or the client's own captured response, would be needed (#65).
    • The Rekognition re-upload task no longer requires the DetectLabels parameter values. Its criterion asks whether Rekognition processed the new upload (#65).
    • The ASG secondary network interface task drops the criterion that the attachment fails for two issues (#65).
    • The Cognito account-recovery task: one criterion drops the absolute "never", one no longer chains four assertions, and a criterion about an app client that does not exist is removed (#78).
    • The Network Firewall routing task: the route-table criterion covers only the subnet on the instance's own traffic path, and states that no route leads to a firewall endpoint (#78).
  • The Glue deployment-region rubric used the old name of a renamed scenario output, and now uses the new name. A lint check validates the judge placeholders (#74).
  • The judge call is retried on transient Bedrock errors (#27). Verifiers call the system-installed rewardkit (#62). check.py uses the task's region scoping (#15).

Reference solutions

  • Every task now ships solution/solve.sh. The six introspection scenarios gained their reference solutions in this release (#13, #14, #16, #19, #21, #26). As a result, the oracle agent validates every scenario and verifier end to end.

Tasks

  • Ground truths are corrected:
    • The Step Functions job-poller task no longer cites execution history (#28).
    • The Glue table type is explicit (#29).
    • The CloudFormation UPDATE_FAILED ground truth is corrected (#30).
    • The EMR task asks for configuration risks (#31).
    • The SFTP alarm ground truth states the correct alarm state, the EventBridge ECS task claim is scoped to configuration, the unused-security-groups instruction asks for the total and the kind, and the ECR claim is scoped to current state (#65, #66).
  • Tasks establish the state they describe on every trial, with task-local hooks:
    • The tasks for the Rekognition re-upload (#33), the Batch job history (#32) and the missing Step Functions state machine (#35) use pre- and post-invoke hooks.
    • The task for the failing Step Functions executions (#34) uses a pre-invoke hook only, because Step Functions has no delete-execution API.
  • One intended fault per task:
    • The two ASG health-check tasks now have a single intended fault (#39, #46).
    • The SNS task excludes infrastructure automation in observable terms (#36).
    • The EC2 Image Builder task pins its build region (#37).
    • The Glue reference solution asserts the failed run (#38).
  • Ten task.toml files with duplicate AWS_REGION keys are fixed, and CI parses every task.toml (#42). AWS service labels are updated (#17).

Scenarios

  • To stop names and outputs from telling the agent that it runs in a benchmark, or where a planted fault is, these are renamed or removed (#58, #69, #72):
    • IAM role names and the role stack are renamed to neutral names. Introspection tasks run as ApplicationReadOnlyRole, and mutation tasks run as ApplicationAdminRole. Both roles come from the AppRoles stack.
    • Several resource names with benchmark terms, such as bench-* and EvalAdmin, are renamed.
    • Several CloudFormation outputs and stack descriptions that revealed a planted fault or an expected answer are removed or renamed.
  • The MSK cluster uses Kafka 3.9.x, because AWS no longer creates Standard broker types on Kafka 3.6.0 (#79).
  • The S3 Vectors read-only policy is auto-named, so the AppRoles stack deploys next to an older QARoles stack (#76).
  • Reset hooks:
    • The reference-architectures reset hook restores the RabbitMQ consumer wiring (#67).
    • The streaming-and-iot hooks receive CLUSTER_NAME and delete orphan Cloud Map namespaces (#64).
  • The compute-and-data and streaming-and-iot scenarios use AWS_REGION in their tasks, hooks, solutions and verifiers (#20).

Contributing

  • This release adds task contribution guidelines and an intake template for new-task proposals:
    • CONTRIBUTING.md holds the task guidelines on difficulty, diversity and quality, with examples. It describes how to propose, build and submit tasks and scenarios (#49).
    • The new-task proposal issue template asks for the task title and description, the contribution type, the environment details, the proposed evaluation strategy and the task complexity (#45). It links to the task guidelines (#49).

Upgrade notes

  • Environments deployed from v0.7.0 have the old role stack. A redeploy creates the AppRoles stack next to the old QARoles stack; you can delete the old stack separately (#76).
  • Four introspection tasks now run in mutating concurrency mode, because their hooks create resources: their trials run one at a time, and the account is reset after each trial (#32, #33, #34, #35).

CHANGELOG.md lists the feature, fix and refactor changes.

1.0.0 (2026-10-20)

Features

  • ec2-multiregion: ec2-multiregion oracle solutions (#13) (630fef3)
  • judge: decomposed per-claim rubrics for all 99 introspection tasks (#68) (75ad8f7)
  • Merge integration-testing into main (#72) (a45996f)
  • oracle: solutions for api-and-observability (#19) (a9dc62e)
  • oracle: solutions for databases-and-storage (#21) (269df70)
  • oracle: solutions for reference-architectures (#14) (5267efc)
  • oracle: solutions for serverless-apps (#16) (90ce537)
  • oracle: solutions for troubleshooting-multiservice (#26) (2bff693)

Bug Fixes

  • ci: send PR notifications for fork PRs (#83) (dda79c6)
  • judge: repair four over-strict per-claim criteria in two tasks (#78) (c0f0845)
  • judge: replace wrong placeholder value in glue-job-wrong-deployment-region judge + add matching lint checker (#74) (820f80d)
  • reset: wire CLUSTER_NAME into streaming-and-iot reset hooks (#64) (c3f54ab)
  • scenarios: drop explicit S3VectorsReadOnlyAccess policy name in AppRolesStack (#75) (#76) (b1b2225)
  • streaming-and-iot: bump MSK Kafka version 3.6.0 -> 3.9.x (#79) (a001c48)
  • tasks: port ground-truth/config fixes to main (#28-#31) (#40) (aafc9db)
  • tasks: remove duplicate AWS_REGION keys (#42) (80f5dd3)
  • verifier: call system rewardkit instead of uvx (#62) (2accd28)
  • verifier: cap litellm below 1.106 in task images (#82) (968b207)
  • verifier: ensure check.py uses correct region scoping (#15) (42cc575)
  • verifier: retry rewardkit judge call on transient Bedrock errors (#27) (96b101f)

Refactoring

  • migrate compute-and-data + streaming-and-iot to AWS_REGION (#20) (7214635)
Assets 2
Loading