[trajectory-grader] Implement exploitation-error

[github-actions]

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: README.md, .github/workflows/shared/graders/README.md, .github/workflows/shared/graders/exploitation-error.md

Implements exploitation-error — Tier 2, rank 4 in the catalog (the highest-priority Not started grader, walking Tier 1 → Tier 2 → Tier 3 top-to-bottom).

What it measures: For runs that left declared objectives unsatisfied, whether failure was due to misusing evidence already gathered rather than insufficient search. It computes unusedObservations / observations (an observation is "unused" if consumedByActionIds is empty/missing), but only when the run gathered enough evidence (observations >= distinctStatesVisited). When evidence was insufficient it reports not-applicable and defers to exploration-error, so the two graders partition failed-run causes without double counting. Runs with all objectives satisfied score 0.

Distinctness from existing built-ins: tool-success-rate and trajectory-efficiency measure call outcomes/redundancy, not whether gathered evidence was actually acted upon. This grader is the complement of exploration-error (already implemented) — together they attribute objective failure to insufficient search vs unused evidence.

Required IR fields: objectives[].satisfiedAtEventIndex, observations[].consumedByActionIds, events[] (kind state_change, ref) with fallback to states[].

See .github/workflows/shared/graders/README.md for the full catalog — 8 of 25 implemented after this PR.

Generated by 🧮 Daily Trajectory Grader Implementer · copilot · auto · 27.8 AIC · ⌖ 17.1 AIC · ⊞ 10.2K · ◷

• expires on Sep 12, 2026, 10:46 PM UTC-08:00

---

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

Create the pull request manually

# Download the artifact from the workflow run
gh run download '33297443045' -n agent -D '/tmp/agent-33297443045'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-33297443045/aw-trajectory-grader-exploitation-error.bundle'
temp_ref='refs/bundles/create-pr-trajectory-grader-exploitation-error-ccc05f3f11061e65-a01d1d3b'
target_ref='refs/heads/trajectory-grader/exploitation-error-ccc05f3f11061e65'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
  bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
  echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
  exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'trajectory-grader/exploitation-error-ccc05f3f11061e65'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'trajectory-grader/exploitation-error-ccc05f3f11061e65'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement exploitation-error' --base 'main' --head 'trajectory-grader/exploitation-error-ccc05f3f11061e65' --repo 'github/gh-aw'