feat: detect max-ai-credits guardrail exhaustion from firewall audit log#37936
Conversation
…ll log - Update aw-harness.md spec: add max-ai-credits as valid harness.budget key (§6.1.1), require writing budget_exceeded audit entry to firewall log (§11.2.2), update T-AW-003 and §11.4 safeguard #1 - Add parseMaxAICreditsExceededFromAuditEntry/parseMaxAICreditsExceededFromAuditLog to ai_credits_context.cjs; update parseAuditLogCombined and resolveAICreditsFailureState to include maxAICreditsExceeded flag - Add buildMaxAICreditsExceededContext to handle_agent_failure.cjs; wire flag into failure categories, skip-condition, and both comment/issue template contexts - Add max_ai_credits_exceeded.md message template - Add {max_ai_credits_exceeded_context} to agent_failure_issue.md and agent_failure_comment.md - Add ai_credits_context.test.cjs (14 tests) and handle_agent_failure_max_ai_credits_exceeded.test.cjs (4 tests) Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…ror template Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Done in the latest commit. The separate
|
There was a problem hiding this comment.
Pull request overview
This PR adds a dedicated detection path for aw-harness “max-ai-credits” budget exhaustion by reading the firewall audit log (instead of agent stderr), and wires that signal into the conclusion job’s failure handling and messaging.
Changes:
- Extend the harness spec to make
harness.budget.max-ai-creditsnormative and require abudget_exceededfirewall audit log entry on hard-limit abort. - Add firewall audit-log parsing for
max_ai_credits_exceeded/budget_exceededhard-limit signals and surface amaxAICreditsExceededfailure state. - Wire the new failure state into
handle_agent_failureand update the existing AI-credits guardrail template text; add targeted tests.
Show a summary per file
| File | Description |
|---|---|
| specs/aw-harness.md | Updates normative budget keys and specifies firewall-audit-based detection requirements for hard-limit budget aborts. |
| actions/setup/md/ai_credits_rate_limit_error.md | Generalizes the user-facing message to cover both proxy rejection and harness abort, plus adds a configuration tip. |
| actions/setup/js/ai_credits_context.cjs | Adds max_ai_credits_exceeded detection (explicit field or budget_exceeded hard-limit event) and exposes it via resolveAICreditsFailureState(). |
| actions/setup/js/handle_agent_failure.cjs | Threads maxAICreditsExceeded into skip-logic, categorization, and AI-credits display context. |
| actions/setup/js/ai_credits_context.test.cjs | Adds unit tests covering positive/negative detection cases and multi-entry logs. |
| actions/setup/js/handle_agent_failure_max_ai_credits_exceeded.test.cjs | Adds tests verifying the rendered AI-credits guardrail messaging when the new condition is present. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 6/6 changed files
- Comments generated: 2
| - When effective tokens reach the **hard limit** (`max-effective-tokens`), the cost-tracker extension **MUST** abort the session immediately by invoking the session's abort API. The harness **MUST NOT** allow additional turns to proceed after the hard limit is reached. | ||
| - When the budget metric reaches the **soft limit** (default: 80% of the configured limit), the cost-tracker extension **MUST** inject a steering message via `session.steer()` informing the agent that it is approaching the budget and **SHOULD** conclude its work soon. | ||
| - When the budget metric reaches the **hard limit**, the cost-tracker extension **MUST** abort the session immediately by invoking the session's abort API. The harness **MUST NOT** allow additional turns to proceed after the hard limit is reached. | ||
| - Upon hard-limit abort, the harness **MUST** emit a `budget_exceeded` JSONL event to stderr containing the final cumulative token count and the configured limit. |
|
@copilot apply pr finisher skill |
|
🧠 Matt Pocock Skills Reviewer failed during the skills-based review. |
|
|
|
🧪 Test Quality Sentinel completed test quality analysis. |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #37936 does not have the 'implementation' label (has_implementation_label=false) and has 0 new lines of code in business logic directories (default_business_additions=0, threshold=100). Neither Condition A nor Condition B is met. |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Applied the pr-finisher pass and pushed
Also re-ran local validation ( |
🧪 Test Quality Sentinel Report✅ Test Quality Score: 82/100 — Excellent
📊 Metrics & Test Classification (18 tests analyzed)
Test Classification Details
Language SupportTests analyzed:
i️ Test Inflation Notice (non-blocking)Both new test files exceed the 2:1 lines-added ratio vs their corresponding production files:
Context: The Suggestion: No action required. The tests justify their size. If desired, future PRs could add JSONL parse-error tests (e.g., malformed JSON lines) to cover production-code defensive paths not yet exercised by any Verdict
📖 Understanding Test ClassificationsDesign Tests (High Value) verify what the system does:
Implementation Tests (Low Value) verify how the system does it:
Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators. References: §27168015427
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
@copilot lint js and typecheck https://github.com/github/gh-aw/actions/runs/27169324203/job/80204457725?pr=37936 |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Addressed in b599c54. I pulled the failing CJS job log, fixed the JS typecheck error in
I also ran the targeted tests for this area:
|
|
@copilot review all comments and address unresolved review feedback.
|
|
Please summarize the remaining blockers and rerun the requested lint/typecheck checks after the review fix.
|
The conclusion job had no dedicated detection path for when the aw-harness aborts a session due to
max-ai-creditsbudget exhaustion — distinct from the existing AWF proxy rate-limit rejection. Detection must come from the firewall audit log, not agent stderr.Spec (
specs/aw-harness.md)max-ai-creditsas a normativeharness.budgetkey (was only in the non-normative example)budget_exceededaudit entry to/tmp/gh-aw/sandbox/firewall/audit/log.jsonlwithmax_ai_credits_exceeded: trueon hard-limit abort — uniform detection path regardless of which budget key is activeai_credits_context.cjsparseMaxAICreditsExceededFromAuditEntry(): detectsmax_ai_credits_exceeded: truefield (snake_case and camelCase) or abudget_exceededevent withreason: "hard_limit"+forced_termination: true; top-level fields only to avoid false positives from nested provider responsesparseMaxAICreditsExceededFromAuditLog(): single-pass with content-guard fast-pathresolveAICreditsFailureState()now returnsmaxAICreditsExceededhandle_agent_failure.cjsmaxAICreditsExceededwired into: skip-failure guard,buildFailureMatchCategories()("max_ai_credits_exceeded"dedup key), both issue and comment template contextsaiCreditsRateLimitError(proxy rejection) andmaxAICreditsExceeded(harness abort) are OR-ed when building the display context, so both detection paths render through the unifiedai_credits_rate_limit_error.mdtemplateTemplates
ai_credits_rate_limit_error.mdupdated with generic language ("hit the configuredmax-ai-creditsguardrail") covering both proxy rejection and harness abort, and now includes a tip to increase themax-ai-creditslimitTests
14 new tests in
ai_credits_context.test.cjs(field detection, event detection, negative cases, multi-entry log) and 4 inhandle_agent_failure_max_ai_credits_exceeded.test.cjs(metrics table, missing data, no-overage, disabled flag).