ACP Claude: 3 GAIA instances consistently fail with UsagePolicyRefusal

[simonrosenberg]

Summary

When running GAIA benchmark with agent_type=acp-claude, 3 specific instances consistently fail with UsagePolicyRefusal errors. These failures are deterministic and occur across multiple runs.

Affected Instances

Instance ID	Task Type	Analysis
2a649bb1-795f-4a01-b3be-9a01868dae73	Scientific research query	False positive - asking about virus testing chemicals (SPFMV/SPCSV)
983bba7c-c092-455f-b6c9-7857003d48fc	Academic cross-reference	False positive - asking about Hafnia alvei papers
2d83110e-a098-4ebb-9987-066c06fa42d0	Reversed text puzzle	Likely triggered by obfuscation detection

Impact

• Failure rate: 3/500 = 0.6% of GAIA validation set
• Benchmarks affected: GAIA only (Commit0 has 0 failures)
• Models affected: claude-opus-4-6 via ACP

Error Details

UsagePolicyRefusal: Internal error: API Error: Claude Code is unable to respond
to this request, which appears to violate our Usage Policy
(https://www.anthropic.com/legal/aup). Try rephrasing the request or attempting
a different approach. If you are seeing this refusal repeatedly, try running
/model claude-sonnet-4-20250514 to switch models.

Characteristics:

• Immediate refusal (~2 seconds)
• Zero tokens consumed
• Not retriable (3-4 retries all fail)
• Pre-flight rejection before model processes input

Task Details

Instance 1: 2a649bb1-... (Virology Research)

What are the EC numbers of the two most commonly used chemicals for the virus
testing method in the paper about SPFMV and SPCSV in the Pearl Of Africa from
2016? Return the semicolon-separated numbers in the order of the alphabetized
chemicals.

• SPFMV = Sweet Potato Feathery Mottle Virus
• SPCSV = Sweet Potato Chlorotic Stunt Virus
• "Pearl of Africa" = Uganda
• EC numbers = Enzyme Commission classification (scientific identifiers)

Instance 2: 983bba7c-... (Microbiology Cross-Reference)

What animals that were mentioned in both Ilias Lagkouvardos's and Olga Tapia's
papers on the alvei species of the genus named for Copenhagen outside the
bibliographies were also present in the 2021 article cited on the alvei species'
Wikipedia page about a multicenter, randomized, double-blind study?

• "Genus named for Copenhagen" = Hafnia (Latin for Copenhagen)
• Hafnia alvei = bacterium studied in gut microbiome research

Instance 3: 2d83110e-... (Reversed Text Puzzle)

.rewsna eht sa "tfel" drow eht fo etisoppo eht etirw ,ecnetnes siht dnatsrednu uoy fI

Decoded: "If you understand this sentence, write the opposite of the word 'left' as the answer."
Expected answer: "right"

Runs Verified

Run ID	eval_limit	UsagePolicyRefusal Count
22935238175	500	3 (same instances)
22928852361	500	3 (same instances)

Proposed Solutions

Short-term Workarounds

1. Instance exclusion list: Add these 3 instance IDs to a known-failing list for ACP Claude scoring adjustments

2. Model switching fallback: Implement auto-fallback to claude-sonnet-4-20250514 when UsagePolicyRefusal is detected:

   if isinstance(error, UsagePolicyRefusalError):
       # Retry with alternative model
       response = prompt_with_model("claude-sonnet-4-20250514", task)

Long-term

1. Report to Claude Code team: Instances 1 & 2 are clear false positives on legitimate scientific research queries
2. Prompt preprocessing: For reversed text instances, consider decoding before sending to Claude Code

Labels

• bug
• acp
• gaia

[simonrosenberg]

Investigation Results: UsagePolicyRefusal Errors in GAIA ACP Jobs (Last 4 Days)

I searched GCS evaluation artifacts and Datadog logs for all GAIA benchmark runs with ACP (Claude Code) agents over the past 4 days.

Summary

Run ID	Model	Date	UsagePolicyRefusal Errors	Other Errors
22928852361	claude-opus-4-6	2026-03-10	3	0
22935238175	claude-opus-4-6	2026-03-11	3	2 (timeout)
22917148198	claude-opus-4-6	2026-03-09	0	0
22921043629	claude-opus-4-6	2026-03-09	0	0
22922404944	claude-opus-4-6	2026-03-09	0	0
22120360177	claude-sonnet-4-6	2026-02-28	0	165 (infra)
22328833272	claude-sonnet-4-6	2026-03-03	0	8 (runtime)

Affected Instances (UsagePolicyRefusal)

All UsagePolicyRefusal errors occur on the same 3 instances consistently:

Instance ID	Description	Category
2a649bb1-795f-4a01-b3be-9a01868dae73	Virology research query (SPFMV/SPCSV enzyme)	False positive - scientific query
983bba7c-c092-455f-b6c9-7857003d48fc	Hafnia alvei bacterium research papers	False positive - scientific query
2d83110e-a098-4ebb-9987-066c06fa42d0	Reversed text puzzle task	Obfuscation detection trigger

GCS Artifact Locations

Results can be reproduced from:

gs://openhands-evaluation-results/gaia/litellm_proxy-anthropic-claude-opus-4-6/22928852361/results.tar.gz
gs://openhands-evaluation-results/gaia/litellm_proxy-anthropic-claude-opus-4-6/22935238175/results.tar.gz

Key Observations

1. Reproducible: The same 3 instances fail consistently across multiple runs
2. Model-specific: Only affects claude-opus-4-6, not observed in claude-sonnet-4-6 runs (which had different infrastructure errors)
3. Recent regression: Earlier opus-4-6 runs (22917148198, 22921043629, 22922404944) completed without UsagePolicyRefusal errors
4. SDK version change: Errors appeared after SDK update from 524e918 to 0b41c05/3081956

Datadog Query

For additional investigation:

kube_namespace:evaluation-jobs UsagePolicyRefusal

[simonrosenberg]

Update: Exhaustive Search Results

After an exhaustive search of all ACP (claude-opus-4-6 and claude-sonnet-4-6) evaluation runs across all benchmarks in GCS, I can confirm:

UsagePolicyRefusal occurrences found:

Benchmark	Run ID	Occurrences
gaia	22928852361	63
gaia	22935238175	63

Benchmarks searched (0 UsagePolicyRefusal found):

• swebench: 5 runs checked (opus-4-6), 1 run (sonnet-4-6)
• swtbench: 1 run checked (opus-4-6), 1 run (sonnet-4-6)
• commit0: 5 runs checked (opus-4-6), 1 run (sonnet-4-6)
• swebenchmultimodal: 1 run checked (opus-4-6), 1 run (sonnet-4-6)
• gaia (other runs): 3 runs checked (opus-4-6 with SDK 524e918), 2 runs (sonnet-4-6)

Conclusion

UsagePolicyRefusal errors are isolated to GAIA benchmark with the same 3 instances failing consistently. Other benchmarks (swebench, swtbench, commit0, swebenchmultimodal) have zero UsagePolicyRefusal errors in their ACP runs.

The 63 occurrences per run = 3 instances × ~21 log entries each (including retries and error propagation).