If OpenAI calls fail for any reason in the metacoder runner, the code gets caught in a try/fail/repeat loop

[valerie-autumn-skye]

As with Issue #18 (involving the eval stage with the Correctness metric), if the OpenAI calls fail for any reason, the metacoder runner also gets caught in a try/fail/repeat loop until manually aborted (or perhaps some internal retry limit is hit, but that is not apparent from my testing). This likely has a similar solution as in the cited issue.

This is reproducible from the same test case as in parent Issue #19 .

(metacoder) PS C:\Users\CTParker\PycharmProjects\metacoder> uv run metacoder eval .\tests\input\goose_no_server_test.yaml
🔬 Running evaluations from: tests\input\goose_no_server_test.yaml
📊 Loaded dataset: pubmed tools evals
   Models: claude-sonnet
   Coders: goose, dummy (all available)
   Cases: 1
   Total evaluations: 2

🚀 Starting evaluations...
Progress: 1/2 - goose/claude-sonnet/PMID_28027860_Full_Text (no servers)
Running goose with claude-sonnet on case 'PMID_28027860_Full_Text'
📁 Preparing workdir: eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_no_servers\claude-sonnet_goose_PMID_28027860_Full_Text
🔒 Obtaining lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_no_servers\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder
🔧 Writing config object: .config/goose/config.yaml type=yaml
🔓 Releasing lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_no_servers\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder
🔒 Obtaining lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_no_servers\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder
🦆 Running command: goose run -t What is the first sentence of section 2 in PMID: 28027860?
🦆 Command took 27.97 seconds
🔓 Releasing lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_no_servers\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder
Evaluating with CorrectnessMetric
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
OpenAI API Key has insufficient quota: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}
OpenAI quota exhausted; downgrading to Claude...
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"

✓ Tests finished 🎉! Run 'deepeval view' to analyze, debug, and save evaluation results on Confident AI.

Progress: 2/2 - goose/claude-sonnet/PMID_28027860_Full_Text with servers: mcp-simple-pubmed
Running goose with claude-sonnet on case 'PMID_28027860_Full_Text'
📁 Preparing workdir: eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text
🔒 Obtaining lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
🔧 Writing config object: .config/goose/config.yaml type=yaml
🔓 Releasing lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
🔒 Obtaining lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
🦆 Running command: goose run -t What is the first sentence of section 2 in PMID: 28027860?
🦆 Command took 25.19 seconds
🔓 Releasing lock for eval_workdir\claude-sonnet_goose_PMID_28027860_Full_Text_mcp-simple-pubmed\claude-sonnet_goose_PMID_28027860_Full_Text; current_dir=C:\Users\CTParker\PycharmProjects\metacoder      
Evaluating with CorrectnessMetric
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.465641 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.780335 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
OpenAI Error: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}} Retrying: 1 time(s)...
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.445647 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.816545 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
OpenAI Error: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}} Retrying: 2 time(s)...
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.395725 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.944088 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
OpenAI Error: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}} Retrying: 3 time(s)...
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.386913 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
Retrying request to /chat/completions in 0.798842 seconds
HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 429 Too Many Requests"
OpenAI Error: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}} Retrying: 4 time(s)...

Aborted!

[github-actions]

🔄 Issue Summary: OpenAI Retry Loop in Metacoder Runner

This issue reports a critical problem where the metacoder runner gets caught in an infinite retry loop when OpenAI API calls fail, similar to but distinct from the evaluation stage issue reported in #18. The runner continues attempting retries indefinitely when encountering HTTP 429 (rate limiting) or quota exhaustion errors, requiring manual intervention to stop.

🔍 Technical Analysis

Root Cause: The issue occurs in the metacoder runner (not the evaluation stage), where OpenAI API failures trigger an infinite retry mechanism without proper error handling or retry limits.

Key Observations from Logs:

1. First evaluation run: Successfully switches to Claude when OpenAI quota is exhausted
2. Second evaluation run: Gets stuck in retry loop with repeated HTTP/1.1 429 Too Many Requests
3. Retry pattern: Shows escalating retry attempts (1, 2, 3, 4 times) but never terminates
4. Manual intervention required: Process only stops when user manually aborts

🔗 Related Issues Cluster

This is part of a systematic error handling problem in metacoder's evaluation framework:

Issue	Component	Problem	Status
#18	Evaluation Stage (CorrectnessMetric)	Infinite loop on OpenAI HTTP errors	Open
#19	Configuration	Missing server_combinations ignores model settings	Open
#20	Runner Stage	Infinite retry loop on API failures	Current

📊 Impact Assessment

User Experience Impact:

• 💸 Cost: Infinite retries burn through API credits unnecessarily
• ⏱️ Time: Users must manually monitor and abort stuck processes
• 🔧 Reliability: Makes metacoder unreliable for automated/unattended evaluation

Technical Impact:

• The same test case (goose_no_server_test.yaml with PMID:28027860) triggers issues across multiple components
• Demonstrates systemic lack of retry limits and graceful error handling

🛠️ Recommended Solutions

Based on analysis of the related issues and logs:

Immediate Fixes:

1. Add retry limits with exponential backoff for OpenAI API calls in the runner
2. Implement circuit breaker pattern for quota exhaustion scenarios
3. Graceful degradation - fail fast on non-recoverable errors (401, 403, quota exceeded)

Systemic Improvements:

1. Pre-flight API validation - check quotas and connectivity before starting evaluations
2. Unified error handling - consistent retry policies across evaluation and runner components
3. Better monitoring - clear progress indicators and error reporting
4. Configuration validation - ensure proper fallback models are configured (relates to When server_combinations are missing from config.yaml, the models settings are ignored, defaulting to openai and gpt40 #19)

🔧 Technical Implementation Areas

The fix likely requires changes in:

• Runner logic: src/metacoder/evals/runner.py - Add retry limits around API calls
• DeepEval integration: Wrap evaluation calls with proper error detection
• Configuration: Ensure fallback models are properly configured when primary models fail

📝 Test Case Context

The failing test involves:

• Paper: PMID:28027860 (research literature query)
• MCP Server: mcp-simple-pubmed for PubMed integration
• Models: Claude Sonnet 4 configured, but falling back to OpenAI
• Evaluation: CorrectnessMetric for assessing response accuracy

This represents a realistic scientific literature evaluation workflow that should be robustly supported.

🎯 Priority Assessment

High Priority - This significantly impacts the usability of metacoder for:

• Automated evaluation pipelines
• Users with limited API quotas
• Production workflows requiring reliability

The issue affects core functionality and creates a poor user experience with potential for significant cost overruns.

---

Related: Issues #18 (evaluation stage loops), #19 (configuration handling), suggesting need for comprehensive error handling review across the evaluation framework.