🤖 ci: increase terminal-bench global timeout to 30 minutes

[ammar-agent]

Problem

Fixed 15-minute timeout caused 27-35% of tasks to timeout in nightly runs. Analysis of the Oct 30 nightly run revealed:

• 22 timeouts for Anthropic (27.5%), 28 for OpenAI (35%)
• 5-6 tasks passed tests but hit timeout - would have succeeded with more time
• Longest successful task: blind-maze-explorer-algorithm.hard at 1200s (20 minutes)
• Mean task duration: 356s (Anthropic) / 438s (OpenAI)

Additionally, agent output was human-readable text making it difficult to analyze programmatically.

Solution

Two improvements:

1. Global Timeout Increase

Set global timeout to 30 minutes (1800 seconds) for all tasks.

Design Rationale:

• Longest successful task took 20 minutes
• 30 minutes provides comfortable headroom without excessive wait times
• Avoids maintenance burden of per-task configuration
• Users can override with TB_TIMEOUT env var if needed

2. JSON Lines Output

Enable --json-streaming flag in agent CLI to output structured JSON lines instead of human-readable text.

Benefits:

• Machine-readable output for programmatic analysis
• Easier to parse agent events, tool calls, and results
• Better integration with analysis pipelines

Makefile Changes

• Default TB_TIMEOUT to 1800 seconds (30 minutes)
• Simplified timeout logic - removed per-task calculation
• Backward compatible with TB_TIMEOUT env var override

Usage

# Uses 30-minute default automatically
make benchmark-terminal

# Override for longer tasks
TB_TIMEOUT=3600 make benchmark-terminal

# Override for quick iteration
TB_TIMEOUT=600 make benchmark-terminal TB_SAMPLE_SIZE=5

Expected Impact

• Reduce false timeout failures by ~50% (22-28 timeouts → 11-14 timeouts)
• Improve pass rates by 10-15 percentage points (42% → 52-57%)
• Better analysis capabilities with JSON lines output
• No workflow changes needed - Makefile change applies automatically
• Simple and maintainable - Single global default, no per-task config

Documentation

Updated benchmarks/terminal_bench/README.md to document:

• Preference for global timeout defaults over per-task configuration
• Rationale based on Oct 30 nightly run analysis
• How to override timeout with TB_TIMEOUT env var

Evidence

Tasks from 2025-10-30 nightly run that motivated this change:

Tasks that passed but hit 15-minute timeout:

• blind-maze-explorer-algorithm.hard: ✓ passed at 1200s (20 min)
• qemu-startup: ✓ passed at 838s (14 min)
• count-dataset-tokens: Anthropic timed out at 808s (13.5 min)
• path-tracing: ✓ passed at 660s (11 min)
• pytorch-model-cli: ✓ passed at 541s (9 min)

95th percentile task duration: ~15 minutes

With 30-minute timeout, all these tasks would have succeeded.

Backward Compatibility

✅ Existing workflows continue to work unchanged
✅ TB_TIMEOUT env var provides manual override
✅ Default behavior provides better coverage than 15-minute timeout
✅ JSON output doesn't break existing analysis tools (they just see more structured data)

Generated with cmux