(feat)Rich logging

[e-dobrowolska]

This PR brings an optional, richer console logging experience to all the benchmark inference scripts.

What changed

• Added a new shared logging module: benchmarks/utils/console_logging.py
• Conversation output is now much richer and shows tool/message/error events live in the console (via benchmarks/utils/conversation.py)
• Switched how child process logging is set up: now uses the new centralized helper in benchmarks/utils/evaluation.py
• Added end-of-instance summary lines (summarize_instance(...)) in all the main benchmark runner scripts:
  • benchmarks/swebench/run_infer.py
  • benchmarks/swebenchmultimodal/run_infer.py
  • benchmarks/swtbench/run_infer.py
  • benchmarks/multiswebench/run_infer.py
  • benchmarks/commit0/run_infer.py
  • benchmarks/gaia/run_infer.py
  • benchmarks/openagentsafety/run_infer.py
• Added some docs: see benchmarks/utils/logging.md for how it looks and acts

Behavior impact

• By default, everything works as before.
• When enabled, you'll see:
  • a startup line for each instance
  • live updates for tools, messages, errors, etc.
  • a short end-of-instance summary
• File logs are still written as before:
  logs/instance_<id>.log
logs/instance_<id>.output.log

How to enable

• export RICH_LOGGING=1

[juanmichelini]

tested both with and without RICH_LOGGING=1, LGTM

[juanmichelini]

thanks @e-dobrowolska !

[juanmichelini]

@OpenHands please fix the precommit hoooks so we can merge it. do minimal changes require to fix the precommit hooks only

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary of Work

I successfully fixed all pre-commit hook failures in PR #413 "(feat)Rich logging" to make it ready for merge.

Request Verification ✅

• Request: Fix the pre-commit hooks with minimal changes required
• Status: ✅ Fully addressed - all pre-commit hooks now pass

Changes Made

File: benchmarks/utils/console_logging.py

1. Added Any to type imports to support the conversation parameter
2. Removed unused variable n_events (line 467)
3. Added None checks before sys.__stdout__.flush() calls (2 locations at lines 249 and 408) to fix pyright type errors
4. Changed conversation parameter type from object to Any in summarize_instance() function to properly support attribute access

Files: benchmarks/gaia/run_infer.py, benchmarks/swebench/run_infer.py, benchmarks/utils/conversation.py
5. Import ordering corrections (auto-fixed by ruff formatter)

Verification

• ✅ All pre-commit hooks passing (Ruff format, Ruff lint, PEP8, Pyright)
• ✅ Changes are minimal and directly address only the pre-commit failures
• ✅ No extraneous changes made
• ✅ Changes committed and pushed to the fork

The PR is now ready to merge! 🎉

[simonrosenberg]

Does this require its own md file? Or should it be a section of the README.md? @juanmichelini

[juanmichelini]

@OpenHands notice the comment by simon and create a PR that addresses only that.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary of Work

I successfully addressed Simon's feedback on PR #413 by creating a new PR #430 that consolidates the logging documentation.

What Was Done

Created PR #430: "docs: Move rich logging documentation to README.md"

• URL: docs: Move rich logging documentation to README.md #430
• Branch: move-logging-docs-to-readme

Changes Made

✅ Moved rich logging documentation from benchmarks/utils/logging.md into the main README.md as a new section (placed after "Running Benchmarks")

✅ Deleted the separate file benchmarks/utils/logging.md to consolidate all user-facing documentation

Verification Checklist

• ✅ Request completely addressed: Simon asked whether logging docs should be separate or in README - they're now in README
• ✅ Changes are concise: Only 2 files modified (+103 lines in README, -101 lines from deleted file)
• ✅ No functionality changes: Pure documentation reorganization
• ✅ All instructions followed: Created a PR addressing ONLY Simon's comment, no other changes
• ✅ Changes pushed and PR created successfully

The PR is now ready for review by @simonrosenberg and @juanmichelini.