Skip to content

Add YAML scenario profiles and recursive evaluation aggregation - #491

Merged
ShuxinLin merged 1 commit into
IBM:mainfrom
ShuxinLin:codex/pr-487-benchmark-evaluation
Jul 29, 2026
Merged

Add YAML scenario profiles and recursive evaluation aggregation#491
ShuxinLin merged 1 commit into
IBM:mainfrom
ShuxinLin:codex/pr-487-benchmark-evaluation

Conversation

@ShuxinLin

@ShuxinLin ShuxinLin commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

This extracts the benchmark and evaluation work from #487 while intentionally excluding every OpenAI-agent-related change.

Benchmark scenario selection

  • replace the category-specific text lists with all.yaml and lite.yaml profiles
  • add named selectors such as fcc_lite, fcc+fmsr_all, lite, and all
  • retain support for custom YAML profiles and plain-text scenario lists
  • update the scenario-suite documentation and unit tests

Evaluation

  • recursively discover trajectory JSON files in nested directories
  • optionally filter scenarios using the benchmark selectors
  • group score summaries by runner and model
  • write a single aggregate report for recursively collected results
  • update evaluation tests for recursive loading, filtering, and aggregation

Scope

The PR changes 22 files, limited to:

  • benchmarks/
  • src/benchmark/
  • src/evaluation/

It excludes OpenAI-agent code, CLI options, launcher scripts, workspace configuration, documentation, examples, and test fixtures.

Testing

uv run pytest src/benchmark/tests/test_scenario_suite_runner.py src/evaluation/tests

Result: 123 passed.

@ShuxinLin ShuxinLin changed the title Add benchmark suite and evaluation updates from #487 Add scenario profiles and recursive evaluation updates from #487 Jul 28, 2026
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin
ShuxinLin force-pushed the codex/pr-487-benchmark-evaluation branch from e79b707 to 7f6e37f Compare July 28, 2026 23:05
@ShuxinLin ShuxinLin changed the title Add scenario profiles and recursive evaluation updates from #487 Add YAML scenario profiles and recursive evaluation aggregation Jul 28, 2026
@ShuxinLin
ShuxinLin requested a review from DhavalRepo18 July 29, 2026 00:23
@ShuxinLin
ShuxinLin merged commit 1e1886f into IBM:main Jul 29, 2026
3 checks passed
@ShuxinLin
ShuxinLin deleted the codex/pr-487-benchmark-evaluation branch July 29, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants