Skip to content

Add suite-level reasoning control for provider/model commissions - #101

Merged
danballance merged 1 commit into
mainfrom
codex/add-reasoning-control-for-provider-model-commission
Apr 28, 2026
Merged

Add suite-level reasoning control for provider/model commissions#101
danballance merged 1 commit into
mainfrom
codex/add-reasoning-control-for-provider-model-commission

Conversation

@danballance

Copy link
Copy Markdown
Collaborator

Motivation

  • Benchmarks always used Pi's maximum thinking level (xhigh) for provider models, preventing experiments with low or no reasoning; suites need per-provider control to benchmark different reasoning levels.
  • The change exposes a simple, explicit reasoning configuration on provider/model commissions so reasoning can be varied per-suite without altering providers or model catalog entries.

Description

  • Added a reasoning field to ProviderConfig (defaulting to "xhigh") in park_bench/models/suite.py so provider entries carry a reasoning level.
  • Updated API request handling to use typed provider configs via ProviderConfig in park_bench/api/schemas/requests.py and serialize provider entries with model_dump() in park_bench/api/routes/suites.py so reasoning is persisted in suite config.
  • Threaded the reasoning value through execution: SuiteRunner passes provider_cfg.reasoning into Orchestrator, Orchestrator forwards reasoning into executor calls, and BaseContainerExecutor includes --reasoning in the container entrypoint command.
  • Plumbed reasoning into the container entrypoint surface by adding reasoning to IterationEntrypointArgs and EntrypointArgumentParser, and passed it into PiLLMClient; PiLLMClient now uses the supplied reasoning for Pi's --thinking argument instead of a hardcoded xhigh.
  • Updated tests to assert the new command wiring and defaults across suite API, executor command construction, and orchestrator behavior.

Testing

  • Ran formatting and linting with uv run ruff format park_bench tests and uv run ruff check --fix park_bench tests with no unresolved issues.
  • Ran complexity validation with uv run python .github/scripts/check_complexity.py (no failures).
  • Ran the full test suite with uv run pytest --cov park_bench --cov-report term-missing tests and all tests passed (991 passed, 5 warnings).

Codex Task

@danballance
danballance merged commit 289b902 into main Apr 28, 2026
1 check passed
@danballance
danballance deleted the codex/add-reasoning-control-for-provider-model-commission branch April 28, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant