Accept provider total token usage in review gate#840
Conversation
GitHub Actions
Inactive
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b3bc39895
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
|
|
||
| def test_validator_accepts_provider_total_only_token_usage(tmp_path: Path) -> None: | ||
| """Some providers expose only total tokens; Prime-RL can still render the row.""" |
There was a problem hiding this comment.
The repo-level AGENTS.md convention requires regression tests to name the PR or commit they guard, but this new regression test only describes the provider behavior. Since this is explicitly added as regression coverage for the token-usage gate, please include the guarded PR/commit in the docstring so future maintainers can trace why the case exists.
Useful? React with 👍 / 👎.
1 participant
Summary
benchflow-experiment-reviewso training-readyresults.jsonlrows can satisfy token usage with providertotal_tokenswhen a provider does not expose input/output splitValidation
uv run --extra dev --locked python -m pytest tests/test_benchflow_experiment_review_validator.py -quv run --extra dev --locked ruff check .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py tests/test_benchflow_experiment_review_validator.pyuv run --extra dev --locked ruff format --check .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py tests/test_benchflow_experiment_review_validator.pyContext
The MobileLite Daytona canary on BenchFlow #828 produced Prime-RL-valid
results.jsonlrows andbench train validatepassed, but Azuregpt-5.4-minionly exposed providertotal_tokens. The review gate should accept that provider-total shape instead of requiring a split that the provider did not return.