Evaluate AGENTS.md and skills with Codex - #70120
Conversation
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-3-test
jason810496
left a comment
jason810496
left a comment
There was a problem hiding this comment.
Thanks for adding the Codex support!
| installed_version = json.loads(package_json.read_text())["version"] | ||
| except (OSError, KeyError, ValueError): | ||
| continue | ||
| if installed_version == expected_version: |
There was a problem hiding this comment.
Does this means that we force the contributor to use the exact same version we pin?
There was a problem hiding this comment.
Yes, but prek installs the pinned version automatically, so contributors do not select or install it themselves.
I removed the version comparison. The code now selects the SDK installed by the hook instead of promptfoo's bundled copy.
| additional_dependencies: ['promptfoo@0.121.17', '@anthropic-ai/claude-agent-sdk@0.3.185'] | ||
| additional_dependencies: | ||
| - 'promptfoo@0.121.17' | ||
| - '@anthropic-ai/claude-agent-sdk@0.3.185' | ||
| - '@openai/codex-sdk@0.144.6' |
There was a problem hiding this comment.
Or does it means we will use the Claude / Codex install by prek? Would it be possible to relex as to use system wise CLI first then fallback to prek installed?
There was a problem hiding this comment.
It is technically possible, but prek would still install the fallback before the hook runs, so it would not reduce the download. It would also make the eval depend on an unpinned system CLI.
I prefer keeping the CLI managed by the separate manual prek hook for reproducible runs. Authentication still comes from the user's existing session.
| entry: ./dev/skill-evals/eval.py | ||
| language: node | ||
| language_version: '22.22.0' | ||
| additional_dependencies: ['promptfoo@0.121.17', '@anthropic-ai/claude-agent-sdk@0.3.185'] |
There was a problem hiding this comment.
I think it's fine for promptfoo as most of the user won't have it but I think we can assume most of the contributor have either Claude or Codex. Since we will pull ~310–350 MB Codex CLI binary, including Claude-only users now.
There was a problem hiding this comment.
Good point. I split Codex into a separate manual hook. The existing Claude hook is unchanged, so Claude-only users get no additional download from this PR.
The Codex hook installs a newer SDK because promptfoo's bundled Codex binary is signed with a revoked certificate and macOS blocks it. This is also reported in openai/codex #22135.
| return {"id": "anthropic:claude-agent-sdk", "label": label, "config": config} | ||
|
|
||
|
|
||
| def build_codex_provider( |
There was a problem hiding this comment.
Not a real comment but a interesting finding by claude.
Fortunately, our next steps is shirking the AGENTS.md haha.
Codex silently truncates AGENTS.md at 32 KiB, and Airflow's AGENTS.md is already 35,417 bytes, so a Codex-runtime eval can measure nothing while recording proof it ran (CONFIRMED)
Codex's DEFAULT_PROJECT_DOC_MAX_BYTES is 32,768 and codex-rs prefix-truncates project docs silently; build_codex_provider sets cli_config only for history persistence and never raises project_doc_max_bytes, even though promptfoo spreads cli_config straight into Codex config overrides. Since arms differ only in AGENTS.md, any guidance edit landing past the 32,768-byte boundary (the file's last ~2.6 KB, where new guidance is typically appended) produces byte-identical visible guidance in both arms — the exact "eval silently measures nothing" failure mode check_claude_md_symlink guards against on the Claude side, and the hash is still recorded as proof. Fix is one line: add project_doc_max_bytes to cli_config.
There was a problem hiding this comment.
Nice catch, thanks! The Codex provider now raises project_doc_max_bytes so the complete AGENTS.md is included in the evaluation.
| SDK_PACKAGES = { | ||
| "claude": ("@anthropic-ai/claude-agent-sdk", "0.3.185"), | ||
| "codex": ("@openai/codex-sdk", "0.144.6"), | ||
| } |
There was a problem hiding this comment.
Or another direction, we could define the lower bound for the CLI version. (Thought I have no idea how should we pin the version at this moment).
There was a problem hiding this comment.
I removed the comparison instead of adding a lower bound. A lower bound could still select promptfoo's bundled SDK, while additional_dependencies already defines the version used by the hook.
cfbf9d2 to
2efdf52
Compare
2efdf52 to
f1d0ecc
Compare
2 participants
cc: @jason810496
Follow up on #69308 by adding Codex as an alternative runtime for evaluating AGENTS.md and skills through promptfoo's built-in Codex SDK provider.
Run the evaluation with Codex:
The evaluation completed all six cases with no provider errors. Four behavioral assertions passed and two existing ambiguous newsfragment cases failed.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.Important
🛠️ Maintainer triage note for @RoyLee1224 · by
@potiuk· 2026-07-28 16:32 UTCSome review feedback from
jason810496is waiting on you:jason810496.The ball is in your court — you've been assigned to this PR. Push a fix or reply in each thread explaining why the feedback does not apply, then mark them resolved and ping the reviewer (
jason810496) for a final look.See the Pull Request quality criteria for how to fix each item. There is no rush.
Note: your branch is 225 commits behind
main— please rebase and push again to get up-to-date CI results.Automated triage — may be imperfect; a maintainer takes the next look. We use this two-stage triage process so maintainers' limited time goes to the conversation with you.