v0.3.6 — built-in candy A/B eval (codexcomp-eval)
The candy A/B eval now ships inside the package as the codexcomp-eval console script — installed users can measure the fix end-to-end right after pip install, no repo checkout needed.
What's new
codexcomp-evalcommand:evals/candy_eval.pymoved into the package ascodexcomp/candy_eval.py(stdlib-only, logic unchanged) and is registered as a console script. Runcodexcomp-eval -m gpt-5.5 -r xhigh -n 5for a small grid or barecodexcomp-evalfor the full 80-run grid;python -m codexcomp.candy_evalworks too. Bothon/offmodes wireopenai_base_urlexplicitly, so no config changes are needed for A/B runs.- Docs: both READMEs document the built-in command, the results.jsonl resume behavior, the systemd-journal per-round fold prerequisite, and the coreutils
timeoutplatform requirement (Linux/WSL; macOS needs coreutils). - Credits: the candy puzzle task and its standalone-21 grading rule come from haowang02/codex-candy-eval — now acknowledged in both READMEs alongside the existing neteroster/CodexCont mechanism credit (attribution-only; the upstream repo declares no license).
No proxy data-path changes: fold/transport behavior is identical to 0.3.5.
中文说明
糖果 A/B 评测现已内置于包内:安装后直接运行 codexcomp-eval(小矩阵示例 codexcomp-eval -m gpt-5.5 -r xhigh -n 5),无需 clone 仓库;on/off 两种模式均显式传 openai_base_url,不依赖本机配置。两份 README 同步更新用法、断点续跑与平台前提说明,并新增对题目来源 haowang02/codex-candy-eval 的致谢(上游未声明 license,按来源归属致谢)。代理数据路径与 0.3.5 完全一致。