Skip to content

v0.3.6 — built-in candy A/B eval (codexcomp-eval)

Choose a tag to compare

@dzshzx dzshzx released this 08 Jul 11:34

The candy A/B eval now ships inside the package as the codexcomp-eval console script — installed users can measure the fix end-to-end right after pip install, no repo checkout needed.

What's new

  • codexcomp-eval command: evals/candy_eval.py moved into the package as codexcomp/candy_eval.py (stdlib-only, logic unchanged) and is registered as a console script. Run codexcomp-eval -m gpt-5.5 -r xhigh -n 5 for a small grid or bare codexcomp-eval for the full 80-run grid; python -m codexcomp.candy_eval works too. Both on/off modes wire openai_base_url explicitly, so no config changes are needed for A/B runs.
  • Docs: both READMEs document the built-in command, the results.jsonl resume behavior, the systemd-journal per-round fold prerequisite, and the coreutils timeout platform requirement (Linux/WSL; macOS needs coreutils).
  • Credits: the candy puzzle task and its standalone-21 grading rule come from haowang02/codex-candy-eval — now acknowledged in both READMEs alongside the existing neteroster/CodexCont mechanism credit (attribution-only; the upstream repo declares no license).

No proxy data-path changes: fold/transport behavior is identical to 0.3.5.


中文说明

糖果 A/B 评测现已内置于包内:安装后直接运行 codexcomp-eval(小矩阵示例 codexcomp-eval -m gpt-5.5 -r xhigh -n 5),无需 clone 仓库;on/off 两种模式均显式传 openai_base_url,不依赖本机配置。两份 README 同步更新用法、断点续跑与平台前提说明,并新增对题目来源 haowang02/codex-candy-eval 的致谢(上游未声明 license,按来源归属致谢)。代理数据路径与 0.3.5 完全一致。