fix(benchmark): harden Toolathlon job lifecycle and result validation - #1656
Conversation
|
Thanks for the thorough lifecycle hardening and regression coverage. The clean-relay-exit race fix, moving the API key out of argv, and reaping the relay process are all useful improvements. I compared this implementation with the current official Toolathlon client/server code and found a few protocol and lifecycle differences that should be addressed before merge:
With these changes, the PR would preserve the useful hardening while remaining compatible with the official Toolathlon protocol and its retry behavior. |
|
Updated in fee18a0. Transient poll and download errors now retry without cancelling the job. Also added timeout, task_names, average_success_rate, and v1.1 handling. Failed archive downloads remain pending locally so they can be retried if the server no longer returns the task name. |
Yunnglin
left a comment
There was a problem hiding this comment.
LGTM. The lifecycle hardening, protocol compatibility updates, score validation, and generated Toolathlon documentation are consistent and well covered. Thanks for addressing the review feedback.
Hardens the Toolathlon private-mode client:
The scoring and lifecycle changes move the benchmark evaluation version to v1.1.