Clarification on B_test usage in Arbor's merge gate and MLE-Bench-Lite comparability

[fskku]

Hi authors,
Thanks for releasing Arbor and the technical report. I have a question about the evaluation protocol, especially the use of held-out test feedback during the optimization loop.
From the paper and the open-source implementation, it seems that Arbor uses B_dev for executor experiments, but GitMergeBranch automatically runs eval_cmd_test on B_test before merging a candidate branch into trunk. The merge decision is therefore conditioned on held-out test performance, and the verified test score is recorded as part of the tree/trunk state.

My question is: should this be considered a form of test-set feedback during optimization, rather than a purely final held-out evaluation?
Even if the agent does not inspect the full test set or manually iterate on individual test examples, the merge gate appears to expose test-set information through either a pass/fail decision or a verified test score. This can influence which branches survive and therefore affect subsequent optimization.
This is especially relevant for the MLE-Bench-Lite result. The paper says Arbor follows the official setting, but if the same merge-gate protocol is used there, then the system seems to use test-set feedback during the run. The official MLE-Bench leaderboard places LoongFlow under “Additional Leaderboard Submissions” with the note “Test-set feedback”; should Arbor’s MLE-Bench-Lite result be similarly annotated, or was a different protocol used for MLE-Bench-Lite where B_test was only evaluated once at the end?

Could you clarify:

1. Was B_test used by GitMergeBranch during the reported MLE-Bench-Lite runs?
2. Should the reported result be categorized as using test-set feedback for leaderboard comparability?

I ask because this distinction affects how the result should be interpreted and compared with methods that only evaluate the held-out test split once at the end.

[ignorejjj]

Thanks for the careful reading. You're right that this counts as test-set feedback, and To keep our setup comparable with other systems evaluated this way, our MLE-Bench-Lite runs do use it: GitMergeBranch runs eval_cmd_test and only keeps a branch if its verified score improves over the current trunk. That said, the feedback is limited to the aggregate metric (the score/medal returned by the grader) used to decide which ideas to retain. The agent never has access to or inspects the individual test labels; the private test data stays hidden from it. So the held-out set influences branch selection only through the scalar metric, not through any per-example label information. We'll annotate the result accordingly for leaderboard comparability.