Skip to content

Respect the metric direction when building the memory block - #9

Merged
YoungDubbyDu merged 1 commit into
InternScience:mainfrom
GuoCheng24:fix-memory-block-metric-direction
Sep 4, 2026
Merged

Respect the metric direction when building the memory block#9
YoungDubbyDu merged 1 commit into
InternScience:mainfrom
GuoCheng24:fix-memory-block-metric-direction

Conversation

@GuoCheng24

Copy link
Copy Markdown
Contributor

SearchNode.fetch_child_memory sorts siblings with a hard-coded reverse=True and a float('-inf') fallback, and reports the best with max(...). MetricValue carries a maximize flag and the rest of the engine honours it — engine/conditions.py has the direction-aware version of exactly this sort, 40 lines away — but this function does not read it.

That block is what goes into the improve / debug / evolution prompts, so on any minimise metric the model is shown the worst sibling as "Attempt #1" and told the worst score is the best. MLE-bench is heavy on RMSE, MAE and logloss, so this is a large fraction of the benchmark.

Reproduced on main with three RMSE siblings:

minimize (RMSE)   order shown as Attempt #1..#3: ['plan_rmse_0.9', 'plan_rmse_0.5', 'plan_rmse_0.1']
                  Summary: 3 total attempts - 3 executed, 3 successful (best: 0.9000)

0.1 is the best RMSE; the prompt says 0.9 is.

With the patch:

minimize (RMSE)   ['plan_rmse_0.1', 'plan_rmse_0.5', 'plan_rmse_0.9']   best: 0.1000
maximize (AUC)    ['auc_0.92', 'auc_0.85', 'auc_0.70']                  best: 0.9200   (unchanged)

The change. Read the direction from the children's own MetricValue.maximize — no signature change, so the three call sites are untouched — and fall back to maximise when unknown, matching the convention in conditions.py. max becomes direction-aware alongside the sort, so the summary line and the ordering agree.

MetricValue carries a maximize flag and the rest of the engine honours it
(conditions.py, evaluation.py), but fetch_child_memory hardcodes reverse=True
and max(). On a minimize metric (RMSE, MAE, logloss) the model is therefore
shown the worst sibling as Attempt InternScience#1 and told the worst score is the best.
@YoungDubbyDu
YoungDubbyDu merged commit 45b4e32 into InternScience:main Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants