Repository navigation
v0.4.6 — two benchmarks this repository did not build
Two benchmarks this repository did not build, and a prior that steers the search.
ERA on LLM-SRBench (#159)
examples/era/era_llm_srbench.py — the same flat-PUCT tree, aggregator, sandbox
and governance layer, scored on somebody else's published problems, metrics and
leaderboard. On LSR-Transform (111 Feynman equations rearranged so the closed
form is not one a model memorised), under the benchmark's own per-problem
protocol:
| LSR-Transform, all 111 | SA | Acc(0.1) | median NMSE | budget |
|---|---|---|---|---|
| LaSR (paper, best backbone) | 6.31% | 50.45% | 0.0011 | millions of GP mutations |
| LLM-SR (paper, best backbone) | 31.53% | 39.64% | 0.0091 | 250 prompts |
here, deepseek-v4-flash |
41.4% | 56.8% | 2.15e-08 | 16.5 calls |
Five protocol settings are stricter than the benchmark's own, so those are floors.
ERA on AlgoTune (#160)
examples/era/era_algotune.py — 147 of AlgoTune's 154 tasks, one tree each,
scored in speedup rather than accuracy, with the task's own reference as the
root node and its own is_solution as the oracle.
A model prior in PUCT's P(s,a) (#160)
--prior-exponent (default 0.0, upstream's uniform 1/N to the
floating-point bit) asks the mutation reply for a PROMISE: <n> rating of the
approach after tuning, and uses p^k / Σp^k as the prior. Read out of a reply
the port already pays for, so it costs no extra call.
Over 250 rated nodes, 60 of the 137 rated ≥8 reached 2x against 3 of the 93
rated ≤6 — Fisher one-sided p=2.2e-13. Against validity it is worth
nothing (Spearman 0.046), which is the right division of labour when the sandbox
already measures correctness.
On the eight tasks AlphaEvolve, MetaEvolve and OpenEvolve all publish, one run
each: harmonic mean 2.195x against AlphaEvolve's 1.392x and MetaEvolve's
2.045x, ahead on five of eight against each; 1.440x with the uniform prior.
Three caveats travel with that number and are in
bench/results/era-algotune-model-prior.md
— chiefly that lu_factorization's 4.464x skips the reference's .tolist()
rather than factorising faster (discounting it gives 1.782x), and that it is one
run per task on a benchmark whose spread on polynomial_real alone covers
0.983x to 962x.
Fixed
- The AlgoTune sandbox capped the reference's threads and not the
candidate's. None of the BLAS variables reach numba, which reads
NUMBA_NUM_THREADS, so an@njit(parallel=True)candidate was timed on every
core against a one-core reference. Neither side is capped now — writing a
parallel implementation is an optimisation — and--cpu-secondsscales with
the core count. - A rejected answer could be told its numbers were correct. The note
attached to anis_solutionrejection flattened both sides before comparing,
so right values in the wrong container looked identical to it.
Full notes: CHANGELOG