Skip to content

Evaluation

Jump to bottom
Rana Faraz edited this page Jun 23, 2026 · 1 revision

Evaluation

TuneForge benchmarks five optimizers (random, TPE, GP-EI, successive halving, Hyperband) against analytic test surfaces with known global minima. The honest metric is simple regret measured only at full fidelity.

Benchmark setup

  • Seeds: 16 independent seeds per (optimizer, surface, regime) combination
  • Budget: 60 full-fidelity-equivalent evaluations per run
  • Metric: simple regret f(best_so_far) - f(global_opt) at end of budget; regret-vs-budget AUC for anytime performance
  • Surfaces: branin (smooth), ackley (rugged+basin), null (structureless control)
  • Fidelity regimes: correlated (cheap proxy ranks consistently) and decorrelated (ablation — cheap proxy is a coin flip)
  • Requirements: numpy only; no API keys, no downloads; deterministic bit-for-bit across Python 3.10-3.12

Effect 1 — Sample-efficiency (model-based vs random)

Advantage = mean simple regret of random / optimizer. >1 means optimizer beats random.

surface structure random TPE GP-EI TPE adv GP-EI adv
branin smooth 0.8729 0.2096 0.0539 4.16x 16.20x
ackley rugged+basin 2.8661 1.6329 1.3227 1.76x 2.17x
null none (control) 0.0155 0.0320 0.0139 0.48x 1.11x

Key finding: Model-based advantage scales with surface structure. GP-EI is 16x better than random on smooth Branin, 2x on rugged Ackley, and ~1x on the structureless null surface. The advantage comes from the structure, not from the optimizer machinery.

Effect 2 — Budget-efficiency (multi-fidelity vs random)

Advantage = mean regret-vs-budget AUC of random / optimizer (AUC = area under regret curve; lower AUC = better anytime performance).

surface fidelity proxy random succ. halving Hyperband SH adv HB adv
branin rank-correlated 3.247 1.918 2.460 1.69x 1.32x
branin decorrelated (ablation) 3.247 5.611 5.094 0.58x 0.64x
ackley rank-correlated 4.265 3.690 3.761 1.16x 1.13x
ackley decorrelated (ablation) 4.265 4.780 4.425 0.89x 0.96x

Key finding: With a rank-correlated fidelity proxy, multi-fidelity screening beats random per unit budget (SH: 1.69x on Branin). Decorrelate the proxy and early eliminations become coin flips — the methods spend budget on bracket overhead while getting no screening benefit, ending up worse than random (SH: 0.58x).

The dissociation

The two effects are independent:

  • Applying the structure ablation (null surface) leaves multi-fidelity methods untouched — their AUC advantage on the null surface is unchanged by the surface type.
  • Applying the fidelity decorrelation leaves model-based methods untouched — their simple regret advantage is not affected by the fidelity regime.

This independence is what makes the benchmark a clean factorial experiment.

Reproduce

pip install -e ".[dev]"
python -m evals.harness    # writes evals/RESULTS.md with the full table
python -m evals.gate       # asserts the dissociation (19/19 shape checks)

The gate checks the shape of the results (wins where expected, collapses where expected), not exact numbers. This makes it robust to minor numerical variation across platforms while still catching regressions to the thesis.

Single-run CLI output

$ tuneforge compare --surface branin --budget 60 --seed 0
surface branin  optimum=0.397887  budget=60  seed=0  regime=correlated
optimizer            family             regret        auc    spent
------------------------------------------------------------------
random               baseline           0.1342     3.3577     60.0
tpe                  model-based        0.0264     2.7450     60.0
gp_ei                model-based        0.0958     3.6407     60.0
successive_halving   multi-fidelity     0.7831     2.1036     60.0
hyperband            multi-fidelity     0.1416     1.6823     59.7

Clone this wiki locally