Architecture

Architecture

TuneForge is built around a single abstraction — a Problem (budgeted black box) that every optimizer talks to through one interface — plus response surfaces with analytically known minima and a fidelity model that can be correlated or decorrelated.

Response surfaces

The choice of test surfaces is deliberate: they cover three distinct difficulty regimes.

Branin (smooth, unimodal basin)

Branin has a single global minimum (f* ≈ 0.3979) in a smooth, relatively easy landscape. It is the easiest surface for model-based methods because a Gaussian process or density model can quickly build an accurate response surface representation. GP-EI achieves 16.20x advantage over random here (16 seeds, budget 60).

Ackley (rugged + basin)

Ackley combines a global basin with many local minima (f* = 0 at the origin). It is harder for model-based methods because the rugged landscape requires more evaluations to map, but the global structure is still present. TPE and GP-EI still win (~1.76x and ~2.17x respectively) but by a smaller margin than on Branin.

Null (structureless control)

The null surface is a grid of independent pseudo-random cells — no gradient, no basin, no structure. It is the key control surface:

• The global minimum exists (it is the minimum of the grid), but there is no structure that a model can exploit.
• On null, TPE achieves 0.48x (worse than random) and GP-EI achieves 1.11x (no better than random).
• This proves that model-based advantages on Branin/Ackley come from structure, not from the optimizer machinery itself.

The null surface is guaranteed to stay structureless: it is generated as independent per-cell values with no smoothing.

Fidelity model

Every surface accepts a fidelity parameter b ∈ (0, 1]:

• At b = 1: the observation is exact (no noise, no bias). This is the score that matters for regret.
• At b < 1: a fidelity bias is added. In the correlated regime the bias is small and smooth, so cheap looks rank configurations consistently with expensive ones. In the decorrelated regime the bias is large and rough — cheap looks are essentially coin flips about the true ranking.

The fidelity model is identical across surfaces; only the regime switches. This means the regime can only change the search path, never the final score read at b=1.

Optimizer implementations

All optimizers are implemented from scratch in numpy (tuneforge/optimizers/).

Random (baseline)

Uniform random sampling over the search space. No model, no memory of past evaluations. This is the honest baseline — any optimizer that cannot beat random on its home turf is not doing useful work.

TPE (Tree Parzen Estimator)

TPE models P(x | y < y*) (the density of good configurations) and P(x | y >= y*) (the density of bad ones) separately using Parzen windows (kernel density estimation). It proposes by maximizing l(x) / g(x) — the ratio of good-to-bad density. TuneForge uses adaptive per-point bandwidths (a single global bandwidth makes TPE worse than random — that was a real bug discovered during development). TPE achieves 4.16x advantage over random on Branin.

GP-EI (Gaussian Process + Expected Improvement)

A Gaussian process with an RBF kernel models the response surface. The acquisition function is Expected Improvement:

EI(x) = E[max(f* - f(x), 0)]

The GP is solved via Cholesky decomposition in numpy. GP-EI achieves the strongest sample-efficiency on smooth surfaces (16.20x on Branin) because the RBF kernel accurately models smooth response surfaces.

Successive Halving

A bracket-based early-stopping method. It evaluates a large initial population at low fidelity, eliminates the worst fraction (by observed score), and promotes survivors to the next higher fidelity, repeating until one winner emerges at full fidelity. The savings are real when cheap looks rank consistently with expensive ones (correlated regime), but turn into losses when they don't (decorrelated regime).

Hyperband

Hyperband runs multiple successive-halving brackets with different starting budgets (eta=3, fidelities 1/9 → 1/3 → 1), providing robustness to the choice of starting population size. It achieves 1.32x advantage over random on Branin (correlated), but drops to 0.64x on Branin (decorrelated) — worse than random — because the overhead of bracket management adds cost without the screening benefit.

Simple regret: why it is the right metric

Simple regret is defined as:

R = f(best_so_far) - f(global_opt)

It measures the quality of the single configuration you actually deploy. This is the right metric for HPO because in practice you run the optimizer once and then deploy one set of hyperparameters.

Regret is read only off b=1 (exact) evaluations, so single-fidelity and multi-fidelity methods compete on one honest axis — total budget spent — rather than on evaluation counts that would favor methods using cheap low-fidelity calls.

The complementary metric, regret-vs-budget AUC, measures anytime performance: how well the optimizer tracks toward the optimum as budget accumulates.

Budget dimension and Hyperband's collapse

Hyperband's advantage over random on Branin (correlated regime) is 1.32x in AUC. When the fidelity proxy is decorrelated, that advantage becomes 0.64x — the early eliminations become coin flips, and the overhead of building and querying the brackets causes multi-fidelity methods to spend budget to do worse than random. This is the key insight: the multi-fidelity advantage is entirely dependent on fidelity-rank correlation, not on the bracket structure itself.