Home

TuneForge

A from-scratch benchmark of black-box hyperparameter optimizers against known optima.

TuneForge answers not just "which optimizer wins" but why — and proves each answer by an ablation that makes the advantage disappear. Every optimizer and every response surface is implemented in pure numpy; no API keys, no model downloads, no black-box objective you have to trust.

Overview

Two kinds of optimizer are famous for beating random search, for two independent reasons. TuneForge isolates each reason and collapses it with a targeted ablation:

• Model-based search (TPE, GP-EI) wins through response-surface structure. The null surface removes that structure and the advantage collapses to ~1x.
• Multi-fidelity search (successive halving, Hyperband) wins through fidelity-rank correlation. The decorrelated proxy regime breaks that correlation and the advantage collapses below random.

The two ablations are orthogonal: structure removal leaves multi-fidelity untouched, decorrelation leaves model-based sample-efficiency untouched.

Architecture overview

flowchart LR
    subgraph optimizers["Optimizers (numpy, from scratch)"]
        O1[random]
        O2[TPE]
        O3[GP-EI]
        O4[successive halving]
        O5[Hyperband]
    end
    subgraph surface["Response surface (known optimum)"]
        S1[branin: smooth]
        S2[ackley: rugged+basin]
        S3[null: structureless control]
        F[["fidelity proxy: correlated / decorrelated"]]
    end
    P[("Problem: budgeted black box")]
    optimizers -->|"ask x at fidelity b"| P
    P -->|"observed value (exact at b=1)"| optimizers
    P --> surface
    optimizers --> R[OptResult: trials + incumbent]
    R --> M["metrics: simple regret, regret-vs-budget AUC"]
    M --> E[eval harness + quality gate]

Loading

Quick start

pip install -e ".[dev]"      # numpy only; offline

tuneforge surfaces                                   # list surfaces + their optima
tuneforge compare --surface branin --budget 60       # all optimizers, one surface
python -m evals.harness                              # full benchmark -> evals/RESULTS.md
python -m evals.gate                                 # assert the dissociation
pytest -q                                            # 56 tests

Docker:

docker build -t tuneforge . && docker run --rm tuneforge

Live demo

tuneforge.dexdevs.com — run it in your browser, free offline backend.

Wiki pages

• Architecture — surfaces, optimizer implementations, the null surface, simple regret, fidelity model
• Evaluation — benchmark setup, results tables, key findings, reproduce command
• Configuration — env vars, .env.example, optional Optuna cross-check
• Development — setup, testing, how to add a new optimizer or surface