Development

Development

Setup

git clone https://github.com/ranafaraz/TuneForge.git
cd TuneForge
python -m venv .venv
# Windows:
.venv\Scripts\activate
# Linux/macOS:
source .venv/bin/activate

pip install -e ".[dev]"

Running tests and lint

pytest -q                  # 56 tests (+ optuna-skipped without [optuna])
ruff check .               # lint
python -m evals.harness    # full benchmark -> evals/RESULTS.md
python -m evals.gate       # assert the dissociation (19/19 checks)

CI runs all of the above offline with no API keys and no model downloads.

Project structure

TuneForge/
  tuneforge/
    __init__.py          # BLAS thread pin before numpy imports
    types.py             # Trial, OptResult, etc.
    config.py            # Settings (env vars, RNG with salt)
    spaces.py            # SearchSpace, parameter types
    problem.py           # Problem: the single budgeted black box
    surfaces/
      base.py            # BaseSurface + fidelity model
      branin.py          # smooth surface, f* = 0.3979
      ackley.py          # rugged+basin surface, f* = 0
      null.py            # structureless control (independent cells)
    optimizers/
      base.py            # BaseOptimizer interface
      random.py          # uniform random baseline
      tpe.py             # Tree Parzen Estimator (adaptive bandwidths)
      gp_ei.py           # GP + Expected Improvement (Cholesky, RBF)
      successive_halving.py
      hyperband.py       # eta=3 brackets over fidelities 1/9->1/3->1
    runner.py            # run_optimizer(optimizer, problem, budget, seed)
    metrics.py           # simple_regret, auc_regret_vs_budget
    cli.py               # tuneforge CLI (compare, optimize, surfaces)
    optuna_check.py      # optional Optuna backend (importorskip in tests)
  evals/
    metrics.py           # aggregate_results across seeds
    harness.py           # full benchmark -> evals/RESULTS.md
    gate.py              # 19 shape assertions (CI gate)
  tests/
    conftest.py          # BLAS pin for tests
    test_surfaces.py
    test_optimizers.py
    test_metrics.py
    test_runner.py
    test_gate.py
    ...
  examples/
    run_optimizer.py
  docs/
    ARCHITECTURE.md
    DECISIONS.md
    demo.gif
  .env.example
  Dockerfile
  pyproject.toml

How to add a new optimizer

1. Create tuneforge/optimizers/my_optimizer.py implementing BaseOptimizer:

from tuneforge.optimizers.base import BaseOptimizer
from tuneforge.types import Trial, OptResult

class MyOptimizer(BaseOptimizer):
    name = "my_optimizer"
    family = "my-family"

    def ask(self, budget_remaining: float) -> tuple[dict, float]:
        """Return (config_dict, fidelity) to evaluate next."""
        ...

    def tell(self, trial: Trial) -> None:
        """Receive a completed trial."""
        ...

2. Register it in tuneforge/optimizers/__init__.py.

3. Add tests in tests/test_optimizers.py — at minimum, test that it completes a run without error and produces valid regret values.

4. Add it to the comparison in evals/harness.py so it appears in RESULTS.md.

5. If you expect it to beat random on a specific surface, add a gate assertion in evals/gate.py. The gate checks shape (wins where expected, collapses where expected), not exact numbers.

Important: Do not tune the optimizer to clear the gate. Tune the experiment design (surface parameters, budget) if a margin is too thin. Optimizer implementations should be honest textbook implementations.

How to add a new test surface

1. Create tuneforge/surfaces/my_surface.py implementing BaseSurface:

from tuneforge.surfaces.base import BaseSurface
import numpy as np

class MySurface(BaseSurface):
    name = "my_surface"
    bounds = [(-5.0, 5.0), (-5.0, 5.0)]  # search space bounds
    global_opt = 0.0  # known global minimum value

    def _evaluate_exact(self, x: np.ndarray) -> float:
        """Return f(x) exactly (no fidelity bias)."""
        ...

2. The fidelity model in BaseSurface adds bias automatically based on the TUNEFORGE_FIDELITY setting. Your _evaluate_exact must return the exact value — the base class handles b < 1.

3. The global minimum must be known in closed form. A surface without a known optimum cannot be used because simple_regret = f(best) - f(opt) would be undefined.

4. Register in tuneforge/surfaces/__init__.py.

5. Add tests confirming the surface evaluates correctly at the known optimum.

Invariants to preserve

• Regret read only at b=1 — never score off low-fidelity values.
• Fidelity model exact at b=1 and identical across surfaces.
• Null surface stays structureless — do not add smoothing.
• All RNG via Settings.rng(SALT, seed, *offsets) — never hash() or clock-based seeds.
• BLAS pinned to 1 thread — never remove the pin from tuneforge/__init__.py.