Skip to content


Repository files navigation

Squirrel: A Switching Hyperparameter Optimizer

Motivated by the fact that different optimizers work well on different problems, our approach switches between different optimizers. Since the team names on the competition's leaderboard were randomly generated, consisting of an adjective and an animal with the same initial letter, we called our approach the Switching Squirrel, short, Squirrel.

In our Squirrel framework, we switched between the following components:

  1. An initial design (for known hyperparameter spaces: found by meta-learning; otherwise: selected by differential evolution) (3 batches);
  2. Optimization using Bayesian optimization by integrating the SMAC optimizer with a portfolio of different triplets of surrogate model, acquisition function, and output space transformation (8 batches); and
  3. Optimization using Differential Evolution with parameter adaptation (5 batches)


Our Squirrel ranked 3rd with a score of 92.551 on offical learderboard, and also won 1st place in alternate leaderboard (with a score of 94.845476 and the organizers' bootstrap analysis showing a 100% confidence in this 1st place ranking).

Run Squirrel locally

We used the Bayesmark benchmark framework for the local experiments with Squirrel. See the Bayesmark documentation for the details.

Create and activate virtual environment
> python3 -m venv venv  # Please use Python 3.6.10.
> source venv/bin/activate
Install requirements
> pip install -r environment.txt -r squirrel-optimizer/requirements.txt
Run Squirrel on Bayesmark
>  ./ squirrel-optimizer/ 3
Final score `100 x (1-loss)` for leaderboard:
squirrel-optimizer_0.0.6_6434ac2    102.238945

Team Members

  • Noor Awad
  • Gresa Shala
  • Difan Deng
  • Neeratyoy Mallik
  • Matthias Feurer
  • Katharina Eggensperger
  • Andre' Biedenkapp
  • Diederick Vermetten
  • Hao Wang
  • Carola Doerr
  • Marius Lindauer
  • Frank Hutter


Our implementation is released under Apache License 2.0.


No description, website, or topics provided.







No releases published


No packages published