error when resuming interrupted multi objective optimization

[dleidreiter]

Description

When running multiobjective optimization using multiple workers, running trials are are added to the runhistory with a stand-in cost of MAXINT. If the optimization is then forcefully terminated (e.g. smac crashed), before the running trials can be finished, they remain like this in the runhistory.

If this runhistory is loaded later on to continue the optimization, smac tries to unpack the single value stand-in cost into the multiple objective values resulting in a crash.

Steps/Code to Reproduce

Run the following code for a few (~10-20) seconds, then forcefully terminate it. Then run it again to try and resume the optimization.

import time
from smac import Scenario, AlgorithmConfigurationFacade
from ConfigSpace import Configuration, Float, ConfigurationSpace


def target(config: Configuration, seed=0):
    time.sleep(2)
    squared = config["x"] ** 2
    cubed = config["x"] ** 3
    return {
        "squared": squared,
        "cubed": cubed,
    }


def main():
    x = Float(name="x", bounds=(0, 100), default=50)

    configspace = ConfigurationSpace()
    configspace.add_hyperparameter(x)

    scenario = Scenario(
        name="test",
        configspace=configspace,
        n_trials=100,
        use_default_config=True,
        objectives=["squared", "cubed"],
        n_workers=5,
    )

    smac = AlgorithmConfigurationFacade(
        scenario=scenario, target_function=target, logging_level=20
    )
    incumbent = smac.optimize()
    print(incumbent)


if __name__ == "__main__":
    main()

Expected Results

During the second optimization run, i would expect smac to be able to continue based on the previous runhistory.

Actual Results

During the second optimization run, smac can't unpack the stand-in cost and crashes with the following traceback:

Traceback (most recent call last):
  File "/home/daniel/minimal_smac_env/bug_1.py", line 38, in <module>
    main()
  File "/home/daniel/minimal_smac_env/bug_1.py", line 30, in main
    smac = AlgorithmConfigurationFacade(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/facade/abstract_facade.py", line 219, in __init__
    self._optimizer = self._get_optimizer()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/facade/abstract_facade.py", line 426, in _get_optimizer
    return SMBO(
           ^^^^^
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 89, in __init__
    self._initialize_state()
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 501, in _initialize_state
    self.load()
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/main/smbo.py", line 396, in load
    self._runhistory.load(runhistory_fn, configspace=self._scenario.configspace)
  File "/home/daniel/miniforge3/envs/minimal_smac_env/lib/python3.11/site-packages/smac/runhistory/runhistory.py", line 859, in load
    cost = [float(x) for x in entry[4]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'float' object is not iterable

Versions

smac: 2.0.2
python: 3.11.6
os: Ubuntu 22.04.3 LTS (on WSL, Windows 10)

[alexandertornede]

Thank you for reporting this, we will have a look into it!