Grid Search creates more trials than candidates

[martinferianc]

Describe the bug
Given a known search space which only consists of 2 parameters, for which there are only 2 choices, the total number of candidates is 4. However, when running a using grid search, even with n_workers=1, the total number of trials is 5. The first trial configuration seems to be duplicated.

To reproduce
Steps to reproduce the behavior:

Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import syne_tune
>>> print(syne_tune.__version__)
0.3.4

The problem example:

#test.py
import logging

from syne_tune import Reporter
from argparse import ArgumentParser
import datetime
import os 

def get_current_time():
    return datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S-%f")

if __name__ == '__main__':
    root = logging.getLogger()
    root.setLevel(logging.INFO)

    parser = ArgumentParser()
    parser.add_argument('--width', type=int)
    parser.add_argument('--height', type=int)
    parser.add_argument('--st_checkpoint_dir', type=str, default=None)
    
    args = parser.parse_args()
    # Create a folder with the current time and save the arguments as a text into it
    current_time = get_current_time()
    os.mkdir(current_time)
    with open(os.path.join(current_time, "args.txt"), "w") as f:
        f.write(str(args))

    report = Reporter()


    for step in range(100):
        dummy_score = (0.1 + args.width * step / 100) ** (-1) + args.height * 0.1
        report(step=step, mean_loss=dummy_score, epoch=step + 1)

#tune.py
from syne_tune.optimizer.baselines import baselines_dict
from syne_tune.backend import LocalBackend
from syne_tune import Tuner, StoppingCriterion
from syne_tune.config_space import choice


def tune() -> None:
    optimizer = "Grid Search"
    config_space = {
        'width': choice([0, 1]),
        'height': choice([0, 1]),
    }
    scheduler_kwargs = {"config_space": config_space, "metric": 'mean_loss',
                        "max_t": 100,  "mode": "min"}
    scheduler_kwargs["search_options"] = {}
    scheduler_kwargs["search_options"]["shuffle_config"] = False
    scheduler_kwargs["search_options"]["allow_duplicates"] = False
    scheduler = baselines_dict[optimizer](**scheduler_kwargs)

    tuner = Tuner(
        trial_backend=LocalBackend(entry_point="test.py"),
        scheduler=scheduler,
        stop_criterion=StoppingCriterion(
            max_wallclock_time=50, max_num_trials_started=10),
        n_workers=1,
    )

    print(f"Number of trials: {len(scheduler._searcher.hp_values_combinations)}, trials: {scheduler._searcher.hp_values_combinations} keys: {scheduler._searcher.hp_keys}")

    tuner.run()


if __name__ == '__main__':
    tune()

Gives:

python3 tune.py 
Number of trials: 4, trials: [(0, 0), (0, 1), (1, 0), (1, 1)] keys: ['height', 'width']
WARNING:syne_tune.optimizer.schedulers.searchers.random_grid_searcher:All the configurations has already been evaluated. Configuration space has size
Tuning is finishing as the whole configuration space got exhausted.
--------------------
Resource summary (last result is reported):
 trial_id    status  iter  width  height  step  mean_loss  epoch  worker-time
        0 Completed   100      0       0    99  10.000000    100     0.001100
        1 Completed   100      0       0    99  10.000000    100     0.001068
        2 Completed   100      1       0    99   0.917431    100     0.001095
        3 Completed   100      0       1    99  10.100000    100     0.001099
        4 Completed   100      1       1    99   1.017431    100     0.001094
0 trials running, 5 finished (5 until the end), 25.22s wallclock-time

mean_loss: best 0.9174311926605504 for trial-id 2

Expected behavior
The expected behavior is that the number of trials is 4, and that the first trial is not duplicated.

Paste the output of the pip freeze command below:

absl-py==1.3.0
aiohttp==3.8.3
aiosignal==1.3.1
alabaster==0.7.13
asttokens==2.2.1
async-timeout==4.0.2
attrs==22.1.0
autograd==1.5
autopep8==2.0.0
Babel==2.11.0
backcall==0.2.0
black==22.3.0
boto3==1.26.51
botocore==1.29.51
botorch==0.8.1
cachetools==5.2.0
certifi==2022.9.24
charset-normalizer==2.1.1
click==8.1.3
coloredlogs==15.0.1
ConfigSpace==0.6.1
contextlib2==21.6.0
contourpy==1.0.6
cramjam==2.6.2
cycler==0.11.0
debugpy==1.6.4
decorator==5.1.1
dill==0.3.6
distlib==0.3.6
docutils==0.17.1
einops==0.6.0
entrypoints==0.4
et-xmlfile==1.1.0
exceptiongroup==1.1.0
executing==1.2.0
fastparquet==2022.12.0
filelock==3.9.0
fire==0.4.0
flake8==6.0.0
flatbuffers==23.1.4
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
future==0.18.3
google-auth==2.14.1
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
gpytorch==1.9.1
grpcio==1.50.0
h5py==3.7.0
humanfriendly==10.0
idna==3.4
imagesize==1.4.1
importlib-metadata==4.13.0
iniconfig==2.0.0
ipykernel==6.17.1
ipython==8.7.0
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.17.3
jupyter_client==7.4.8
jupyter_core==5.1.0
kiwisolver==1.4.4
latexcodec==2.0.1
lightning-utilities==0.6.0
linear-operator==0.3.0
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe==2.1.1
matplotlib==3.6.2
matplotlib-inline==0.1.6
mccabe==0.7.0
mdit-py-plugins==0.3.3
mdurl==0.1.2
more-itertools==9.0.0
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.2
multipledispatch==0.6.0
multiprocess==0.70.14
mypy-extensions==0.4.3
myst-parser==0.18.1
natsort==8.2.0
nest-asyncio==1.5.6
numpy==1.23.5
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
oauthlib==3.2.2
onnxruntime==1.13.1
openpyxl==3.0.10
opt-einsum==3.3.0
packaging==21.3
pandas==1.5.2
parso==0.8.3
pathos==0.3.0
pathspec==0.10.3
patsy==0.5.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
platformdirs==2.6.0
pluggy==1.0.0
portalocker==2.6.0
pox==0.3.2
ppft==1.7.6.6
prompt-toolkit==3.0.36
protobuf==3.20.1
protobuf3-to-dict==0.1.5
psutil==5.9.4
ptflops==0.6.9
ptyprocess==0.7.0
pure-eval==0.2.2
pyaml==21.10.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybtex==0.24.0
pybtex-docutils==1.0.2
pycodestyle==2.10.0
pyflakes==3.0.1
Pygments==2.13.0
pyparsing==3.0.9
pyro-api==0.1.2
pyro-ppl==1.8.4
pyrsistent==0.19.3
pytest==7.2.1
pytest-timeout==2.1.0
python-dateutil==2.8.2
pytorch-lightning==1.9.0
pytz==2022.6
PyYAML==6.0
pyzmq==24.0.1
ray==2.2.0
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.9
s3fs==0.4.2
s3transfer==0.6.0
sagemaker==2.128.0
schema==0.7.5
SciencePlots==2.0.0
scikit-learn==1.1.3
scikit-optimize==0.9.0
scipy==1.9.3
six==1.16.0
sklearn==0.0.post1
smdebug-rulesconfig==1.0.1
snowballstemmer==2.2.0
Sphinx==5.3.0
sphinx-copybutton==0.5.1
sphinx-rtd-theme==1.1.1
sphinx_autodoc_typehints==1.23.4
sphinxcontrib-bibtex==2.5.0
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib.applehelp==1.0.3
stack-data==0.6.2
statsmodels==0.13.5
sympy==1.11.1
syne-tune==0.3.4
tabulate==0.9.0
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
termcolor==2.1.1
threadpoolctl==3.1.0
tomli==2.0.1
torch==1.13.1
torchdata==0.5.1
torchmetrics==0.10.3
torchtext==0.14.1
torchvision==0.14.1
tornado==6.2
tqdm==4.64.1
traitlets==5.6.0
typing_extensions==4.4.0
ujson==5.7.0
urllib3==1.26.12
virtualenv==20.17.1
wcwidth==0.2.5
Werkzeug==2.2.2
wget==3.2
xgboost==1.7.3
xlrd==2.0.1
yahpo-gym==1.0.1
yarl==1.8.1
-e git+https://github.com/martinferianc/MIMO@a6cc1334409cad1590a41f71fdd42ca4874dc451#egg=yauq
zipp==3.10.0

Desktop (please complete the following information):

• OS: [Ubuntu Linux]
• Syne Tune Version [v0.3.4]

Thank you for your help and amazing work!

[martinferianc]

Please note that I have now checked the most recent version of the code and it does not seem to be happening. I have not changed anything in the configuration but simply cloned the repo and reinstalled the dependency.

[mseeger]

Yes, this was fixed, I'd have been surprised.