[tune] remove legacy search algorithms

[matthewdeng]

Why are these changes needed?

This PR removes legacy search algorithms that are not seeing much usage. This list consists of:

• Dragonfly
• (FLAML) BlendSearch, CFO
• Nevergrad
• SigOpt

Related issue number

Follow-up to #41348.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[pcmoritz]

Very nice! Do we also need to regenerate requirements_compiled.txt?

[matthewdeng]

@pcmoritz yep waiting for it to generate here and will update the PR with the update file!

[anhnami]

Nevergrad's OnePlusOne and the recent LogNormalOnePlusOne have been my favorite. They yielded better performance than Optuna for me. Kind of sad because it got removed for no valid reason.

[anhnami]

Just want to add, this does not allow me to load my old tune results.

[327](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:327) fs, fs_path = get_fs_and_path(path_or_uri, storage_filesystem) [328](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:328) with fs.open_input_file(os.path.join(fs_path, _TUNER_PKL)) as f: --> [329](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:329) tuner_state = pickle.loads(f.readall()) [331](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:331) old_trainable_name, flattened_param_space_keys = self._load_tuner_state( [332](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:332) tuner_state [333](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:333) ) [335](https://vscode-remote+ssh-002dremote-002be13k-002eeu-002eorg-002e47.vscode-resource.vscode-cdn.net/home/anhnht/csaiomodel/~/miniforge3/envs/dev/lib/python3.11/site-packages/ray/tune/impl/tuner_internal.py:335) # Perform validation and set the re-specified trainableandparam_space`

ModuleNotFoundError: No module named 'ray.tune.search.nevergrad'`

[JustasB]

@matthewdeng @pcmoritz I second @anhnami 's comment. My team uses Nevergrad extensively, and removal of it from ray tune is a major loss for us.

[teytaud]

I am one of the developers of Nevergrad. In Nevergrad's group we see users mentioning that we are not available anymore in Ray; is there anything we should do in Nevergrad so that it is not a problem for people taking care of Ray ? Happy to do anything that can help.

[teytaud]

Nevergrad's OnePlusOne and the recent LogNormalOnePlusOne have been my favorite. They yielded better performance than Optuna for me. Kind of sad because it got removed for no valid reason.

I have added LogNormalOnePlusOne recently and I love it. As seemingly I am not the only one maybe I should add variants of it in Nevergrad as well.

[matthewdeng]

Apologies all for the disruption, I'll get Nevergrad added back into Ray Tune.

[daveqs]

I presently use FLAML with Ray Tune to run automl jobs. With FLAML's blendsearch and cfo algos no longer available in Ray Tune, is their a recommended alternative for using Tune to perform automl?