Introduce a rich benchmark environment in CI

[HideakiImamura]

This is the prerequisite task for #2964, #2906 and #2951.

Motivations

Recently, a benchmarking environment using kurobako was introduced in Optuna for benchmarking the performance evaluation of sampling and pruning algorithms. The benchmarking feature itself is still experimental and we need to add more problems/features.

Please feel free to leave any comments/feedback for the benchmarking feature.

What should we do

• Support the multi-objective problem. In kurobako, there is some multi-objective problem such that ZDD and NAS-bench (C). We can just use those problems. In addition, visualization of benchmark results is different from single-objective problems. We can show the pareto-front plot and optimization-history plot for each objective coordinate. (We should discuss this point.) Benchmaking for multi-objectives samplers #3271
  • Add mo-kurobako benchmark to CI #3691
• Support problems other than kurobako. We plan to support Bayesmark. Add performance benchmarks using bayesmark #3354. See Make bayesmark benchmark consistent with kurobako benchmark #3464 for the remaining tasks.
• Make NAS-Bench-201 and 301 available in benchmark. See https://arxiv.org/abs/2201.13396 and https://arxiv.org/abs/2008.09777.
  • Add NASBench201 from NASLib #3465
  • [ ] NAS-Bench-301
• Make WFG problems available in benchmark. See https://github.com/HideakiImamura/WFGtestSuite. Add WFG benchmark test #3349

References

See https://github.com/optuna/optuna/tree/master/benchmarks for more details about the current benchmark environment.

[HideakiImamura]

Note that @xadrianzetx is working on the second item in https://github.com/xadrianzetx/dev-optuna-bayesmark.

[HideakiImamura]

Now, NAS-Bench-201 and 301 are proposed, but they are not available in kurobako. It would be great to make it available in kurobako.

• https://arxiv.org/abs/2201.13396
• https://arxiv.org/abs/2008.09777

[HideakiImamura]

Note that @kei-mo is working on the fourth item.

[hvy]

For the record, let me add an additional reference #3155.

[HideakiImamura]

Note that @contramundum53 will work on the third item.

[divyanshugit]

@HideakiImamura If anything is left in this issue then I would love to work on it.

[HideakiImamura]

We have decided not to take a benchmark for multi-objective optimization in Optuna V3. Let me close this issue. I will create another issue to setup the benchmark environment for multi-objective optimization.