v3.0.0-rc0

This is the release note of v3.0.0-rc0.
This is a release candidate of Optuna V3. We plan to release the major version within a few weeks. Please try this version and report bugs!

Highlights

Constrained Optimization Support for TPE

TPESampler, the default sampler of Optuna, now supports constrained optimization. It takes a function constraints_func as an argument, and examines whether trials are feasible or not. Feasible trials are prioritized over infeasible ones similarly to NSGAIISampler. See #3506 for more details.

def objective(trial):
    # Binh and Korn function with constraints.
    x = trial.suggest_float("x", -15, 30)
    y = trial.suggest_float("y", -15, 30)

    # Store the constraints as user attributes so that they can be restored after optimization.
    c0 = (x - 5) ** 2 + y ** 2 - 25
    c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
    trial.set_user_attr("constraints", (c0, c1))

    v0 = 4 * x ** 2 + 4 * y ** 2
    v1 = (x - 5) ** 2 + (y - 5) ** 2

    return v0, v1

def constraints(trial):
    return trial.user_attrs["constraints"]

if __name__ == "__main__":
    sampler = optuna.samplers.TPESampler(
        constraints_func=constraints,
    )
    study = optuna.create_study(
        directions=["minimize", "minimize"],
        sampler=sampler,
    )
    study.optimize(objective, n_trials=1000)

    optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()

MOTPE without constraints	MOTPE with constraints

A Major Refactoring of Visualization Module

We have undertaken major refactoring of the visualization features as one of the major tasks of Optuna V3. The current situation is as follows.

Unification of implementations of different backends: plotly and matplotlib

Historically, the implementations of Optuna's visualization features were split between two different backends, plotly and matplotlib. Many of these implementations were duplicated and unmaintainable, and many were implemented as a single large function, resulting in poor testability and, as a result, becoming the cause of many bugs. We clarified the specifications that each visualization function in Optuna must meet and defined the backend-independent information needed to perform the visualization. By using this information commonly across different backends, we achieved a highly maintainable and testable implementation, and improved the stability of the visualization functions dramatically. We are currently rewriting the unit tests, and the resulting tests will be simple yet powerful.

Visual Regression Test

It is very important to detect hidden bugs in the implementation through PR reviews. However, visualizations are likely to contain bugs that are difficult to find just by reading the code, and many of these bugs are only revealed when the visualization is actually performed. Therefore, we introduced the Visual Regression Test to improve the review process. In the PR for visualization features, you can jump to the Visual Regression Test link by clicking on the link generated from within the PR. Reviewers can verify that the PR implementation is performing the visualization properly.

Improve Code Quality Including Many Bugfix

In the latter development cycle of Optuna v3, we put emphasis on improving the overall code quality of the library. We fixed several bugs and possible corruption of internal data structures on e.g. handling Inf/NaN values (#3567, #3592, #3738, #3739, #3740) and invalid inputs (#3668, #3808, #3814, #3819). For example, there had been bugs before v3 when NaN values were used in a CategoricalDistribution or GridSampler. In several other functions, NaN values were unacceptable but the library failed silently without any warning or error. Such bugs are fixed in this release.

New Features

• Add constraints option to TPESampler (#3506)
• Add skip_if_exists argument to enqueue_trial (#3629)
• Remove experimental from plot_pareto_front (#3643)
• Add popsize argument to CmaEsSampler (#3649)
• Add seed argument for BoTorchSampler (#3756)
• Add seed argument for SkoptSampler (#3791)
• Revert AllenNLP integration back to experimental (#3822)

Enhancements

• Move is_heartbeat_enabled from storage to heartbeat (#3596)
• Refactor ImportanceEvaluators (#3597)
• Avoid maximum limit when MLflow saves information (#3651)
• Control metric decimal digits precision in bayesmark benchmark report (#3693)
• Support inf values for crowding distance (#3743)
• Normalize importance values (#3828)

Bug Fixes

• Fix CategoricalDistribution with NaN (#3567)
• Fix NaN comparison in grid sampler (#3592)
• Fix bug in IntersectionSearchSpace (#3666)
• Remove trial_values records whose values are None (#3668)
• Fix PostgreSQL primary key unsorted problem (#3702, thanks @wattlebirdaz!)
• Raise error on NaN in _constrained_dominates (#3738)
• Fix inf-related issue on implementation of _calculate_nondomination_rank (#3739)
• Raise errors for NaN in constraint values (#3740)
• Fix _calculate_weights such that it throws ValueError on invalid weights (#3742)
• Change warning for axis_order of plot_pareto_front (#3802)
• Fix check for number of objective values (#3808)
• Raise ValueError when waiting trial is told (#3814)
• Fix Study.tell with invalid values (#3819)
• Fix infeasible case in NSGAII test (#3839)

Installation

• Add a version constraint of cached-path (#3665)

Documentation

• Add documentation of SHAP integration (#3623)
• Remove news entry on Optuna user survey (#3645)
• Introduce optuna-fast-fanova (#3647)
• Add github discussions link (#3660)
• Fix a variable name of ask-and-tell tutorial (#3663)
• Clarify which trials are used for importance evaluators (#3707)
• Fix typo in Study.optimize (#3720, thanks @29Takuya!)
• Update link to plotly's jupyterlab-support page (#3722, thanks @29Takuya!)
• Update CONTRIBUTING.md (#3726)
• Remove "Edit on Github" button (#3777, thanks @cfkazu!)
• Remove duplicated period at the end of copyright (#3778)
• Add note for deprecation of plot_pareto_front's axis_order (#3803)
• Describe the purpose of prepare_study_with_trials (#3809)
• Fix a typo in docstring of ShapleyImportanceEvaluator (#3810)
• Add a reference for MOTPE (#3838, thanks @y0z!)

Examples

• Introduce stale bot (optuna/optuna-examples#119)
• Use Hydra 1.2 syntax (optuna/optuna-examples#122)
• Fix CI due to thop (optuna/optuna-examples#123)
• Hotfix allennlp dependency (optuna/optuna-examples#124)
• Remove unreferenced variable in pytorch_simple.py (optuna/optuna-examples#125)
• set OMPI_MCA_rmaps_base_oversubscribe=yes before mpirun (optuna/optuna-examples#126)

Tests

• Simplify multi-objective TPE tests (#3653)
• Add edge cases to multi-objective TPE tests (#3662)
• Remove tests on TypeError (#3667)
• Add edge cases to the tests of the parzen estimator (#3673)
• Add tests for _constrained_dominates (#3683)
• Refactor tests of constrained TPE (#3689)
• Add inf and NaN tests for test_constraints_func (#3690)
• Fix calling storage API in study tests (#3695, thanks @wattlebirdaz!)
• DRY test_frozen.py (#3696)
• Unify the tests of plot_contours (#3701)
• Add test cases for crossovers of NSGAII (#3705)
• Enhance the tests of NSGAIISampler._crowding_distance_sort (#3706)
• Unify edf test files (#3730)
• Fix test_calculate_weights_below (#3741)
• Refactor test_intermediate_plot.py (#3745)
• Test samplers are reproducible (#3757)
• Add tests for _dominates function (#3764)
• DRY importance tests (#3785)
• Move tests for create_trial (#3794)
• Remove with_c_d option from prepare_study_with_trials (#3799)
• Use DeterministicRelativeSampler in test_trial.py (#3807)

Code Fixes

• Add typehint for deprecated and experimental (#3575)
• Move heartbeat-related thread operation in _optimize.py to _heartbeat.py (#3609)
• Remove useless object inheritance (#3628, thanks @harupy!)
• Remove useless except clauses (#3632, thanks @harupy!)
• Rename optuna.testing.integration with optuna.testing.pruner (#3638)
• Cosmetic fix in Optuna CLI (#3641)
• Enable strict_equality for mypy #3579 (#3648, thanks @wattlebirdaz!)
• Make file names in testing consistent with optuna module (#3657)
• Remove the implementation of read_trials_from_remote_storage in the all storages apart from CachedStorage (#3659)
• Remove unnecessary deep copy in Redis storage (#3672, thanks @wattlebirdaz!)
• Workaround mypy bug (#3679)
• Unify plot_contours (#3682)
• Remove storage.get_all_study_summaries(include_best_trial: bool) (#3697, thanks @wattlebirdaz!)
• Unify the logic of edf functions (#3698)
• Unify the logic of plot_param_importances functions (#3700)
• Enable disallow_untyped_calls for mypy (#3704, thanks @29Takuya!)
• Use get_trials with states argument to filter trials depending on trial state (#3708)
• Return Python's native float values (#3714)
• Simplify bayesmark benchmark report rendering (#3725)
• Unify the logic of intermediate plot (#3731)
• Unify the logic of slice plot (#3732)
• Unify the logic of plot_parallel_coordinates (#3734)
• Unify implementation of plot_optimization_history between plotly and matplotlib (#3736)
• Extract fail_objective and pruned_objective for tests (#3737)
• Remove deprecated storage functions (#3744, thanks @29Takuya!)
• Remove unnecessary optionals from visualization/_pareto_front.py (#3752)
• Change types inside _ParetoInfoType (#3753)
• Refactor pareto front (#3754)
• Use _ContourInfo to plot in plot_contour (#3755)
• Follow up #3465 (#3763)
• Refactor importances plot (#3765)
• Remove no_trials option of prepare_study_with_trials (#3766)
• Follow the coding style of comments in plot_contour files (#3767)
• Raise ValueError for invalid returned type of target in _filter_nonfinite (#3768)
• Fix value error condition in plot_contour (#3769)
• DRY constraints in Sampler.after_trial (#3775)
• DRY stop_objective (#3786)
• Refactor non-exist param test in plot_contour test (#3787)
• Remove less_than_two and more_than_three options from prepare_study_with_trials (#3789)
• Fix return value's type of _get_node_value (#3818)
• Remove unused type: ignore (#3832)
• Fix typos and remove unused argument in QMCSampler (#3837)

Continuous Integration

• Use coverage directly (#3347, thanks @higucheese!)
• Install 3rd party libraries in CI for lint (#3580)
• Make bayesmark benchmark results comparable to kurobako (#3584)
• Enable warn_unused_ignores for mypy (#3627, thanks @harupy!)
• Add onnx and version constrained protobuf to document dependencies (#3658)
• Add mo-kurobako benchmark to CI (#3691)
• Enable mypy's strict configs (#3710)
• Run visual regression tests to find regression bugs of visualization module (#3721)
• Remove downloading old libomp for mac tests (#3728)
• Match Python versions between bayesmark CI jobs (#3750)
• Set OMPI_MCA_rmaps_base_oversubscribe=yes before mpirun (#3758)
• Add budget option to benchmarks (#3774)
• Add n_concurrency option to benchmarks (#3776)
• Use n-runs instead of repeat to represent the number of studies in the bayesmark benchmark (#3780)
• Fix type hints for mypy 0.971 (#3797)
• Pin scipy to avoid the CI failure (#3834)
• Extract float value from tensor for trial.report in PyTorchLightningPruningCallback (#3842)

Other

• Clarify the criteria to assign reviewers in the PR template (#3619)
• Bump up version number to v3.0.0rc0.dev (#3621)
• Make tox.ini consistent with checking (#3654)
• Avoid to stale description-checked issues (#3816)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@29Takuya, @HideakiImamura, @c-bata, @cfkazu, @contramundum53, @g-votte, @harupy, @higucheese, @himkt, @hvy, @keisuke-umezawa, @knshnb, @not522, @nzw0301, @sile, @toshihikoyanase, @wattlebirdaz, @xadrianzetx, @y0z