Add import for public API of optuna.terminator to optuna/terminator/__init__.py

[cross32768]

Motivation

To make optuna.terminator related import more user-friendly, writing some import sentence to optuna/terminator/__init__.py is needed.

For example, current RegretBoundEvaluator requires

from optuna.terminator.improvement.evaluator import RegretBoundEvaluator

This PR make it more user-friendly as below:

from optuna.terminator import RegretBoundEvaluator

Description of the changes

Write some import sentence to optuna/terminator/__init__.py and changes import-style in tests and visualization to make it simpler.

[cross32768]

Discussion-needed point:

Below is the list of API that this PR make its import more user-friendly (they can be imported by from optuna.terminator ). Is this reasonable choice?

• TerminatorCallback
• BaseErrorEvaluator
• CrossValidationErrorEvaluator
• report_cross_validation_scores
• StaticErrorEvaluator
• BaseImprovementEvaluator
• RegretBoundEvaluator
• BaseTerminator
• Terminator

[codecov-commenter]

Codecov Report

Merging #4669 (d536f0c) into master (d199673) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@           Coverage Diff           @@
##           master    #4669   +/-   ##
=======================================
  Coverage   90.62%   90.63%           
=======================================
  Files         186      187    +1     
  Lines       14292    14302   +10     
=======================================
+ Hits        12952    12962   +10     
  Misses       1340     1340           

Impacted Files	Coverage Δ
optuna/terminator/__init__.py	100.00% <100.00%> (ø)
optuna/visualization/_terminator_improvement.py	36.70% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[HideakiImamura]

@toshihikoyanase Could you review this PR?

[HideakiImamura]

LGTM.

[toshihikoyanase]

Thank you for your PR. The change looks great. Let me note some offline discussions.

Alternative approach: Export sub-modules instead of individual classes

The PR contains many Evaluators in optuna.terminator, which suggests that perhaps we should consider exporting sub-modules.

from optuna.terminator import callback
from optuna.terminator import erroreval
from optuna.terminator import improvement
from optuna.terminator.terminator import BaseTerminator
from optuna.terminator.terminator import Terminator

We decided against this approach, primarily because most users typically use just a few:

from optuna.terminator import report_cross_validation_scores, TerminatorCallback

If we were to only export sub-modules, the import statements would likely become more complex, as shown here:

from optuna.terminator.errorval import report_cross_validation_scores
from optuna.terminator.callback import TerminatorCallback

Avoid exporting base classes

It's important to note that base classes, such as BaseTerminator and BaseImprovementEvaluator, are still experimental. As a result, we might introduce some significant changes in future versions. If that happens, any terminators defined by users won't work with the updated versions. To prevent this, we may choose not to export these base classes to discourage users from defining their own classes.

Conversely, we recognize the potential for considerable improvements to our APIs by creating user-defined terminators. So we're eager to promote the use of these base classes and highly appreciate any user feedback.

[toshihikoyanase]

We can see optuna.terminator.callback in the docstring of Terminator, but we can update it after #4675.