Add optimizer from bayesian-optimization

[spline2hg]

This PR adds the optimizer from bayesian-optimization library.

Implemented Features

• Basic Bayesian Optimization algorithm interface
• Support for built-in acquisition functions (UCB, EI, POI)
• Sequential Domain Reduction (SDR)
• Added documentation and tests

Remaining Work

• Support for nonlinear constraints
  constraint handling is currently skipped. Some changes are required in the way constraints are processed, before this can be properly implemented.
• Implementation of meta-acquisition functions (ConstantLiar and GPHedge)

[codecov]

Codecov Report

Attention: Patch coverage is 90.69767% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/optimagic/optimizers/bayesian_optimizer.py	90.09%	10 Missing ⚠️
src/optimagic/config.py	60.00%	2 Missing ⚠️

Files with missing lines	Coverage Δ
src/optimagic/algorithms.py	86.02% <100.00%> (+0.08%)	⬆️
src/optimagic/optimizers/iminuit_migrad.py	94.87% <100.00%> (-2.43%)	⬇️
src/optimagic/typing.py	86.84% <100.00%> (+0.17%)	⬆️
src/optimagic/config.py	70.12% <60.00%> (+0.68%)	⬆️
src/optimagic/optimizers/bayesian_optimizer.py	90.09% <90.09%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[janosg]

Hi @spline2hg, thanks for the great PR.

The PR is quite good already but I have made many small comments about the code style.

I'm happy to talk if there are questions.

[janosg]

Looks very nice. Just two very small comments.

[timmens]

@spline2hg I had a brief look at the mypy errors.

1. Can you please make sure that we are using the same mypy version in the pre-commit hooks and the development environment.
2. It seems that most errors occur because a type ignore statement is not necessary (anymore). In this case you can simply remove it.

[timmens]

Hey again! Thanks for the changes. I had a deeper look into the mypy issues, some of which were unrelated to this PR, and fixed them in #601, which was just merged into main.

You should be able to update this branch with main to resolve the issues. For now, we will stick with mypy version 1.14.1.

Please ping me if there are remaining errors!

[spline2hg]

@timmens, the pre-commit mypy check is passing, but the mypy in environment is raising an error about assigning to Any.Adding ignore to it is causing pre-commit mypy to complain.
Do we need to change the way we’re handling the optional dependency with bayes_opt?

[janosg]

Have you tried to re-create your local conda environment to get the same version of mypy as in the CI runs?

[spline2hg]

@janosg yes i have updated the environment, current mypy version in my environment is 1.14.1

[timmens]

@spline2hg @janosg

I've had a look at the situation and have a proposal. I did not, however, find out why this is a problem here and not with the iminuit library (my hunch is that it is because bayes_opt is typed but iminuit is not). Let me know what you think!

Proposal:

1. We import all runtime dependencies inside of functions that are only called when the optional dependency (here bayes_opt) is installed.
2. We import all required type annotations separately in a TYPE_CHECKING block.
3. We put the type annotations that come from the optional dependency in quotes so that the code does not fail at runtime if the optional dependency is not installed

Example:

1.

def _solve_internal_problem(
    self, problem: InternalOptimizationProblem, x0: NDArray[np.float64]
) -> InternalOptimizeResult:
    if not IS_BAYESOPT_INSTALLED:
        raise NotInstalledError(
            "To use the 'bayes_opt' optimizer you need to install bayes_opt. "
            "Use 'pip install bayesian-optimization'. "
            "Check the documentation for more details: "
            "https://bayesian-optimization.github.io/BayesianOptimization/index.html"
        )
        
    from bayes_opt import BayesianOptimization, SequentialDomainReductionTransformer

2.

if TYPE_CHECKING:
    from bayes_opt import BayesianOptimization
    from bayes_opt.acquisition import AcquisitionFunction

Since we import the runtime dependencies inside the function, we only need these objects for the type checking, and therefore also only if TYPE_CHECKING is true. If the optional dependency is not installed this would raise an ImportError, but we only type check in an environment where this is installed. If we want to avoid this altogether we could add and IS_BAYES_OPT_INSTALLED.

3.

def _process_bayes_opt_result(
    optimizer: "BayesianOptimization",
    x0: NDArray[np.float64],
    problem: InternalOptimizationProblem,
) -> InternalOptimizeResult:

Implications:

If we decide to follow this route, then we should do the same for iminuit!

Additionally:

We should remove bayes_opt from [[tool.mypy.overrides]] since it is actually typed.

[spline2hg]

@timmens Thanks for the detailed proposal! I had missed that bayes_opt is actually typed ,and that was causing the issue, thanks for pointing it out.

I’ve added the following changes:

• runtime imports inside required functions
• type imports under TYPE_CHECKING
• added from future import annotations so annotations are automatically stringified (instead explicit quotes)
• dropped the ignore_missing_imports stanza for bayes_opt

Happy to mirror the same fix for iminuit in the same PR.

[timmens]

Thanks! This looks good to me, also the from __future__ import annotations part! Before you mirror this for iminuit, lets see what @janosg says.

[janosg]

Thanks @timmens and @spline2hg. This looks good!

• Please mirror it for iminuit!
• Could you also remove iminuit from the dependency list in pyproject.toml (here)? I know it's not related to your PR but we discovered it while looking at the problem.

[janosg]

Thanks. This is ready to merge!