Add constraints option to TPESampler #3506

knshnb · 2022-04-24T15:39:32Z

Motivation

I implemented a constrained optimization option in multi-objective TPESampler with the similar strategy as NSGA-II (#2175).
Interfaces are same as NSGAIISampler.

Description of the changes

Add constraints_func argument
Make _get_observation_pairs also return constraints of trials
When dividing into below and above in TPE, consider constraint values before HSSP

I'm considering if I should enable the constrained optimization in TPESampler, which can be naturally introduced when implementing constrained optimization in MOTPESampler.

himkt · 2022-04-25T05:49:39Z

Could you please be a reviewer @HideakiImamura @toshihikoyanase?

knshnb · 2022-04-25T11:42:14Z

I checked if it works in a toy example, which is modified a little from Binh and Corn function with constraints.

from typing import Tuple

import optuna


def objective(trial: optuna.Trial) -> Tuple[float, float]:
    # Binh and Korn function with constraints.
    x = trial.suggest_float("x", -15, 30)
    y = trial.suggest_float("y", -15, 30)

    v0 = 4 * x**2 + 4 * y**2
    v1 = (x - 5) ** 2 + (y - 5) ** 2

    # Modified constraints function.
    trial.set_user_attr("constraint", (1000 - v0,))

    return v0, v1


def constraints(trial: optuna.Trial) -> Tuple[float]:
    return trial.user_attrs["constraint"]


if __name__ == "__main__":
    samplers = {
        "nsgaii": optuna.samplers.NSGAIISampler,
        "motpe": optuna.samplers.TPESampler,
    }
    for use_const in [False, True]:
        for sampler_name in ["nsgaii", "motpe"]:
            sampler = samplers[sampler_name](
                seed=0, constraints_func=constraints if use_const else None
            )
            study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler)
            study.optimize(objective, n_trials=500)
            optuna.visualization.plot_pareto_front(
                study, target_names=["v0", "v1"], constraints_func=constraints
            ).write_image(f"result/{sampler_name}_{'constraints' if use_const else ''}_pareto.jpg")

knshnb · 2022-04-25T13:28:02Z

MOTPE without constraints
MOTPE with constraints (this PR)
NSGA2 without constraints
NSGA2 with constraints

MOTPE without constraints seemed to exploit in early trials and did not work well compared to NSGA2 without constraints. Therefore, MOTPE improved a lot by considering constraints and worked comparably as NSGA2 with constraints.

knshnb · 2022-04-25T13:32:24Z

As discussed privately with @HideakiImamura and @toshihikoyanase, I'll consider some other strategies of how to calculate weights for _ParzenEstimator by testing on a task with a smaller feasible region.

knshnb · 2022-05-06T09:17:46Z

I tried another toy example that has a smaller feasible region, as suggested by @HideakiImamura.

from typing import Tuple

import numpy as np

import optuna


def objective(trial: optuna.Trial) -> Tuple[float, float]:
    x = trial.suggest_float("x", 0, 6)
    y = trial.suggest_float("y", 0, 6)

    c = np.sin(x) * np.sin(y) + 0.95
    trial.set_user_attr("constraint", (c,))

    f1 = np.sin(x) - y
    f2 = np.cos(x) + y**2

    return f1, f2


def constraints(trial: optuna.Trial) -> Tuple[float]:
    return trial.user_attrs["constraint"]


if __name__ == "__main__":
    samplers = {
        "nsgaii": optuna.samplers.NSGAIISampler,
        "motpe": optuna.samplers.TPESampler,
    }
    for use_const in [False, True]:
        for sampler_name in ["nsgaii", "motpe"]:
            sampler = samplers[sampler_name](
                seed=0, constraints_func=constraints if use_const else None
            )
            study = optuna.create_study(sampler=sampler, directions=["minimize", "minimize"])
            study.optimize(objective, n_trials=400)

            fig = optuna.visualization.plot_pareto_front(study, constraints_func=constraints)
            fig.write_image(
                f"result/small_feasible/{sampler_name}_{'constraints' if use_const else ''}_pareto.jpg"
            )

MOTPE without constraints
MOTPE with constraints (this PR)
NSGA2 without constraints
NSGA2 with constraints

The current implementation already works fine (seems slightly better than NSGA2 with constraints), but anyway I will try another strategy for the weight calculation.

knshnb · 2022-05-06T10:35:52Z

In the current implementation, I set EPS weights for _ParzenEstimator to all the infeasible trials.

Another option we can consider is to use the same strategy of weight calculation as the single-objective TPE: put more weights on new trials. With this (knshnb#1), it seems to exploit in the earlier phase and does not explore well.

knshnb · 2022-05-06T11:33:20Z

I tried another strategy (knshnb#2) of weight calculation: just ignore all the infeasible trials in below (similar to this PR, but different in that infeasible trials are ignored also in hypervolume calculation).
I think this is natural and it seems to be working fine.

knshnb · 2022-05-12T06:50:05Z

I implemented another strategy: ignore all infeasible trials in hypervolume calculation for ParzenEstimator's weight and set infeasible trials' weight EPS.

The experimental results are fine overall, although they are dependent on random seeds.
Seed 0:

Seed 1:

Seed 2:

Seed 3:

Seed 4:

knshnb · 2022-05-12T06:57:40Z

As a side note, there is some room for improvement, such as

using EPS as infeasible trials' weight might be too small
- it depends on the scale of hypervolume, so we might need to consider normalization in output space
we set the same weight for all infeasible trials in below, regardless of constraints values

However, I think the current implementation is enough as a straightforward extension of the method used in NSGAIISampler.

knshnb · 2022-05-12T09:56:35Z

I supported sample_independent and tested on multivariate=True as well.
The results might be less dependent on random seeds when multivariate=True.

Seed 0:

Seed 1:

Seed 2:

Seed 3:

Seed 4:

HideakiImamura

Thanks for the great improvement of TPESampler! I have several comments. PTAL.

optuna/samplers/_tpe/sampler.py

HideakiImamura · 2022-05-18T06:54:12Z

optuna/samplers/_tpe/sampler.py

+) -> Tuple[
+    Dict[str, List[Optional[float]]],
+    List[Tuple[float, List[float]]],
+    Optional[List[Sequence[float]]],


Why not force it to be a tuple, for example, instead of an arbitrary Sequence? Ambiguities will be reduced and bugs will be less likely to be created in the future.

The return type of constraints_func is Sequence[float] following NSGAIISampler. Do you suggest to cast from Sequence[float] to Tuple[float] somewhere?

optuna/optuna/samplers/nsgaii/_sampler.py

Line 121 in 0964da2

constraints_func: Optional[Callable[[FrozenTrial], Sequence[float]]] = None,

Yes, I mean we can cast it.

Now we do not need to cast by 2f1ff5a.

HideakiImamura · 2022-05-18T07:04:01Z

optuna/samplers/_tpe/sampler.py

-    return values, scores
+        if get_constraints:
+            assert constraints is not None
+            constraints.append(cast(Sequence[float], trial.system_attrs.get(_CONSTRAINTS_KEY)))


If trial.system_attrs misses the key of _CONSTRAINTS_KEY, the result of cast(Sequence[float], trial.system_attrs.get(_CONSTRAINTS_KEY)) will be None. So, constraints can include None.

(1) The type of constraints should be Optional[List[Optional[Sequence[float]]]].
(2) The handling of constraints after this function returns should be implemented with the consideration that there may be None inside. (I have not checked the handling after this function yet.)

Resolved by 2f1ff5a.

HideakiImamura · 2022-05-18T07:14:23Z

optuna/samplers/_tpe/sampler.py

+    # 3. Feasible trials are sorted by loss_vals.
+    if constraints is not None:
+        # 1-dimensional violation value (violation_1d==0 <=> feasible, >0 <=> infeasible).
+        violation_1d = np.maximum(np.array(constraints), 0).sum(1)


If constraints includes None, the type of each item in np.array(constraints) includes None. How about specifying the dtype like np.array(constraints, dtype=float)? It convert None to np.nan, so we need to filter np.nan before taking np.maximum to compare violation_1d[idx[n_below]] > 0.

I did not consider the case constraints includes None. When constraints include None, it cannot be simply converted using np.array because constraints might be something like [(1.0, 1.0), None, (2.0, 2.0)] and the dimension does not match. I need to consider how to handle this.

I decided to handle None inside _get_observation_pairs. This makes _get_observation_pairs a bit more complex, but the whole logic and return type are simplified if we need to consider None.
2f1ff5a#diff-2cc3a4088f95a10e806b68655d375f6f7f5fe2de161a0182c16168e378be8065R713-R715

HideakiImamura · 2022-05-19T07:53:04Z

By the way, I think it is good idea to add some unit tests to validate the behavior.

knshnb · 2022-06-03T10:38:11Z

@toshihikoyanase Thanks for pointing out the case that causes an error! It happens when n_startup_trials is 0 or 1 (not related to constant_liar). This was caused because I did not consider the case that _split_observation_pars is called for a very small number of trials (By default, RandomSampler is used in such cases).

I added a test for that case (2290708) and fixed the code (7a6ca0b).

@HideakiImamura Thanks for the comments! Please take a look again 🙏

toshihikoyanase · 2022-06-07T09:46:50Z

@knshnb Thank you for your investigation! I confirmed that the current TPESampler works with n_startup_trials=1. I'll check the details of the code next!

HideakiImamura

Thanks for the update. I have several comments. PTAL.

optuna/samplers/_tpe/sampler.py

HideakiImamura · 2022-06-07T06:56:01Z

optuna/samplers/_tpe/sampler.py

+        if n_below >= len(idx) or violation_1d[idx[n_below]] > 0:
+            # if violation_1d[idx[n_below]] > 0:
+            # Below is filled by all feasible trials and trials with smaller violation values.
+            indices_below = idx[:n_below]
+            indices_above = idx[n_below:]


Let me clarify this part for my own understanding (I assume that we remove taking max in line 713.)

violation_1d[idx] is a sequence of the values of violations in ascending order, for example, [-10.0, -1.0, 0.0, 1.0, 2.0, 5.0, 10.0]. The first 3 values are feasible.

Let n_below = 5. Then violation_1d[idx[n_below]] = 5.0 > 0, so all feasible trials [-10.0, -1.0, 0.0] are included in the below. In addition, we should include 1.0 and 2.0 in the below. We select the infeasible trials with smaller violations.

HideakiImamura · 2022-06-08T08:19:59Z

tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py

-def test_multi_objective_get_observation_pairs() -> None:
+@pytest.mark.parametrize("constraints_enabled", [False, True])
+@pytest.mark.parametrize("constraint_value", [-2, 2])
+def test_multi_objective_get_observation_pairs(


HideakiImamura · 2022-06-08T08:40:41Z

tests/samplers_tests/tpe_tests/test_sampler.py

    )
    assert list(indices_below) == [0, 3]
    assert list(indices_above) == [1, 2]


+def test_split_observation_pairs_with_constraints() -> None:


Why not test both cases, ``if n_below >= len(idx) or violation_1d[idx[n_below]] > 0` or not then?

toshihikoyanase

The code basically looks good to me. Let me add some cosmetic comments.

toshihikoyanase · 2022-06-08T07:51:19Z

optuna/samplers/_tpe/sampler.py

+            warnings.warn(
+                "The constraints_func option is an experimental feature."
+                " The interface can change in the future.",
+                ExperimentalWarning,
+            )


Do we add a test about this ExperimentalWanring ? The multivariate argument has such a test case.

optuna/tests/samplers_tests/tpe_tests/test_sampler.py

Lines 35 to 37 in 83c8022

def test_multivariate_experimental_warning() -> None:

with pytest.warns(optuna.exceptions.ExperimentalWarning):

optuna.samplers.TPESampler(multivariate=True)

optuna/samplers/_tpe/sampler.py

toshihikoyanase · 2022-06-08T08:44:35Z

tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py

-def test_multi_objective_get_observation_pairs() -> None:
+@pytest.mark.parametrize("constraints_enabled", [False, True])
+@pytest.mark.parametrize("constraint_value", [-2, 2])
+def test_multi_objective_get_observation_pairs(


I agree with @HideakiImamura . This test changes the expected values depending on the parametrized value (i.e., constraints_enabled), and I think it is a kind of conditional test logic. As you mentioned, we can work on it in a follow-up PR.

tests/samplers_tests/tpe_tests/test_sampler.py

toshihikoyanase · 2022-06-08T08:51:28Z

tests/samplers_tests/tpe_tests/test_sampler.py

+    assert _tpe.sampler._get_observation_pairs(
+        study, ["y"], True, constraints_enabled=constraints_enabled
+    ) == ({"y": []}, [], [] if constraints_enabled else None)


ditto. This test function also changes the expected values depending on the parametrized value.

Co-authored-by: Toshihiko Yanase <toshihiko.yanase@gmail.com>

knshnb · 2022-06-09T08:48:52Z

@HideakiImamura @toshihikoyanase I updated the code! PTAL.

HideakiImamura

Thanks for the update. Almost LGTM. I have a nit comment.

By the way, since we have slightly changed the implementation, could you take a benchmark again to confirm the no performance degradation?

HideakiImamura · 2022-06-10T05:58:17Z

optuna/samplers/_tpe/sampler.py

+        else:
+            # All trials in below are feasible.
+            # Feasible trials with smaller loss_vals are selected.
+            (feasible_idx,) = (violation_1d == 0).nonzero()


Suggested change

(feasible_idx,) = (violation_1d == 0).nonzero()

(feasible_idx,) = (violation_1d <= 0).nonzero()

As we privately discussed, I keep this line because violation_1d is guaranteed to be greater than or equal to 0.
https://github.com/optuna/optuna/pull/3506/files#diff-2cc3a4088f95a10e806b68655d375f6f7f5fe2de161a0182c16168e378be8065R712-R713

knshnb · 2022-06-10T08:46:59Z

Since there has been a large amount of code modification, I confirmed that the current code works fine on the toy problem #3506 (comment).

seed=0

seed=1

HideakiImamura

Thanks for the long running work! LGTM!

toshihikoyanase

LGTM. Thank you for accomplishing the long running PR!

[Notes] I checked the behavior of constrained MOTPE with the following simple code.

import optuna

def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    y = trial.suggest_float("y", -10, 10)
    trial.set_user_attr("constraint", [y - 5])
    return x, y

def constraints_func(trial):
    return trial.user_attrs["constraint"]

study = optuna.create_study(
    directions=["maximize", "maximize"],
    sampler=optuna.samplers.TPESampler(
        multivariate=True,
        constraints_func=constraints_func,
    ),
)
study.optimize(objective, n_trials=100)

optuna.visualization.plot_pareto_front(study, constraints_func=constraints_func).show()
optuna.visualization.plot_optimization_history(study, target=lambda t: t.values[0]).show()
optuna.visualization.plot_optimization_history(study, target=lambda t: t.values[1]).show()

The constrained MOTPE intensively search around x=10, y=5 instead of x=10, y=10 since y>5 is infeasible.

By comparing optimization histories of x and y, we can see that the sampler gradually focused on y=5 and I guess it can successfully choose the search points considering the constraint.

Optimization history of x

Optimization history of y

toshihikoyanase · 2022-06-14T13:20:33Z

I found a minor bug of TPESampler during the review and reported in #3670.

okaikov · 2022-11-30T14:47:13Z

is it possible to implement constraints for single objective TPE? if so, are you planning to add this feature?

knshnb · 2022-12-01T06:55:08Z

Actually, constrained optimization for single objective TPE itself is supported by this PR as well. However, we haven't developed the related features (such as log, visualization, etc.) and dogfooded well yet. Feel free to request if you encounter any problems!

import optuna


def objective(trial):
    x = trial.suggest_float("x", -1, 1)
    trial.set_user_attr("constraint", (-x,))
    return x


def constraints_func(trial):
    return trial.user_attrs["constraint"]


sampler = optuna.samplers.TPESampler(constraints_func=constraints_func)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=30)

Sorry, log of the best value does not take the constraint into account, but the optimization itself can work.

okaikov · 2022-12-01T07:11:29Z

The reason I mentioned this is because I get the following warning on all trials when running single objective TPE with constraints:
optuna/samplers/_tpe/sampler.py:680: UserWarning: Trial 0 does not have constraint values. It will be treated as a lower priority than other trials.

knshnb · 2022-12-01T07:14:26Z

Could you share the reproducible code?

okaikov · 2022-12-01T08:25:07Z

this happens when constant_liar==True.
just add "constant_liar=True" to the TPE sampler

knshnb · 2022-12-01T11:00:54Z

@okaikov Thanks for the report!
I created an issue. It seems that the warning is a bug and you can ignore it for now.

okaikov · 2023-04-27T09:28:16Z

Is it possible to provide a parameter to the constraints function? Example: constraints_func(trial=trial, metrics=metrics)
The use case is that I want to change the constraints during the study, so I don't want to set hardcoded values to the usr_attr and read it from there, instead I will compute the constraints dynamically.

knshnb · 2023-04-27T09:35:45Z

Thanks for the comment! Could you create a new issue with a more detailed explanation of how you want to change the constraints during the study?

Implement constraints for sample_independent in TPESampler

5dc852d

github-actions bot added the optuna.samplers Related to the `optuna.samplers` submodule. This is automatically labeled by github-actions. label Apr 24, 2022

toshihikoyanase added the feature Change that does not break compatibility, but affects the public interfaces. label Apr 25, 2022

himkt assigned toshihikoyanase and HideakiImamura Apr 25, 2022

knshnb changed the title ~~[WIP] Add constraints option to MOTPESampler~~ [WIP] Add constraints option to TPESampler Apr 25, 2022

knshnb added 2 commits April 25, 2022 20:35

Revert modification of MOTPESampler

9e495bb

Add docstring to TPESampler

edfc341

Fix mypy

11f8b20

knshnb added 2 commits May 12, 2022 15:39

Rename local variable

9f12426

Ignore infeasible trials in below when calculating hypervolume

bdc7222

knshnb added 5 commits May 12, 2022 16:18

Fix infeasible weight from 0.1 to EPS!

eff6f7d

Refactor and support constraints in sample_relative

521cd3d

Sort each of split indices

00f427d

Add comments

66f3ed9

Format

1e2bca1

HideakiImamura reviewed May 18, 2022

View reviewed changes

knshnb added 2 commits May 18, 2022 17:20

Add experimental in docstring

134cf32

Make constraints_func argment keyward-only

0fc2fd6

HideakiImamura reviewed Jun 8, 2022

View reviewed changes

toshihikoyanase reviewed Jun 8, 2022

View reviewed changes

knshnb and others added 4 commits June 8, 2022 20:01

Delete debug comment

8b4eea5

Test both cases of _split_observation_pairs

233c91b

Add test of warninig

d98a33f

Apply suggestions from code review

95e6dd3

Co-authored-by: Toshihiko Yanase <toshihiko.yanase@gmail.com>

Refactor

f9b7cf7

HideakiImamura reviewed Jun 10, 2022

View reviewed changes

HideakiImamura approved these changes Jun 14, 2022

View reviewed changes

toshihikoyanase approved these changes Jun 14, 2022

View reviewed changes

toshihikoyanase added this to the v3.0.0-rc0 milestone Jun 14, 2022

toshihikoyanase merged commit d3c53e0 into optuna:master Jun 14, 2022

This was referenced Jun 15, 2022

Improve constrained TPE #3671

Closed

Refactor tests of constrained TPE #3689

Merged

nzw0301 mentioned this pull request Jul 12, 2022

DRY constraints in Sampler.after_trial #3775

Merged

knshnb deleted the motpe-constraints branch October 3, 2022 08:55

knshnb mentioned this pull request Dec 1, 2022

Constrained single-objective TPE always throws unnecessary warnings when constant_liar=True #4221

Closed

	def test_multivariate_experimental_warning() -> None:
	with pytest.warns(optuna.exceptions.ExperimentalWarning):
	optuna.samplers.TPESampler(multivariate=True)

	(feasible_idx,) = (violation_1d == 0).nonzero()
	(feasible_idx,) = (violation_1d <= 0).nonzero()

Add constraints option to TPESampler #3506

Add constraints option to TPESampler #3506

Conversation

knshnb commented Apr 24, 2022 • edited

Motivation

Description of the changes

himkt commented Apr 25, 2022

knshnb commented Apr 25, 2022 • edited

knshnb commented Apr 25, 2022 • edited

knshnb commented Apr 25, 2022

knshnb commented May 6, 2022

knshnb commented May 6, 2022

knshnb commented May 6, 2022 • edited

knshnb commented May 12, 2022

knshnb commented May 12, 2022

knshnb commented May 12, 2022 • edited

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knshnb May 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HideakiImamura commented May 19, 2022

knshnb commented Jun 3, 2022

toshihikoyanase commented Jun 7, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toshihikoyanase left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knshnb commented Jun 9, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knshnb commented Jun 10, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

toshihikoyanase left a comment

Choose a reason for hiding this comment

toshihikoyanase commented Jun 14, 2022

okaikov commented Nov 30, 2022

knshnb commented Dec 1, 2022 • edited

okaikov commented Dec 1, 2022

knshnb commented Dec 1, 2022

okaikov commented Dec 1, 2022 • edited

knshnb commented Dec 1, 2022

okaikov commented Apr 27, 2023

knshnb commented Apr 27, 2023

knshnb commented Apr 24, 2022 •

edited

knshnb commented Apr 25, 2022 •

edited

knshnb commented Apr 25, 2022 •

edited

knshnb commented May 6, 2022 •

edited

knshnb commented May 12, 2022 •

edited

knshnb May 18, 2022 •

edited

knshnb commented Dec 1, 2022 •

edited

okaikov commented Dec 1, 2022 •

edited