Add log-linear algorithm for 2d Pareto front. #2503

parsiad · 2021-03-20T04:33:25Z

Motivation

study.best_trials can be made faster when there are only two objectives.

Description of the changes

Uses a log-linear algorithm algorithm to construct the Pareto frontier in the case of two objectives that works as follows:

Sort trials by their x coordinate.
Iterate through the sorted trials and drop any trial that does not have a better y coordinate than the current best y coordinate.

codecov-io · 2021-03-20T08:27:49Z

Codecov Report

Merging #2503 (54c51a6) into master (50290f0) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2503      +/-   ##
==========================================
+ Coverage   89.78%   89.83%   +0.04%     
==========================================
  Files         135      135              
  Lines       11308    11338      +30     
==========================================
+ Hits        10153    10185      +32     
+ Misses       1155     1153       -2

Impacted Files	Coverage Δ
optuna/_multi_objective.py	`93.15% <100.00%> (+4.77%)`	⬆️
optuna/integration/botorch.py	`98.65% <0.00%> (+0.89%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50290f0...54c51a6. Read the comment docs.

crcrpar · 2021-03-21T01:52:34Z

Thank you for the great improvement.

Tests are okay, and outputs also look solid.

Stable: https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.plot_pareto_front.html?highlight=pareto

This pr: https://66655-122299416-gh.circle-artifacts.com/0/docs/build/html/reference/visualization/generated/optuna.visualization.plot_pareto_front.html#optuna.visualization.plot_pareto_front

--

Casual benchmark taken with pytest-benchmark, code: https://github.com/crcrpar/optuna/tree/benchmark-new-2dparetofront

This pull request

----------------------------------------------- benchmark: 1 tests ----------------------------------------------
Name (time in ms)           Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------
test_bench_pareto2d     51.4555  73.0225  58.9390  6.6572  56.6273  6.4804       3;1  16.9667      11           1
-----------------------------------------------------------------------------------------------------------------

Current master branch

------------------------------------------------- benchmark: 1 tests -------------------------------------------------
Name (time in ms)            Min       Max      Mean   StdDev    Median      IQR  Outliers     OPS  Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------
test_bench_pareto2d     131.8478  160.9289  141.8700  10.6478  139.5337  14.7414       1;0  7.0487       7           1
----------------------------------------------------------------------------------------------------------------------

keisuke-umezawa · 2021-03-21T06:27:01Z

@HideakiImamura
Could you also review it?

parsiad · 2021-03-21T19:32:14Z

I have pushed another commit because the algorithm was missing an edge case. For example, if the directions are ['minimize', 'maximize'] and the input points are [1, 1], [2, 2], [3, 2], the algorithm would have erroneously included all of them instead of dropping [3, 2]. In general, if the x coordinate gets worse (2 -> 3) but the y coordinate stays the same (2 -> 2), the algorithm was erroneously not classifying the new point as dominated.

I tried to run tests twice, but it looks like they are failing due to an unrelated issue:

E   OSError: libopenblas.so.0: cannot open shared object file: No such file or directory

For what it's worth, tests pass locally on my machine.

crcrpar · 2021-03-22T00:57:19Z

It seems like the latest mxnet caused that error, and I confirmed the tests on which this pull request has some effect pass: https://github.com/optuna/optuna/pull/2503/checks?check_run_id=2161074877#step:8:36.

Sorry, but Tests (Integration) is flakier than Tests because it has many requirements.

parsiad · 2021-03-22T02:14:58Z

Sorry, but Tests (Integration) is flakier than Tests because it has many requirements.

No worries. I'll just leave it as is unless you need a passing ✔️ to merge.

Crissman · 2021-03-23T02:50:17Z

PR #2508 fixed the MXNet problem. Please re-base with this PR. 🙇‍♂️

Crissman

LGTM. Thank you for the PR!

parsiad · 2021-03-23T05:42:31Z

PR #2508 fixed the MXNet problem. Please re-base with this PR. bowing_man

Done. Tests pass now.

Thank you for your review!

HideakiImamura

Thanks for the PR! I have a comment to improve the code readability. Could take a look?

HideakiImamura · 2021-03-24T05:53:30Z

optuna/_multi_objective.py

@@ -8,7 +8,46 @@
 from optuna.trial import TrialState


-def _get_pareto_front_trials(study: "optuna.study.BaseStudy") -> List[FrozenTrial]:
+def _get_pareto_front_trials_2d(study: "optuna.study.BaseStudy") -> List[FrozenTrial]:


How about sorting the trials themself, and using _dominates function as done in _get_pareto_front_trials_nd?

pareto_front = [] trials = [trial for trial in study.trials if trial.state == TrialState.COMPLETE] trials.sort(key=lambda t: _normalize_value(t.values[1], study.directions[1])) trials.sort(key=lambda t: _normalize_value(t.values[0], study.directions[0])) last_nondominated_trial: Optional[FrozenTrial] = None for i in range(len(trials)): if i == 0: pareto_front.append(trials[0]) last_nondominated_trial = trials[0] continue assert last_nondominated_trial is not None if not _dominates(last_nondominated_trial, trials[i], study.directions): pareto_front.append(trials[i]) last_nondominated_trial = trials[i]

This code does not preserve the order of all trials and pareto front trials. so it will be fixed.

I like the idea of using _dominates to simplify the code. I implemented your suggestion.

The only difference between my implementation and your suggestion is that I am still using mask because the order of trials needs to be preserved when filtering (as you point out in your comment).

To preserve the order of trials, we only need to sort the Pareto front trials with its trial.number. Could you remove the mask? (I have noticed this simple fix after posting the above messages. Sorry.)

Nice catch. Done.

g-votte · 2021-03-24T07:23:12Z

optuna/_multi_objective.py

+def _get_pareto_front_trials(study: "optuna.study.BaseStudy") -> List[FrozenTrial]:
+    if len(study.directions) == 2:
+        return _get_pareto_front_trials_2d(study)  # Log-linear in number of trials.
+    return _get_pareto_front_trials_nd(study)  # Quadratic in number of trials.


The non-2d scenario seems not to be covered by any test case in test_study.py. We may want to add new ones, since it's risky to leave the general logic not tested.

Added a 3d test case.

HideakiImamura

LGTM!

crcrpar added enhancement Change that does not break compatibility and not affect public interfaces, but improves performance. optuna.visualization Related to the `optuna.visualization` submodule. This is automatically labeled by github-actions. labels Mar 21, 2021

keisuke-umezawa assigned crcrpar and HideakiImamura Mar 21, 2021

Crissman self-requested a review March 22, 2021 02:39

Crissman approved these changes Mar 23, 2021

View reviewed changes

parsiad added 2 commits March 22, 2021 22:02

Add log-linear algorithm for 2d Pareto front.

11b5187

Add edge case to 2d Pareto front algorithm.

54c51a6

HideakiImamura reviewed Mar 24, 2021

View reviewed changes

Use _dominates in 2d Pareto front algorithm.

0fbc94a

g-votte reviewed Mar 24, 2021

View reviewed changes

crcrpar assigned HideakiImamura and unassigned HideakiImamura and crcrpar Mar 24, 2021

parsiad added 2 commits March 24, 2021 00:48

Add 3d Pareto front test case.

846b78f

Remove mask from 2d Pareto front algorithm.

ab17d55

HideakiImamura approved these changes Mar 25, 2021

View reviewed changes

HideakiImamura merged commit df067d9 into optuna:master Mar 25, 2021

HideakiImamura added this to the v2.7.0 milestone Mar 25, 2021

c-bata mentioned this pull request Mar 25, 2021

Use log-linear algorithm to get pareto front trials. optuna/optuna-dashboard#63

Closed

parsiad deleted the log-linear-2d-pareto branch March 25, 2021 04:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add log-linear algorithm for 2d Pareto front. #2503

Add log-linear algorithm for 2d Pareto front. #2503

parsiad commented Mar 20, 2021

codecov-io commented Mar 20, 2021 •

edited

crcrpar commented Mar 21, 2021 •

edited

keisuke-umezawa commented Mar 21, 2021

parsiad commented Mar 21, 2021 •

edited

crcrpar commented Mar 22, 2021

parsiad commented Mar 22, 2021

Crissman commented Mar 23, 2021

Crissman left a comment

parsiad commented Mar 23, 2021

HideakiImamura left a comment

HideakiImamura Mar 24, 2021

parsiad Mar 24, 2021 •

edited

HideakiImamura Mar 24, 2021 •

edited

parsiad Mar 24, 2021

g-votte Mar 24, 2021

parsiad Mar 24, 2021 •

edited

HideakiImamura left a comment

Add log-linear algorithm for 2d Pareto front. #2503

Add log-linear algorithm for 2d Pareto front. #2503

Conversation

parsiad commented Mar 20, 2021

Motivation

Description of the changes

codecov-io commented Mar 20, 2021 • edited

Codecov Report

crcrpar commented Mar 21, 2021 • edited

keisuke-umezawa commented Mar 21, 2021

parsiad commented Mar 21, 2021 • edited

crcrpar commented Mar 22, 2021

parsiad commented Mar 22, 2021

Crissman commented Mar 23, 2021

Crissman left a comment

Choose a reason for hiding this comment

parsiad commented Mar 23, 2021

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura Mar 24, 2021

Choose a reason for hiding this comment

parsiad Mar 24, 2021 • edited

Choose a reason for hiding this comment

HideakiImamura Mar 24, 2021 • edited

Choose a reason for hiding this comment

parsiad Mar 24, 2021

Choose a reason for hiding this comment

g-votte Mar 24, 2021

Choose a reason for hiding this comment

parsiad Mar 24, 2021 • edited

Choose a reason for hiding this comment

HideakiImamura left a comment

Choose a reason for hiding this comment

codecov-io commented Mar 20, 2021 •

edited

crcrpar commented Mar 21, 2021 •

edited

parsiad commented Mar 21, 2021 •

edited

parsiad Mar 24, 2021 •

edited

HideakiImamura Mar 24, 2021 •

edited

parsiad Mar 24, 2021 •

edited