Add WFG benchmark test #3349

kei-mo · 2022-02-27T08:51:54Z

Motivation

Based on private discussion with @HideakiImamura and @hvy , we're going to add WFG multi-objective optimization benchmark test to benchmarks/run_mo_kurobako.py

Description of the changes

Confirm how to call a custom problem from benchmarks/run_mo_kurobako.py.
Create a kurobako WFG benchmark test by using @HideakiImamura 's code as a reference.
Test new benchmark with random sampler and/or TPE sampler.

…ectives' into add_wfg_benchmark

codecov-commenter · 2022-02-27T09:24:17Z

Codecov Report

Merging #3349 (24f5662) into master (2a208f0) will decrease coverage by 0.29%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3349      +/-   ##
==========================================
- Coverage   91.84%   91.55%   -0.30%     
==========================================
  Files         156      158       +2     
  Lines       12292    12226      -66     
==========================================
- Hits        11290    11193      -97     
- Misses       1002     1033      +31

Impacted Files	Coverage Δ
...a/visualization/matplotlib/_parallel_coordinate.py	`98.37% <0.00%> (-1.63%)`	⬇️
optuna/integration/wandb.py	`98.43% <0.00%> (-1.57%)`	⬇️
optuna/visualization/_pareto_front.py	`98.57% <0.00%> (-1.43%)`	⬇️
optuna/integration/chainermn.py	`93.93% <0.00%> (-1.22%)`	⬇️
optuna/_transform.py	`94.44% <0.00%> (-0.88%)`	⬇️
optuna/integration/botorch.py	`97.80% <0.00%> (-0.88%)`	⬇️
optuna/samplers/nsgaii/_sampler.py	`95.33% <0.00%> (-0.85%)`	⬇️
optuna/samplers/nsgaii/_crossover.py	`91.17% <0.00%> (-0.61%)`	⬇️
optuna/integration/tensorboard.py	`91.30% <0.00%> (-0.54%)`	⬇️
optuna/visualization/matplotlib/_utils.py	`95.65% <0.00%> (-0.51%)`	⬇️
... and 35 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

hvy · 2022-03-01T00:52:24Z

@kei-mo, thanks for the last discussion and swift PR. 🙇
Could you possibly add to the PR description what exact tasks remain, given its still work in progress?

@sile, not sure how busy you are but would you be able to review this change later on, as the author of Kurobako?
As for the background, it's documented in #3245.
In short, the usage of Kurobako has been somewhat streamlined recently https://github.com/optuna/optuna/blob/master/benchmarks/README.md, and we're now integrating multi-objective problems (this PR) and solvers (in addition to single-objective).

sile · 2022-03-01T03:33:02Z

I'm willing to review this PR. However, it would take some time because I'm a bit busy for a while. If you want to merge this PR within a few days or weeks, please re-assign to another person instead of me 🙏

hvy · 2022-03-01T03:56:55Z

I see, thanks for the quick response. @nzw0301 do you happen to be available?

kei-mo · 2022-03-01T12:22:39Z

@kei-mo, thanks for the last discussion and swift PR. bow Could you possibly add to the PR description what exact tasks remain, given its still work in progress?

@hvy , I've updated the PR description. I'm currently working on transferring @HideakiImamura san's WFG code to kurobako problem. I think this would take some time (maybe a week).

kei-mo · 2022-03-01T12:27:39Z

@HideakiImamura , if you have the script for running the WFG code, it would be very helpful. Could you share if you don't mind?

HideakiImamura · 2022-03-02T05:20:47Z

@kei-mo Thanks for the update! This is the usage of the WFG. Please take a look.

from WFGtestSuite import wfg

import numpy as np
import math


class WFG:
    def __init__(self, problem_name, n_var, n_obj=2, k=2):
        if problem_name == "wfg1":
            self.wfg = wfg.WFG1(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg2":
            self.wfg = wfg.WFG2(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg3":
            self.wfg = wfg.WFG3(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg4":
            self.wfg = wfg.WFG4(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg5":
            self.wfg = wfg.WFG5(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg6":
            self.wfg = wfg.WFG6(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg7":
            self.wfg = wfg.WFG7(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg8":
            self.wfg = wfg.WFG8(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg9":
            self.wfg = wfg.WFG9(n_arguments=n_var, n_objectives=n_obj, k=k)
        else:
            raise ValueError("not found", problem_name)
        self._low = 0
        self._high = 2
        self.n_var = n_var

    def objective(self, trial):
        x = np.array(
            [
                trial.suggest_float(f"x{i}", self._low, self._high * i)
                for i in range(1, self.n_var + 1)
            ]
        )
        v = self.wfg(x)
        if math.isnan(v[0]) or math.isinf(v[0]):
            raise ValueError
        if math.isnan(v[1]) or math.isinf(v[1]):
            raise ValueError
        return v[0], v[1]

objective_name = HOGE  # Set wfg1 ~ wfg9
dim = FUGA   # I think `dim` cam be 2, 3, 5, and 10.
if objective_name == "wfg2" or objective_name == "wfg3":
    if dim == 2 or dim == 3:
        raise ("error: dim:2,dim:3, wfg2, wfg3")
    elif dim == 5:
        wfg = WFG(objective_name, dim, k=1)
    else:
        wfg = WFG(objective_name, dim, k=2)
elif objective_name == "wfg7":
    wfg = WFG(objective_name, dim, k=1)
elif objective_name == "wfg8":
    if dim == 3:
        wfg = WFG(objective_name, dim, k=2)
    else:
        wfg = WFG(objective_name, dim, k=1)
elif objective_name == "wfg9":
    wfg = WFG(objective_name, dim, k=1)
else:
    if dim == 2:
        wfg = WFG(objective_name, dim, k=1)
    else:
        wfg = WFG(objective_name, dim, k=2)

HideakiImamura · 2022-03-02T07:20:52Z

By the way, you can pass the CI job Checks by running formats.sh.

HideakiImamura

Thanks for the PR! I have several early comments. Please take a look.

At first, how about renaming the WFGtestSuite directory to wfg? It it simple and easy to understand.

HideakiImamura · 2022-03-14T12:12:34Z

benchmarks/problems/WFGtestSuite/wfg.py

+from . import shape_functions
+from . import transformation_functions


Please use the absolute import.

Suggested change

from . import shape_functions

from . import transformation_functions

from benchmarks.problems.wfg import shape_functions

from benchmarks.problems.wfg import transformation_functions

HideakiImamura · 2022-03-14T12:13:17Z

benchmarks/problems/binh_and_korn_problem.py

@@ -0,0 +1,46 @@
+from kurobako import problem


I think we can remove this file from this PR.

HideakiImamura · 2022-03-14T12:14:11Z

benchmarks/problems/wfg_problem.py

@@ -0,0 +1,86 @@
+import math


Cannot we directly use wfg.py or fix it to use with kurobako? I think the codes of this file are redundant.

github-actions · 2022-03-28T23:05:15Z

This pull request has not seen any recent activity.

github-actions · 2022-04-12T23:05:29Z

This pull request has not seen any recent activity.

kei-mo · 2022-04-24T01:35:48Z

@HideakiImamura
I'm working on this task as a part of Optuna Sprint 6

…ehiron/optuna into add_wfg_benchmark

kei-mo · 2022-04-26T11:59:35Z

Here's what I got by running TPE on the WFG benchmarks.
@HideakiImamura -san could you take a look at these results?
Please let me know if you need any additional information.

pip list | grep kurobako

-> 0.2.1

python benchmarks/run_mo_kurobako.py \
          --path-to-kurobako <path-to-kurobako> \
          --name "performance-benchmarks" \
          --n-runs 1 \
          --n-jobs 4 \
          --sampler-list "RandomSampler TPESampler " \
          --sampler-kwargs-list "{} {}" \
          --seed 0 \
          --data-dir "." \
          --out-dir "out"

WFG1

WFG2

WFG3

WFG4

WFG5

WFG6

WFG7

WFG8

WFG9

github-actions · 2022-05-10T23:05:34Z

This pull request has not seen any recent activity.

HideakiImamura · 2022-05-17T09:48:56Z

The benchmark result basically looks good to me. It might be good to increate the number of trials like 300.

HideakiImamura · 2022-05-18T08:45:54Z

@kei-mo Since this PR has a very large number of lines and is expected to take some time to review, why not hold a mob review once to get an overview in real time?

kei-mo · 2022-05-19T14:46:11Z

@kei-mo Since this PR has a very large number of lines and is expected to take some time to review, why not hold a mob review once to get an overview in real time?

@HideakiImamura Sounds good to me!

HideakiImamura · 2022-05-20T05:40:19Z

@kei-mo Thanks! Please check discord to to a mob-review.

HideakiImamura

Thanks, LGTM. I have two memos.

HideakiImamura · 2022-05-26T08:06:11Z

benchmarks/run_mo_kurobako.py

+        if n_wfg == 8:
+            n_dim = 3
+            k = 2
+        elif n_wfg in (7, 8, 9):
+            n_dim = 2
+            k = 1
+        else:
+            n_dim = 10
+            k = 2


memo: investigate the appropriate setups

HideakiImamura · 2022-05-26T08:14:46Z

benchmarks/run_mo_kurobako.py

+        "WFG1": {"xmin": 2.7, "xmax": 3.05, "ymin": 4.7, "ymax": 5.05},
+        "WFG2": {"xmin": 2.0, "xmax": 2.8, "ymin": 3.0, "ymax": 4.8},
+        "WFG3": {"xmin": 2.0, "xmax": 2.8, "ymin": 3.0, "ymax": 4.8},
+        "WFG4": {"xmin": 2.0, "xmax": 3.0, "ymin": 0.0, "ymax": 3.6},
+        "WFG5": {"xmin": 2.0, "xmax": 3.0, "ymin": 2.5, "ymax": 5.0},
+        "WFG6": {"xmin": 2.0, "xmax": 3.0, "ymin": 3.4, "ymax": 5.0},
+        "WFG7": {"xmin": 2.0, "xmax": 3.0, "ymin": 4.0, "ymax": 5.0},
+        "WFG8": {"xmin": 2.0, "xmax": 3.0, "ymin": 3.4, "ymax": 5.0},
+        "WFG9": {"xmin": 2.0, "xmax": 3.0, "ymin": 0.0, "ymax": 5.0},


memo: These range may change depending on above n_dim.

sile

I couldn't review the detail of the implementations of WFG functions. But the overall code looks good to me 👍

kei-mo added 4 commits February 27, 2022 17:34

Merge remote-tracking branch 'drumehiron/fix/benchmarks_for_multi_obj…

d3b4a79

…ectives' into add_wfg_benchmark

changed relative path to run_mo_kurobako.py

64fe01b

add binh and korn problem

7fb6608

wip add wfg problem

a08d26f

HideakiImamura self-assigned this Feb 27, 2022

HideakiImamura added the sprint-20220227 PR from the online sprint event Feb 27, 2022. label Feb 27, 2022

hvy mentioned this pull request Feb 27, 2022

Benchmaking for multi-objectives samplers #3271

Merged

1 task

hvy assigned sile Mar 1, 2022

kei-mo added 3 commits March 1, 2022 22:45

reorganized problem directory

95d4a9f

formate

1844441

migrated wfg code from HideakiImamura's repo

ba1a5b9

kei-mo added 8 commits March 2, 2022 22:44

wip reorganized

a53872f

add wfg1

9b6686c

formatted

c8744d0

wip add wfg2~9

5dc4be5

formatted

81242dc

modified wfg1~9

6c4da1d

reoriganized

0af4a7f

formatted

f92ad3b

HideakiImamura reviewed Mar 14, 2022

View reviewed changes

github-actions bot added the stale Exempt from stale bot labeling. label Mar 28, 2022

github-actions bot added the stale Exempt from stale bot labeling. label Apr 12, 2022

kei-mo added 6 commits April 24, 2022 15:54

add type annotations

8610025

fixed code to match black format

9e4833e

fixed code to match mypy format

bffde92

fixed code to match black format

135851a

Merge branch 'fix/benchmarks_for_multi_objectives' of github.com:drum…

9100c75

…ehiron/optuna into add_wfg_benchmark

fixed code to match black format

24f5662

github-actions bot removed the stale Exempt from stale bot labeling. label Apr 24, 2022

github-actions bot added the stale Exempt from stale bot labeling. label May 10, 2022

github-actions bot removed the stale Exempt from stale bot labeling. label May 17, 2022

kei-mo added 4 commits May 26, 2022 17:00

Merge branch 'master' of github.com:optuna/optuna into add_wfg_benchmark

7d79823

merged master

2857a83

reformat

6b06b05

fixed mypy

b0ff609

HideakiImamura approved these changes May 26, 2022

View reviewed changes

sile approved these changes May 29, 2022

View reviewed changes

sile merged commit 9618a93 into optuna:master May 29, 2022

toshihikoyanase added the other Issue/PR that does not fit into any other label. label May 30, 2022

HideakiImamura added CI Continuous integration. v3 Issue/PR for Optuna version 3. and removed other Issue/PR that does not fit into any other label. labels May 31, 2022

HideakiImamura added this to the v3.0.0-b1 milestone May 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add WFG benchmark test #3349

Add WFG benchmark test #3349

kei-mo commented Feb 27, 2022 •

edited by HideakiImamura

codecov-commenter commented Feb 27, 2022 •

edited

hvy commented Mar 1, 2022

sile commented Mar 1, 2022

hvy commented Mar 1, 2022

kei-mo commented Mar 1, 2022

kei-mo commented Mar 1, 2022

HideakiImamura commented Mar 2, 2022 •

edited

HideakiImamura commented Mar 2, 2022

HideakiImamura left a comment

HideakiImamura Mar 14, 2022

HideakiImamura Mar 14, 2022

HideakiImamura Mar 14, 2022

github-actions bot commented Mar 28, 2022

github-actions bot commented Apr 12, 2022

kei-mo commented Apr 24, 2022

kei-mo commented Apr 26, 2022 •

edited

github-actions bot commented May 10, 2022

HideakiImamura commented May 17, 2022

HideakiImamura commented May 18, 2022 •

edited

kei-mo commented May 19, 2022

HideakiImamura commented May 20, 2022

HideakiImamura left a comment

HideakiImamura May 26, 2022

HideakiImamura May 26, 2022

sile left a comment

		from . import shape_functions
		from . import transformation_functions

Add WFG benchmark test #3349

Add WFG benchmark test #3349

Conversation

kei-mo commented Feb 27, 2022 • edited by HideakiImamura

Motivation

Description of the changes

codecov-commenter commented Feb 27, 2022 • edited

Codecov Report

hvy commented Mar 1, 2022

sile commented Mar 1, 2022

hvy commented Mar 1, 2022

kei-mo commented Mar 1, 2022

kei-mo commented Mar 1, 2022

HideakiImamura commented Mar 2, 2022 • edited

HideakiImamura commented Mar 2, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura Mar 14, 2022

Choose a reason for hiding this comment

HideakiImamura Mar 14, 2022

Choose a reason for hiding this comment

HideakiImamura Mar 14, 2022

Choose a reason for hiding this comment

github-actions bot commented Mar 28, 2022

github-actions bot commented Apr 12, 2022

kei-mo commented Apr 24, 2022

kei-mo commented Apr 26, 2022 • edited

github-actions bot commented May 10, 2022

HideakiImamura commented May 17, 2022

HideakiImamura commented May 18, 2022 • edited

kei-mo commented May 19, 2022

HideakiImamura commented May 20, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura May 26, 2022

Choose a reason for hiding this comment

HideakiImamura May 26, 2022

Choose a reason for hiding this comment

sile left a comment

Choose a reason for hiding this comment

kei-mo commented Feb 27, 2022 •

edited by HideakiImamura

codecov-commenter commented Feb 27, 2022 •

edited

HideakiImamura commented Mar 2, 2022 •

edited

kei-mo commented Apr 26, 2022 •

edited

HideakiImamura commented May 18, 2022 •

edited