Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WFG benchmark test #3349

Merged
merged 29 commits into from
May 29, 2022
Merged

Add WFG benchmark test #3349

merged 29 commits into from
May 29, 2022

Conversation

kei-mo
Copy link
Contributor

@kei-mo kei-mo commented Feb 27, 2022

Motivation

Based on private discussion with @HideakiImamura and @hvy , we're going to add WFG multi-objective optimization benchmark test to benchmarks/run_mo_kurobako.py

Description of the changes

  • Confirm how to call a custom problem from benchmarks/run_mo_kurobako.py.
  • Create a kurobako WFG benchmark test by using @HideakiImamura 's code as a reference.
  • Test new benchmark with random sampler and/or TPE sampler.

@HideakiImamura HideakiImamura self-assigned this Feb 27, 2022
@HideakiImamura HideakiImamura added the sprint-20220227 PR from the online sprint event Feb 27, 2022. label Feb 27, 2022
@codecov-commenter
Copy link

codecov-commenter commented Feb 27, 2022

Codecov Report

Merging #3349 (24f5662) into master (2a208f0) will decrease coverage by 0.29%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3349      +/-   ##
==========================================
- Coverage   91.84%   91.55%   -0.30%     
==========================================
  Files         156      158       +2     
  Lines       12292    12226      -66     
==========================================
- Hits        11290    11193      -97     
- Misses       1002     1033      +31     
Impacted Files Coverage Δ
...a/visualization/matplotlib/_parallel_coordinate.py 98.37% <0.00%> (-1.63%) ⬇️
optuna/integration/wandb.py 98.43% <0.00%> (-1.57%) ⬇️
optuna/visualization/_pareto_front.py 98.57% <0.00%> (-1.43%) ⬇️
optuna/integration/chainermn.py 93.93% <0.00%> (-1.22%) ⬇️
optuna/_transform.py 94.44% <0.00%> (-0.88%) ⬇️
optuna/integration/botorch.py 97.80% <0.00%> (-0.88%) ⬇️
optuna/samplers/nsgaii/_sampler.py 95.33% <0.00%> (-0.85%) ⬇️
optuna/samplers/nsgaii/_crossover.py 91.17% <0.00%> (-0.61%) ⬇️
optuna/integration/tensorboard.py 91.30% <0.00%> (-0.54%) ⬇️
optuna/visualization/matplotlib/_utils.py 95.65% <0.00%> (-0.51%) ⬇️
... and 35 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@hvy hvy assigned sile Mar 1, 2022
@hvy
Copy link
Member

hvy commented Mar 1, 2022

@kei-mo, thanks for the last discussion and swift PR. 🙇
Could you possibly add to the PR description what exact tasks remain, given its still work in progress?

@sile, not sure how busy you are but would you be able to review this change later on, as the author of Kurobako?
As for the background, it's documented in #3245.
In short, the usage of Kurobako has been somewhat streamlined recently https://github.com/optuna/optuna/blob/master/benchmarks/README.md, and we're now integrating multi-objective problems (this PR) and solvers (in addition to single-objective).

@sile
Copy link
Member

sile commented Mar 1, 2022

I'm willing to review this PR. However, it would take some time because I'm a bit busy for a while. If you want to merge this PR within a few days or weeks, please re-assign to another person instead of me 🙏

@hvy
Copy link
Member

hvy commented Mar 1, 2022

I see, thanks for the quick response. @nzw0301 do you happen to be available?

@kei-mo
Copy link
Contributor Author

kei-mo commented Mar 1, 2022

@kei-mo, thanks for the last discussion and swift PR. bow Could you possibly add to the PR description what exact tasks remain, given its still work in progress?

@hvy , I've updated the PR description. I'm currently working on transferring @HideakiImamura san's WFG code to kurobako problem. I think this would take some time (maybe a week).

@kei-mo
Copy link
Contributor Author

kei-mo commented Mar 1, 2022

@HideakiImamura , if you have the script for running the WFG code, it would be very helpful. Could you share if you don't mind?

@HideakiImamura
Copy link
Member

HideakiImamura commented Mar 2, 2022

@kei-mo Thanks for the update! This is the usage of the WFG. Please take a look.

from WFGtestSuite import wfg

import numpy as np
import math


class WFG:
    def __init__(self, problem_name, n_var, n_obj=2, k=2):
        if problem_name == "wfg1":
            self.wfg = wfg.WFG1(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg2":
            self.wfg = wfg.WFG2(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg3":
            self.wfg = wfg.WFG3(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg4":
            self.wfg = wfg.WFG4(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg5":
            self.wfg = wfg.WFG5(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg6":
            self.wfg = wfg.WFG6(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg7":
            self.wfg = wfg.WFG7(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg8":
            self.wfg = wfg.WFG8(n_arguments=n_var, n_objectives=n_obj, k=k)
        elif problem_name == "wfg9":
            self.wfg = wfg.WFG9(n_arguments=n_var, n_objectives=n_obj, k=k)
        else:
            raise ValueError("not found", problem_name)
        self._low = 0
        self._high = 2
        self.n_var = n_var

    def objective(self, trial):
        x = np.array(
            [
                trial.suggest_float(f"x{i}", self._low, self._high * i)
                for i in range(1, self.n_var + 1)
            ]
        )
        v = self.wfg(x)
        if math.isnan(v[0]) or math.isinf(v[0]):
            raise ValueError
        if math.isnan(v[1]) or math.isinf(v[1]):
            raise ValueError
        return v[0], v[1]

objective_name = HOGE  # Set wfg1 ~ wfg9
dim = FUGA   # I think `dim` cam be 2, 3, 5, and 10.
if objective_name == "wfg2" or objective_name == "wfg3":
    if dim == 2 or dim == 3:
        raise ("error: dim:2,dim:3, wfg2, wfg3")
    elif dim == 5:
        wfg = WFG(objective_name, dim, k=1)
    else:
        wfg = WFG(objective_name, dim, k=2)
elif objective_name == "wfg7":
    wfg = WFG(objective_name, dim, k=1)
elif objective_name == "wfg8":
    if dim == 3:
        wfg = WFG(objective_name, dim, k=2)
    else:
        wfg = WFG(objective_name, dim, k=1)
elif objective_name == "wfg9":
    wfg = WFG(objective_name, dim, k=1)
else:
    if dim == 2:
        wfg = WFG(objective_name, dim, k=1)
    else:
        wfg = WFG(objective_name, dim, k=2)

@HideakiImamura
Copy link
Member

By the way, you can pass the CI job Checks by running formats.sh.

Copy link
Member

@HideakiImamura HideakiImamura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I have several early comments. Please take a look.

At first, how about renaming the WFGtestSuite directory to wfg? It it simple and easy to understand.

Comment on lines 5 to 6
from . import shape_functions
from . import transformation_functions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use the absolute import.

Suggested change
from . import shape_functions
from . import transformation_functions
from benchmarks.problems.wfg import shape_functions
from benchmarks.problems.wfg import transformation_functions

@@ -0,0 +1,46 @@
from kurobako import problem
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove this file from this PR.

@@ -0,0 +1,86 @@
import math
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cannot we directly use wfg.py or fix it to use with kurobako? I think the codes of this file are redundant.

@github-actions
Copy link
Contributor

This pull request has not seen any recent activity.

@github-actions github-actions bot added the stale Exempt from stale bot labeling. label Mar 28, 2022
@github-actions
Copy link
Contributor

This pull request has not seen any recent activity.

@github-actions github-actions bot added the stale Exempt from stale bot labeling. label Apr 12, 2022
@kei-mo
Copy link
Contributor Author

kei-mo commented Apr 24, 2022

@HideakiImamura
I'm working on this task as a part of Optuna Sprint 6

@github-actions github-actions bot removed the stale Exempt from stale bot labeling. label Apr 24, 2022
@kei-mo
Copy link
Contributor Author

kei-mo commented Apr 26, 2022

Here's what I got by running TPE on the WFG benchmarks.
@HideakiImamura -san could you take a look at these results?
Please let me know if you need any additional information.

pip list | grep kurobako

-> 0.2.1

python benchmarks/run_mo_kurobako.py \
          --path-to-kurobako <path-to-kurobako> \
          --name "performance-benchmarks" \
          --n-runs 1 \
          --n-jobs 4 \
          --sampler-list "RandomSampler TPESampler " \
          --sampler-kwargs-list "{} {}" \
          --seed 0 \
          --data-dir "." \
          --out-dir "out"

WFG1
wfg1-performance-benchmarks-tpesampler-84645d9a0fdbcf6cff7bfd9f753086931eab1917e807169b73fb83d391eaa7e9

WFG2
wfg2-performance-benchmarks-tpesampler-4c87d66fb8befaeb65724cfff5b2ebd716d89db68d1789aa6a0e805072e94b37

WFG3
wfg3-performance-benchmarks-tpesampler-1c15c96150a573bf04f022589ddbd59e6cea49a252ad07aebc266f5c0fa834ad

WFG4
wfg4-performance-benchmarks-tpesampler-953facdda1af6dd206ce3a45d8714f24ad588b7caf46b98ffcace351160266f5

WFG5
wfg5-performance-benchmarks-tpesampler-9a7eddcd2b60a7de41aade998b90a5dc9f155da8c379762fe67490c10ec21629

WFG6
wfg6-performance-benchmarks-tpesampler-bf249f43ae50ae142375417d1cb0db7574434d3896a2714a48afcbe1d70342e2

WFG7
wfg7-performance-benchmarks-tpesampler-c6a6cbac7ba0f41075190b81713a6f142945b29112509f04a6d4c88f3d774485

WFG8
wfg8-performance-benchmarks-tpesampler-4b29fc819549293ae1af39767dd76f171a3263e77f75f7a9e79abe62d07af881

WFG9
wfg9-performance-benchmarks-tpesampler-41e17c29f942fd00c8c753f93dfb2d250538070673b774a6862a4ae9fa143145

@github-actions
Copy link
Contributor

This pull request has not seen any recent activity.

@github-actions github-actions bot added the stale Exempt from stale bot labeling. label May 10, 2022
@HideakiImamura
Copy link
Member

The benchmark result basically looks good to me. It might be good to increate the number of trials like 300.

@github-actions github-actions bot removed the stale Exempt from stale bot labeling. label May 17, 2022
@HideakiImamura
Copy link
Member

HideakiImamura commented May 18, 2022

@kei-mo Since this PR has a very large number of lines and is expected to take some time to review, why not hold a mob review once to get an overview in real time?

@kei-mo
Copy link
Contributor Author

kei-mo commented May 19, 2022

@kei-mo Since this PR has a very large number of lines and is expected to take some time to review, why not hold a mob review once to get an overview in real time?

@HideakiImamura Sounds good to me!

@HideakiImamura
Copy link
Member

@kei-mo Thanks! Please check discord to to a mob-review.

Copy link
Member

@HideakiImamura HideakiImamura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I have two memos.

Comment on lines 31 to 39
if n_wfg == 8:
n_dim = 3
k = 2
elif n_wfg in (7, 8, 9):
n_dim = 2
k = 1
else:
n_dim = 10
k = 2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

memo: investigate the appropriate setups

Comment on lines +109 to +117
"WFG1": {"xmin": 2.7, "xmax": 3.05, "ymin": 4.7, "ymax": 5.05},
"WFG2": {"xmin": 2.0, "xmax": 2.8, "ymin": 3.0, "ymax": 4.8},
"WFG3": {"xmin": 2.0, "xmax": 2.8, "ymin": 3.0, "ymax": 4.8},
"WFG4": {"xmin": 2.0, "xmax": 3.0, "ymin": 0.0, "ymax": 3.6},
"WFG5": {"xmin": 2.0, "xmax": 3.0, "ymin": 2.5, "ymax": 5.0},
"WFG6": {"xmin": 2.0, "xmax": 3.0, "ymin": 3.4, "ymax": 5.0},
"WFG7": {"xmin": 2.0, "xmax": 3.0, "ymin": 4.0, "ymax": 5.0},
"WFG8": {"xmin": 2.0, "xmax": 3.0, "ymin": 3.4, "ymax": 5.0},
"WFG9": {"xmin": 2.0, "xmax": 3.0, "ymin": 0.0, "ymax": 5.0},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

memo: These range may change depending on above n_dim.

Copy link
Member

@sile sile left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't review the detail of the implementations of WFG functions. But the overall code looks good to me 👍

@sile sile merged commit 9618a93 into optuna:master May 29, 2022
@toshihikoyanase toshihikoyanase added the other Issue/PR that does not fit into any other label. label May 30, 2022
@HideakiImamura HideakiImamura added CI Continuous integration. v3 Issue/PR for Optuna version 3. and removed other Issue/PR that does not fit into any other label. labels May 31, 2022
@HideakiImamura HideakiImamura added this to the v3.0.0-b1 milestone May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous integration. sprint-20220227 PR from the online sprint event Feb 27, 2022. v3 Issue/PR for Optuna version 3.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants