Benchmaking for multi-objectives samplers #3271

drumehiron · 2022-01-30T07:51:32Z

Motivation

#3245

Description of the changes

This PR is extend 'optuna/benchmarks/run_kurobako.py' to multi-objectives.

TODOs:

fix --sampler-kwargs-list argument

codecov-commenter · 2022-01-30T08:39:24Z

Codecov Report

Merging #3271 (0d7e03a) into master (2a208f0) will decrease coverage by 0.31%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3271      +/-   ##
==========================================
- Coverage   91.84%   91.53%   -0.32%     
==========================================
  Files         156      158       +2     
  Lines       12292    12195      -97     
==========================================
- Hits        11290    11163     -127     
- Misses       1002     1032      +30

Impacted Files	Coverage Δ
...a/visualization/matplotlib/_parallel_coordinate.py	`98.37% <0.00%> (-1.63%)`	⬇️
optuna/visualization/_pareto_front.py	`98.57% <0.00%> (-1.43%)`	⬇️
optuna/integration/chainermn.py	`93.93% <0.00%> (-1.22%)`	⬇️
optuna/_transform.py	`94.44% <0.00%> (-0.88%)`	⬇️
optuna/integration/botorch.py	`97.80% <0.00%> (-0.88%)`	⬇️
optuna/samplers/nsgaii/_sampler.py	`95.33% <0.00%> (-0.85%)`	⬇️
optuna/samplers/nsgaii/_crossover.py	`91.17% <0.00%> (-0.61%)`	⬇️
optuna/integration/tensorboard.py	`91.30% <0.00%> (-0.54%)`	⬇️
optuna/visualization/matplotlib/_utils.py	`95.65% <0.00%> (-0.51%)`	⬇️
optuna/integration/skopt.py	`98.08% <0.00%> (-0.29%)`	⬇️
... and 33 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

HideakiImamura · 2022-02-08T01:47:12Z

Thanks for the PR! I'd like to investigate how to use sampler-kwargs-list.

HideakiImamura

Thanks for the PR! I made some modifications to make it work with the latest Optuna version. Check it out!

benchmarks/mo_runner.py

benchmarks/run_mo_kurobako.py

HideakiImamura · 2022-02-10T09:40:00Z

@drumehiron PTAL!

drumehiron · 2022-02-16T04:36:10Z

Sorry for the late reply and thank you for review.
I check your review.

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

drumehiron · 2022-02-17T04:09:50Z

benchmarks/run_mo_kurobako.py

+
+    # Create study.
+    cmd = (
+        f"{kurobako_cmd} studies --budget 300 "


I have confirmed that it works well on my computer with --budget 10 .
In the case of -- budget 300, the computation time is too long, so I haven't checked if it works properly yet.

Agree. How about using the same budget with the existing single-objective one? It is 80. https://github.com/optuna/optuna/blob/master/benchmarks/run_kurobako.py#L69

HideakiImamura · 2022-02-24T02:52:45Z

You can fix the CI error of Checks by formatting.

$ bash formats.sh

…r_multi_objectives

drumehiron · 2022-04-05T14:49:10Z

@nzw0301 Thank you for your review. I fix the points.

HideakiImamura

Thanks for the update. I have several comment mainly to enable the sampler-kwargs option. Please take a look.

benchmarks/mo_create_study.py

benchmarks/run_mo_kurobako.py

HideakiImamura · 2022-04-12T12:13:16Z

benchmarks/run_mo_kurobako.py

+
+    os.makedirs(args.out_dir, exist_ok=True)
+    study_json_filename = os.path.join(args.out_dir, "studies.json")
+    subprocess.check_call(f"echo >| {study_json_filename}", shell=True)


Looks good not to use -n option since it does not work in macOS.

Thank you for checking it.

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

drumehiron

Thank you for review. I checked it.

drumehiron · 2022-04-16T02:07:15Z

benchmarks/run_mo_kurobako.py

+
+    os.makedirs(args.out_dir, exist_ok=True)
+    study_json_filename = os.path.join(args.out_dir, "studies.json")
+    subprocess.check_call(f"echo >| {study_json_filename}", shell=True)


Thank you for checking it.

drumehiron · 2022-04-16T09:49:05Z

I share the graphs obtained by running the program locally.

Results

NASBench(A)

ZDT1

ZDT2

ZDT3

ZDT4

ZDT5

ZDT6

HideakiImamura · 2022-04-18T03:02:48Z

@contramundum53 Could you review this PR if you have time?

HideakiImamura

Thanks for the update! Almost, LGTM. I have several minor comments. PTAL.

HideakiImamura · 2022-04-18T05:21:02Z

benchmarks/mo_create_study.py

@@ -0,0 +1,44 @@
+import json


The benchmarks directory becomes a bit complicated. How about creating a benchmarks/kurobako directory and moving mo_create_study.py into this?

I agree your comments. I move mo_create_study.py into into benchmarks/kurobako.

benchmarks/mo_create_study.py

benchmarks/run_mo_kurobako.py

contramundum53

Thanks for your PR! It looks almost good to me. I left some minor comments.

benchmarks/run_mo_kurobako.py

drumehiron

Thank you for review. I will update this PR.

drumehiron · 2022-04-19T11:26:53Z

benchmarks/mo_create_study.py

@@ -0,0 +1,44 @@
+import json


I agree your comments. I move mo_create_study.py into into benchmarks/kurobako.

benchmarks/mo_create_study.py

benchmarks/run_mo_kurobako.py

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com> Co-authored-by: contramundum53 <contramundum53@outlook.com>

drumehiron · 2022-04-25T14:50:05Z

I share the graphs obtained by running the program locally. NSGA-II seems to be behaving better.

NAS Bench

ZDT1

ZDT2

ZDT3

ZDT4

ZDT5

ZDT6

contramundum53

Thank you!

contramundum53 · 2022-04-28T07:00:36Z

Maybe we should also change the single-objective version to have budget 120 for consistency, but I think that can be another PR.

add benchmark for multi_objectives

437e011

nzw0301 added v3 Issue/PR for Optuna version 3. sprint-20220130 PR from the online sprint event Jan 30, 2022. labels Jan 30, 2022

update scripts

08f9591

HideakiImamura mentioned this pull request Feb 7, 2022

Introduce a rich benchmark environment in CI #3245

Closed

6 tasks

HideakiImamura self-assigned this Feb 7, 2022

HideakiImamura added the feature Change that does not break compatibility, but affects the public interfaces. label Feb 8, 2022

HideakiImamura reviewed Feb 10, 2022

View reviewed changes

drumehiron and others added 16 commits February 16, 2022 13:44

Update benchmarks/mo_runner.py

043cedf

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/mo_runner.py

c46bd10

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/mo_runner.py

c20b659

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/mo_runner.py

b4a2fa3

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/mo_runner.py

8341fdc

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

ae56e6e

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

67bf8b7

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

956c2c0

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

a01081e

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

5c3d3eb

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

8148501

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

34c9dee

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

ad08994

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

3a55295

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

fcd944b

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

update fix bug

4724687

drumehiron commented Feb 17, 2022

View reviewed changes

drumehiron added 2 commits April 5, 2022 23:44

Merge remote-tracking branch 'upstream/master' into fix/benchmarks_fo…

0fc9a99

…r_multi_objectives

fix nzw0301-san's points

a004d26

HideakiImamura reviewed Apr 12, 2022

View reviewed changes

drumehiron and others added 6 commits April 16, 2022 11:06

Update benchmarks/mo_create_study.py

d3e4e3b

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/mo_create_study.py

e900257

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

5f4eb44

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

5eca226

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

e4edb4d

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update benchmarks/run_mo_kurobako.py

5457413

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

drumehiron commented Apr 16, 2022

View reviewed changes

drumehiron added 3 commits April 16, 2022 11:40

format benchmarks/run_mo_kurobako.py

2f93ed2

update benchmarks/run_mo_kurobako.py

322f552

format benchmarks/run_mo_kurobako.py

533caf1

HideakiImamura assigned contramundum53 Apr 18, 2022

HideakiImamura reviewed Apr 18, 2022

View reviewed changes

contramundum53 reviewed Apr 18, 2022

View reviewed changes

benchmarks/run_mo_kurobako.py Outdated Show resolved Hide resolved

benchmarks/run_mo_kurobako.py Outdated Show resolved Hide resolved

benchmarks/run_mo_kurobako.py Outdated Show resolved Hide resolved

drumehiron commented Apr 19, 2022

View reviewed changes

Apply suggestions from code review

0d7e03a

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com> Co-authored-by: contramundum53 <contramundum53@outlook.com>

nzw0301 requested a review from a team April 19, 2022 12:05

himkt requested review from a team and removed request for a team April 19, 2022 12:25

change mo_create_study.py directory

85a5b8a

contramundum53 approved these changes Apr 28, 2022

View reviewed changes

contramundum53 merged commit 97f29ce into optuna:master Apr 28, 2022

knshnb added this to the v3.0.0-b1 milestone Jun 3, 2022

Benchmaking for multi-objectives samplers #3271

Benchmaking for multi-objectives samplers #3271

Conversation

drumehiron commented Jan 30, 2022 • edited Loading

Motivation

Description of the changes

TODOs:

codecov-commenter commented Jan 30, 2022 • edited Loading

Codecov Report

HideakiImamura commented Feb 8, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura commented Feb 10, 2022

drumehiron commented Feb 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HideakiImamura commented Feb 24, 2022

drumehiron commented Apr 5, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drumehiron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drumehiron commented Apr 16, 2022

NASBench(A)

ZDT1

ZDT2

ZDT3

ZDT4

ZDT5

ZDT6

HideakiImamura commented Apr 18, 2022

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

contramundum53 left a comment

Choose a reason for hiding this comment

drumehiron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drumehiron commented Apr 25, 2022

NAS Bench

ZDT1

ZDT2

ZDT3

ZDT4

ZDT5

ZDT6

contramundum53 left a comment

Choose a reason for hiding this comment

contramundum53 commented Apr 28, 2022

drumehiron commented Jan 30, 2022 •

edited

Loading

codecov-commenter commented Jan 30, 2022 •

edited

Loading