Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make `TPESampler` pruning aware. #439

Merged
merged 15 commits into from Jul 18, 2019
Merged

Make `TPESampler` pruning aware. #439

merged 15 commits into from Jul 18, 2019

Conversation

@sile
Copy link
Collaborator

sile commented Jun 25, 2019

This PR makes TPESampler pruning aware.
More specifically, this PR changes TPESampler to account for the pruned steps and intermediate values of pruned trials in addition to complete ones.

The following benchmark results show that this modification can dramatically improve optimization performance when TPESampler is used with pruning.

(1) Benchmark Results

Please refer to Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization for the datasets and search space used by this benchmarking.

(1-1) Pruning Benchmark

result0 result1

result2result3

Ranking

$ cat *.json | grep -v noprune | kurobako stats ranking

Sampler Borda Firsts
(a) optuna#master:minimize:asha 4 0
(b) optuna#master:minimize:median 0 0
(c) optuna#pruning-aware-tpe:minimize:asha 9 4
(d) optuna#pruning-aware-tpe:minimize:median 8 3

(1-2) No Pruning Benchmark

Ranking

noprune-0noprune-1

noprune-2 noprune-3

$ cat *.json | grep noprune | kurobako stats ranking

Samplers Borda Firsts
(a) optuna#master:maximize:noprune 0 4
(b) optuna#master:minimize:noprune 0 3
(c) optuna#pruning-aware-tpe:maximize:noprune 1 4
(d) optuna#pruning-aware-tpe:minimize:noprune 0 4

In no pruning cases, there are no statistically significant performance degradations by this modification.

(2) Benchmark Commands

// Install packages
$ cargo install kurobako
$ kurobako --version
kurobako 0.0.14

$ pip install git+http://github.com/sile/kurobako-py.git
$ pip install git+http://github.com/pfnet/optuna.git@pruning-aware-tpe2

// Download datasets
$ wget http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz

// Run benchmarks
$ kurobako benchmark --problems $(kurobako problem-suite fc-net fcnet_tabular_benchmarks) --solvers $(kurobako solver --tag pruning-aware-tpe:minimize:asha optuna --pruner asha --asha-min-resource 3 --asha-reduction-factor 3) --budget 50 --iterations 40 --evaluation-points 3 9 27 81 100 | kurobako run > pruning-aware-tpe-minimize-asha.json
$ kurobako benchmark --problems $(kurobako problem-suite fc-net fcnet_tabular_benchmarks) --solvers $(kurobako solver --tag master:minimize:median optuna) --budget 50 --iterations 40 --evaluation-points 10 20 30 40 50 60 70 80 90 100 | kurobako run > master-minimize-median.json
$ kurobako benchmark --problems $(kurobako problem-suite fc-net fcnet_tabular_benchmarks) --solvers $(kurobako solver --tag pruning-aware-tpe:maximize:noprune optuna --maximize) --budget 50 --iterations 40 --evaluation-points 100 | kurobako run > pruning-aware-tpe-maximize-noprune.json
$ kurobako benchmark --problems $(kurobako problem-suite fc-net fcnet_tabular_benchmarks) --solvers $(kurobako solver --tag pruning-aware-tpe:maximize:noprune optuna) --budget 50 --iterations 40 --evaluation-points 100 | kurobako run > pruning-aware-tpe-minimize-noprune.json

// Plot results
$ cat *.json | grep -v noprune | kurobako select --problems fcnet_slice_localization_data | kurobako plot -o /tmp/plot-result/ --ymin 0.0 --ymax 0.002
$ cat *.json | grep -v noprune | kurobako select --problems fcnet_protein_structure_data | kurobako plot -o /tmp/plot-result/ --ymin 0.2 --ymax 0.4
$ cat *.json | grep -v noprune | kurobako select --problems fcnet_parkinsons_telemonitoring_data | kurobako plot -o /tmp/plot-result/ --ymin 0 --ymax 0.05
$ cat *.json | grep -v noprune | kurobako select --problems fcnet_naval_propulsion_data | kurobako plot -o /tmp/plot-result/ --ymin 0 --ymax 0.002
@sile sile force-pushed the pruning-aware-tpe2 branch 3 times, most recently from e531e64 to c94725e Jun 27, 2019
@sile sile force-pushed the pruning-aware-tpe2 branch from c94725e to 8a810ea Jun 27, 2019
@sile sile marked this pull request as ready for review Jun 28, 2019
Copy link
Collaborator

toshihikoyanase left a comment

I think it is helpful for the future development of TPESampler if you add an explanation of _get_observation_pair. In my understanding, it returns the (param_value, (step, value)) to enable secondary sorting with keys = (step, value). Because of this change, TPESampler can use the running trials if it does not have enough complete trials.

optuna/storages/base.py Show resolved Hide resolved
@sile

This comment has been minimized.

Copy link
Collaborator Author

sile commented Jul 2, 2019

@toshihikoyanase

I think it is helpful for the future development of TPESampler if you add an explanation of _get_observation_pair.

Thank you for your suggestion. I added documentation at the commit 3ba0d19.

@sile sile added the enhancement label Jul 9, 2019
@sile sile added this to the v0.14.0 milestone Jul 9, 2019
Copy link
Collaborator

toshihikoyanase left a comment

LGTM

Copy link
Collaborator

g-votte left a comment

LGTM

@g-votte g-votte merged commit 7073fb0 into master Jul 18, 2019
7 checks passed
7 checks passed
ci/circleci: checks Your tests passed on CircleCI!
Details
ci/circleci: codecov Your tests passed on CircleCI!
Details
ci/circleci: document Your tests passed on CircleCI!
Details
ci/circleci: tests-python27 Your tests passed on CircleCI!
Details
ci/circleci: tests-python35 Your tests passed on CircleCI!
Details
ci/circleci: tests-python36 Your tests passed on CircleCI!
Details
ci/circleci: tests-python37 Your tests passed on CircleCI!
Details
@g-votte g-votte deleted the pruning-aware-tpe2 branch Jul 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.