Add HyperbandPruner

[crcrpar]

This depends on #808
The algorithm would behave differently from the paper if the sampler is TPESampler because there is no way to filter out the trials of brackets other than that of the running trial.

---

Although #785 and #805 are for Hyperband, this PR follows another design policy.
The core design of #785 is to realize filtering by adding an attribute that represents pruner to Trial.
It will make Study more complex while it can keep the current loosely connected relationship between Sampler and Pruner.

On the other hand, the policy of this PR does not implement any filtering of trials.
This means that we will need to add some interaction between HyperbandPruner and Sampler, especially TPESampler.

EDIT:
In this implementation, trials are filtered by bracket_id that is computed by HyperbandPruner._get_bracket_id and this filtering is enabled by implementing _BracketStudy which just wraps Study for the HyperbandPruner use only. The idea is suggested in #809 (comment).

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@2ebb157). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master     #809   +/-   ##
=========================================
  Coverage          ?   90.13%           
=========================================
  Files             ?      108           
  Lines             ?     8914           
  Branches          ?        0           
=========================================
  Hits              ?     8035           
  Misses            ?      879           
  Partials          ?        0

Impacted Files	Coverage Δ
optuna/pruners/base.py	62.5% <ø> (ø)
setup.py	0% <ø> (ø)
optuna/study.py	93.62% <100%> (ø)
optuna/pruners/hyperband.py	78.57% <100%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ebb157...7bc4874. Read the comment docs.

[sile]

Thank you for the swift update. The current design looks good for me.
By the way, I left some minor comments about the code detail.

[hvy]

Other methods besides get_trials might behaive unexpectedly as they are not filtered. I believe we can either override and raise, or avoid inheriting from Study.

[crcrpar]

I tried to disable other methods of Study in a naive way. PTAL.

[crcrpar]

As to n_brackets, following #809 (comment) and #809 (comment), I'd like to make it defaults to 4.

(This is the clarification as the discussion is getting long and not accessible.)

[c-bata]

LGTM with nits!! I think current implementation is reasonable for now 👍 Good job @crcrpar.

[crcrpar]

Discussed how to compute budgets and allocate trials to brackets with @sile, and one way is to merge this without any modification because

• currently, there doesn't seem to be any colossal problem like poor performance
• it might take a long time to design a reasonable algorithm and the PR for it might be not tiny

I don't mean we should stop here, merge first without further improvement. From the perspective of the granularity and duration of the PR, merging is one option.

[hvy]

LGTM!

[crcrpar]

Thank you all for the review and discussion. Though this PR satisfies the condition to get merged, I'd like to add TODO comment about resource allocation to brackets. Thank you for your understanding beforehand.

…

On Thu, Jan 9, 2020 at 18:10 Hiroyuki Vincent Yamazaki < ***@***.***> wrote: ***@***.**** approved this pull request. LGTM! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#809?email_source=notifications&email_token=AD3Q7U5FA7PXESFAHYK5ESTQ43SZ5A5CNFSM4J4IZEZKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCRE3OIY#pullrequestreview-340375331>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD3Q7UY55TF64IPMV3GGTOTQ43SZ5ANCNFSM4J4IZEZA> .

[crcrpar]

Added the TODO.

[sile]

Added the TODO.

Thanks!
I'll merge this PR.

[PhilipMay]

Thanks for this implementation! It's awsome! :-)