Add TensorBoard integration. #1244

VladSkripniuk · 2020-05-14T13:29:21Z

Description of the changes

This PR introduces optuna.integration.TensorBoardCallback. This callback saves values of objective function and parameters to HParams dashboard.

VladSkripniuk · 2020-05-14T13:33:41Z

To let Tensorboard know the range of values for parameters I use param_distributions parameter in TensorBoardCallback.__init__. I think we need to discuss if we can somehow pass the distribution from trial.suggest to the callback.

codecov-io · 2020-05-16T18:19:37Z

Codecov Report

Merging #1244 into master will decrease coverage by 0.09%.
The diff coverage is 73.46%.

@@            Coverage Diff             @@
##           master    #1244      +/-   ##
==========================================
- Coverage   85.97%   85.88%   -0.10%     
==========================================
  Files          92       93       +1     
  Lines        6751     6800      +49     
==========================================
+ Hits         5804     5840      +36     
- Misses        947      960      +13

Impacted Files	Coverage Δ
optuna/integration/tensorboard.py	`73.46% <73.46%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b6e303...36d38a5. Read the comment docs.

HideakiImamura

Thank you for the significant PR! I have some comments on optuna/integration/tensorboard.py.

In my opinion, it seems that specifying thee distribution classes in optuna/distributions confuses users, who are not familiar with Optuna deeply. How about letting the user simply specify the type of distribution and the information needed (low, high, choices, etc.) in the form of a tuple? It is sufficient to create the class of optuna/distributions internally in optuna/integration/tensorboard.py.

HideakiImamura · 2020-05-18T17:35:16Z

optuna/integration/tensorboard.py

+    Args:
+        metric_name:
+            Name of the metric. Since the metric itself is just a number,
+            `metric_name` can be used to give it a name. So you know later
+            if it was roc-auc or accuracy.


Please specify whole information of all arguments. (dirname and param_distributions)

optuna/integration/tensorboard.py

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

HideakiImamura

Thanks for the updates! I have a minor comment.

HideakiImamura · 2020-05-21T03:25:32Z

examples/tensorboard_simple.py

+def train_test_model(num_units, dropout_rate, optimizer):
+    # type: (int, float, str) -> float


We can use the new style for the type hinting. For example, please refer to here.

codecov-commenter · 2020-05-21T09:09:54Z

Codecov Report

Merging #1244 into master will increase coverage by 0.90%.
The diff coverage is 70.76%.

@@            Coverage Diff             @@
##           master    #1244      +/-   ##
==========================================
+ Coverage   85.97%   86.87%   +0.90%     
==========================================
  Files          92       95       +3     
  Lines        6751     7298     +547     
==========================================
+ Hits         5804     6340     +536     
- Misses        947      958      +11

Impacted Files	Coverage Δ
optuna/integration/tensorboard.py	`70.76% <70.76%> (ø)`
optuna/trial/_frozen.py	`81.25% <0.00%> (-3.75%)`	⬇️
optuna/integration/chainermn.py	`75.70% <0.00%> (-1.13%)`	⬇️
optuna/testing/storage.py	`82.35% <0.00%> (-0.51%)`	⬇️
optuna/storages/rdb/models.py	`95.19% <0.00%> (-0.44%)`	⬇️
optuna/storages/rdb/storage.py	`95.83% <0.00%> (-0.39%)`	⬇️
optuna/pruners/hyperband.py	`97.84% <0.00%> (ø)`
optuna/pruners/successive_halving.py	`93.58% <0.00%> (ø)`
optuna/multi_objective/samplers/__init__.py	`100.00% <0.00%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b6e303...3e3c4a3. Read the comment docs.

HideakiImamura

Thanks for the update! I have some minor comments.

HideakiImamura · 2020-05-21T17:39:20Z

examples/tensorboard_simple.py

+if type_checking.TYPE_CHECKING:
+    from typing import Dict  # NOQA
+    from typing import Any  # NOQA


In the new type hinting style, we can remove the type_checking, just import typing/Dict or Any outside if type_checking.TYPE_CHECKING: and remove NOQAs.

HideakiImamura · 2020-05-21T17:39:28Z

optuna/integration/tensorboard.py

+if type_checking.TYPE_CHECKING:
+    from typing import Any  # NOQA
+    from typing import Dict  # NOQA
+    from typing import Optional  # NOQA
+    from typing import Tuple  # NOQA


HideakiImamura · 2020-05-22T06:40:07Z

Hi @VladSkripniuk. I would like to discuss the user interface of TensorBoardCallback. I think that we can remove the param_distributions argument by automatically looking up the first finished trial. And it might be possible to add parameters by looking up finished trials automatically. What do you think?

HideakiImamura

Hi @VladSkripniuk! I'm sorry I left a vague comment. I'm going to comment specifically on what I was thinking. Could you take a look?

optuna/integration/tensorboard.py

examples/tensorboard_simple.py

tests/integration_tests/test_tensorboard.py

github-actions · 2020-06-22T23:08:38Z

This pull request has not seen any recent activity.

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

VladSkripniuk · 2020-06-26T13:56:44Z

Hi @HideakiImamura thank for your great help! I've applied proposed changes.

HideakiImamura

Thanks for the update! Basically, LGTM. I left some minor comments.

examples/tensorboard_simple.py

optuna/integration/tensorboard.py

tests/integration_tests/test_tensorboard.py

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

HideakiImamura

Thanks for your quick updates! I have other minor comments. Could you take a look?

examples/tensorboard_simple.py

tests/integration_tests/test_tensorboard.py

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

HideakiImamura

Thanks for your swift action! LGTM!

toshihikoyanase

LGTM. Thank you for your great work!

I think we'll make a followup PR:

Add experimental decorator
Filter out non-COMPLETE trials in __call__
Skip execution of the example in Python 3.8.

Add TensorBoard integration.

410e7a4

VladSkripniuk added 4 commits May 14, 2020 15:58

Fix mypy.

3a74fd3

Fix flake8.

55b8f09

Add TensorFlow for python 3.8.

48fb496

Fix black on setup.py.

36d38a5

HideakiImamura requested changes May 18, 2020

View reviewed changes

VladSkripniuk and others added 3 commits May 20, 2020 14:21

Update optuna/integration/tensorboard.py

73ca09b

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Specify distribution types as tuples.

300f62a

Fix CI failures.

b96631f

HideakiImamura reviewed May 21, 2020

View reviewed changes

Change type hinting to new style.

3e3c4a3

HideakiImamura reviewed May 21, 2020

View reviewed changes

HideakiImamura self-assigned this May 28, 2020

HideakiImamura requested changes Jun 7, 2020

View reviewed changes

github-actions bot added the stale Exempt from stale bot labeling. label Jun 22, 2020

VladSkripniuk and others added 10 commits June 25, 2020 19:16

Update optuna/integration/tensorboard.py

cd1ac10

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update optuna/integration/tensorboard.py

0358bf3

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update optuna/integration/tensorboard.py

b7aca19

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update examples/tensorboard_simple.py

d3196be

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update examples/tensorboard_simple.py

0910311

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update examples/tensorboard_simple.py

f623788

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

a2d2f7a

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

728d74a

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

d40a4f7

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Lazy initialization in TensorBoardCallback.

2b5a791

VladSkripniuk added 6 commits June 25, 2020 21:07

Merge master.

0396757

Remove old __call__ in TensorBoardCallback.

0617d3b

Fix black.

0f20e11

Skip TensorBoard tests for python 3.8.

1b191e4

Fix black.

bb37a86

Fix flake8.

79a3aa6

github-actions bot removed the stale Exempt from stale bot labeling. label Jun 28, 2020

HideakiImamura reviewed Jun 30, 2020

View reviewed changes

VladSkripniuk and others added 6 commits June 30, 2020 08:52

Update examples/tensorboard_simple.py

903dbbe

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update examples/tensorboard_simple.py

b471d62

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update optuna/integration/tensorboard.py

6382843

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

f43f4c6

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

0144c61

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update examples/tensorboard_simple.py

5ab96ed

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

HideakiImamura reviewed Jul 1, 2020

View reviewed changes

examples/tensorboard_simple.py Outdated Show resolved Hide resolved

tests/integration_tests/test_tensorboard.py Outdated Show resolved Hide resolved

tests/integration_tests/test_tensorboard.py Outdated Show resolved Hide resolved

VladSkripniuk and others added 3 commits July 1, 2020 10:40

Update examples/tensorboard_simple.py

9dba236

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

38a53b9

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

Update tests/integration_tests/test_tensorboard.py

b2aed84

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

HideakiImamura approved these changes Jul 1, 2020

View reviewed changes

toshihikoyanase added feature Change that does not break compatibility, but affects the public interfaces. optuna.integration Related to the `optuna.integration` submodule. This is automatically labeled by github-actions. labels Jul 2, 2020

toshihikoyanase approved these changes Jul 3, 2020

View reviewed changes

toshihikoyanase merged commit c927373 into optuna:master Jul 3, 2020

toshihikoyanase added this to the v2.0.0 milestone Jul 3, 2020

HideakiImamura mentioned this pull request Jul 3, 2020

Followup TensorBoard integration. #1475

Merged

toshihikoyanase modified the milestones: v2.0.0-rc0, v2.0.0 Jul 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TensorBoard integration. #1244

Add TensorBoard integration. #1244

VladSkripniuk commented May 14, 2020

VladSkripniuk commented May 14, 2020

codecov-io commented May 16, 2020

HideakiImamura left a comment

HideakiImamura May 18, 2020

HideakiImamura left a comment

HideakiImamura May 21, 2020

codecov-commenter commented May 21, 2020

HideakiImamura left a comment

HideakiImamura May 21, 2020

HideakiImamura May 21, 2020

HideakiImamura commented May 22, 2020

HideakiImamura left a comment

github-actions bot commented Jun 22, 2020

VladSkripniuk commented Jun 26, 2020

HideakiImamura left a comment

HideakiImamura left a comment

HideakiImamura left a comment

toshihikoyanase left a comment

		def train_test_model(num_units, dropout_rate, optimizer):
		# type: (int, float, str) -> float

Add TensorBoard integration. #1244

Add TensorBoard integration. #1244

Conversation

VladSkripniuk commented May 14, 2020

Description of the changes

VladSkripniuk commented May 14, 2020

codecov-io commented May 16, 2020

Codecov Report

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura May 18, 2020

Choose a reason for hiding this comment

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura May 21, 2020

Choose a reason for hiding this comment

codecov-commenter commented May 21, 2020

Codecov Report

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura May 21, 2020

Choose a reason for hiding this comment

HideakiImamura May 21, 2020

Choose a reason for hiding this comment

HideakiImamura commented May 22, 2020

HideakiImamura left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 22, 2020

VladSkripniuk commented Jun 26, 2020

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura left a comment

Choose a reason for hiding this comment

toshihikoyanase left a comment

Choose a reason for hiding this comment