Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timeline plot with plotly as backend #4470

Merged
merged 32 commits into from
Mar 23, 2023

Conversation

eukaryo
Copy link
Collaborator

@eukaryo eukaryo commented Feb 27, 2023

Motivation

To provide an easy way to check the execution status, I want to implement a function to plot the Trials as a timeline.
The timeline here is a Gantt chart with time on the horizontal axis and trial number on the vertical axis, which makes it clear at a glance whether trials are being executed in parallel and whether there are any trials with abnormally long execution times.

Description of the changes

  • Add timeline plot using plotly's horizontal bar graph in optuna/visualization/_timeline.py.

  • Add Unit tests in tests/visualization_tests/test_timeline.py.

test-fig-plotly

@github-actions github-actions bot added the optuna.visualization Related to the `optuna.visualization` submodule. This is automatically labeled by github-actions. label Feb 27, 2023
@eukaryo eukaryo added the feature Change that does not break compatibility, but affects the public interfaces. label Feb 27, 2023
@toshihikoyanase toshihikoyanase self-assigned this Feb 27, 2023
@toshihikoyanase
Copy link
Member

Thank you for your PR! Let me check it.
If possible, could you add a figure of a example plot to the description, please?

@eukaryo
Copy link
Collaborator Author

eukaryo commented Feb 27, 2023

I see. I have added a figure of timeline plot to the description above. Here is sample code to generate the figure, which is basically the same code as in the docstring of optuna.visualization.plot_timeline.

import time
import plotly
import optuna

def objective(trial):
    x = trial.suggest_float("x", 0, 1)
    time.sleep(x * 0.1)
    if x > 0.8:
        raise ValueError()
    if x > 0.4:
        raise optuna.TrialPruned()
    return x**2

study = optuna.create_study(direction="minimize")
study.enqueue_trial({"x": 0.3})
study.enqueue_trial({"x": 0.5})
study.enqueue_trial({"x": 0.9})
study.optimize(objective, n_trials=50, n_jobs=2, catch=(ValueError,))

ax = optuna.visualization.plot_timeline(study)
plotly.io.write_image(ax, "test-fig-plotly.png", format="png")

@HideakiImamura HideakiImamura self-assigned this Mar 6, 2023
Copy link
Member

@HideakiImamura HideakiImamura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I have several comments. Please take a look.

optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
optuna/visualization/_timeline.py Show resolved Hide resolved
optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
@codecov-commenter
Copy link

codecov-commenter commented Mar 9, 2023

Codecov Report

Merging #4470 (19750bd) into master (65558b7) will increase coverage by 0.02%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##           master    #4470      +/-   ##
==========================================
+ Coverage   90.32%   90.35%   +0.02%     
==========================================
  Files         183      184       +1     
  Lines       14142    14191      +49     
==========================================
+ Hits        12774    12822      +48     
- Misses       1368     1369       +1     
Impacted Files Coverage Δ
optuna/visualization/__init__.py 100.00% <100.00%> (ø)
optuna/visualization/_pareto_front.py 97.08% <100.00%> (-0.22%) ⬇️
optuna/visualization/_timeline.py 100.00% <100.00%> (ø)
optuna/visualization/_utils.py 95.55% <100.00%> (+0.68%) ⬆️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@toshihikoyanase toshihikoyanase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your PR and example. Let me share some comments on the implementation of the timeline plot.
Next, I'll check the test cases.

optuna/visualization/_utils.py Show resolved Hide resolved
optuna/visualization/_pareto_front.py Outdated Show resolved Hide resolved
optuna/visualization/_timeline.py Show resolved Hide resolved
optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
Copy link
Member

@toshihikoyanase toshihikoyanase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me share my comments on the tests.

tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
Copy link
Member

@toshihikoyanase toshihikoyanase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your update. Overall, it looks almost good to me, but I have some additional comments.

tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
tests/visualization_tests/test_timeline.py Outdated Show resolved Hide resolved
Copy link
Member

@HideakiImamura HideakiImamura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Basically, LGTM. I have 2 minor comments. PTAL.

optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
optuna/visualization/_timeline.py Outdated Show resolved Hide resolved
Co-authored-by: Toshihiko Yanase <toshihiko.yanase@gmail.com>
Copy link
Member

@toshihikoyanase toshihikoyanase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your update. LGTM!

@toshihikoyanase toshihikoyanase removed their assignment Mar 17, 2023
Copy link
Member

@HideakiImamura HideakiImamura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! LGTM!

@HideakiImamura HideakiImamura merged commit b410c00 into optuna:master Mar 23, 2023
@HideakiImamura HideakiImamura added this to the v3.2.0 milestone Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Change that does not break compatibility, but affects the public interfaces. optuna.visualization Related to the `optuna.visualization` submodule. This is automatically labeled by github-actions.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants