-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use callback to report results to MLflow. #799
Use callback to report results to MLflow. #799
Conversation
@toshihikoyanase The script looks much simpler now. Thanks! |
examples/mlflow/keras_mlflow.py
Outdated
|
||
|
||
if __name__ == '__main__': | ||
study = optuna.create_study(direction='minimize') | ||
study.optimize(objective, n_trials=100, timeout=600) | ||
study = optuna.create_study(study_name='keras-wine', direction='minimize') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think 'study_name' argument can be removed since this example doesn't use an RDB storage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your comment.
I set it to show somewhat meaningful names rather than no-name-xxx
in MLflow UI. But we can distinguish studies if we remove the name. So I'll remove it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
This is just a thought. Current default study names are not human-friendly, so it might worth to improve the name generation logic in the future (e.g., adopt the naming rule being used by Docker).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with you. I posted an issue about the name generation as #803. Thank you.
By the way, the following part of my comment #799 (comment) is not true because all studies created with InMemoryStorage
have the same default name (i.e., no-name-00000000-0000-0000-0000-000000000000
). I'm sorry to give you a wrong information.
But we can distinguish studies if we remove the name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I posted an issue about the name generation as #803.
Thanks!
all studies created with InMemoryStorage have the same default name (i.e., no-name-00000000-0000-0000-0000-000000000000).
Oops. That is, there is no actual merit to use the default name...?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's right. Maybe storage classes should share the name generator instead of adding name generation code for InMemoryStorage
to this example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Codecov Report
@@ Coverage Diff @@
## master #799 +/- ##
=======================================
Coverage 90.15% 90.15%
=======================================
Files 106 106
Lines 8769 8769
=======================================
Hits 7906 7906
Misses 863 863 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
This PR uses a callback function to report optimization results to MLflow.
In this PR, I used
study_name
asrun_name
of MLflow, but I'm not fully confident if it is a relevant usage ofrun_name
. Please let me know if you have any ideas about it.