[tune] Custom Logging, Trial Name #3465

richardliaw · 2018-12-04T19:52:47Z

TODO:

Add tests for custom sync
Add test for custom logger

This should address #3034, #2985, and #3390.

AmplabJenkins · 2018-12-04T20:11:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9745/
Test FAILed.

hartikainen · 2018-12-06T05:27:25Z

python/ray/tune/logger.py

+        custom_loggers (list): List of custom logger creators.
+        sync_cmd_tmpl (str): Optional template for syncer to run. Needs to
+            include replacement fields "{local_dir}" and "{remote_dir}".
+            See ray/python/ray/tune/log_sync.py


Nice! I've been thinking that it would also be nice to have something similar for the trial logs folder names. Currently, many of my grid_search values include tuples etc., which tend to clutter the log dirs, while I usually only care about the id of the trial.

Hey @hartikainen, can you leave some feedback in #3034? The sentiment there is similar, but it would be great to get some consolidated feedback before implementing.

AmplabJenkins · 2018-12-07T01:55:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9828/
Test PASSed.

AmplabJenkins · 2018-12-07T03:58:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9830/
Test FAILed.

AmplabJenkins · 2018-12-07T22:44:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9839/
Test FAILed.

richardliaw · 2018-12-07T23:44:13Z

cc @llan-ml, @hartikainen

This addresses issues and requests you've made. It would be really great if you could take a glance at this PR!

AmplabJenkins · 2018-12-07T23:51:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9840/
Test FAILed.

ericl · 2018-12-07T23:48:50Z

python/ray/tune/examples/logging_example.py

+
+
+def trial_str_creator(trainable_name, trial_id, config):
+    return "{}_{}_123".format(trainable_name, trial_id)


How about we just pass the trial object here?

ericl · 2018-12-07T23:49:40Z

python/ray/tune/config_parser.py

@@ -106,6 +106,21 @@ def make_parser(parser_creator=None, **kwargs):
        default="",
        type=str,
        help="Optional URI to sync training results to (e.g. s3://bucket).")
+    parser.add_argument(
+        "--trial-string-creator",


trial_name_creator?

ericl · 2018-12-07T23:50:56Z

python/ray/tune/config_parser.py

+    parser.add_argument(
+        "--custom-loggers",
+        default=None,
+        help="Optional URI to sync training results to (e.g. s3://bucket).")


Help is wrong.

ericl · 2018-12-07T23:51:50Z

python/ray/tune/config_parser.py

+        default="",
+        type=str,
+        help="Optional template for syncer to run. Needs to "
+        "include replacement fields '{local_dir}' and '{remote_dir}'.")


Instead of a string how about just have sync_function here. You can also say "btw if you pass a string it will be used by this default function".

AmplabJenkins · 2018-12-08T00:47:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9841/
Test FAILed.

AmplabJenkins · 2018-12-08T00:56:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9844/
Test FAILed.

AmplabJenkins · 2018-12-08T01:05:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9842/
Test FAILed.

hartikainen · 2018-12-08T01:37:03Z

Apart from @ericl's comments, this looks good to me. I really like it, thanks @richardliaw!

llan-ml · 2018-12-08T02:21:58Z

LGTM!

…om_logging

AmplabJenkins · 2018-12-08T22:42:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9876/
Test FAILed.

ericl · 2018-12-08T23:02:18Z

doc/source/tune-usage.rst

+
+        Returns:
+            trial_name (str): String representation of Trial.
+        """


Missing body.

ericl · 2018-12-08T23:02:33Z

doc/source/tune-usage.rst

+.. code-block:: python
+
+    def custom_sync_func(local_dir, remote_dir):
+        # run sync


AmplabJenkins · 2018-12-08T23:19:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9877/
Test FAILed.

AmplabJenkins · 2018-12-09T01:09:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9880/
Test FAILed.

AmplabJenkins · 2018-12-09T19:32:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9900/
Test FAILed.

AmplabJenkins · 2018-12-09T23:16:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9906/
Test FAILed.

AmplabJenkins · 2018-12-10T09:32:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9921/
Test FAILed.

richardliaw · 2018-12-10T20:02:00Z

jenkins retest this please

AmplabJenkins · 2018-12-10T21:36:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9926/
Test FAILed.

richardliaw · 2018-12-11T08:53:20Z

jenkins retest this please

AmplabJenkins · 2018-12-11T10:44:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9955/
Test FAILed.

AmplabJenkins · 2018-12-11T20:36:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9963/
Test PASSed.

First pass at making a better logging mechanism

5dea9f3

fix up some docs

95fd1d0

hartikainen reviewed Dec 6, 2018

View reviewed changes

richardliaw added 3 commits December 6, 2018 16:14

custom logger

d0e5af2

fix up tests

6c4d746

fix up tests

1a10482

richardliaw added 2 commits December 7, 2018 14:12

prefix creator

aad0ddb

better string

7fda85d

richardliaw added 5 commits December 7, 2018 15:26

custom_logging

67c2e41

done

8a42dc6

lint

a93d5d8

Update trainable.py

4c4d580

Update agent.py

e2b6c13

richardliaw changed the title ~~[wip][tune] Custom Logging~~ [tune] Custom Logging Dec 7, 2018

richardliaw changed the title ~~[tune] Custom Logging~~ [tune] Custom Logging, Trial Name Dec 7, 2018

richardliaw assigned ericl Dec 7, 2018

richardliaw requested a review from ericl December 7, 2018 23:44

ericl reviewed Dec 7, 2018

View reviewed changes

Comments, rename, change signature

26e0b65

richardliaw added 3 commits December 8, 2018 14:18

sync func

2a2cf42

yapf

7617460

Merge branch 'custom_logging' of github.com:richardliaw/ray into cust…

c057199

…om_logging

richardliaw added 2 commits December 8, 2018 14:56

docs

2603e63

Merge branch 'master' into custom_logging

ef3b58b

ericl approved these changes Dec 8, 2018

View reviewed changes

comments,tests

cb7f558

fix up tunefunction example

e09560d

fix

5d8b736

test fix

a1c3468

clarity

ff36fd3

richardliaw mentioned this pull request Dec 11, 2018

[Tune] Log csv files using double quotes #3514

Closed

richardliaw merged commit e0fbb68 into ray-project:master Dec 11, 2018

richardliaw deleted the custom_logging branch December 11, 2018 21:42

richardliaw mentioned this pull request Dec 29, 2018

[rllib] Make results-dir/ logdir prefix consistent #2855

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tune] Custom Logging, Trial Name #3465

[tune] Custom Logging, Trial Name #3465

richardliaw commented Dec 4, 2018 •

edited

Loading

AmplabJenkins commented Dec 4, 2018

hartikainen Dec 6, 2018

richardliaw Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

richardliaw commented Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

ericl Dec 7, 2018

ericl Dec 7, 2018

ericl Dec 7, 2018

ericl Dec 7, 2018

richardliaw Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

hartikainen commented Dec 8, 2018

llan-ml commented Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

ericl Dec 8, 2018

ericl Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

AmplabJenkins commented Dec 9, 2018

AmplabJenkins commented Dec 9, 2018

AmplabJenkins commented Dec 9, 2018

AmplabJenkins commented Dec 10, 2018

richardliaw commented Dec 10, 2018

AmplabJenkins commented Dec 10, 2018

richardliaw commented Dec 11, 2018

AmplabJenkins commented Dec 11, 2018

AmplabJenkins commented Dec 11, 2018



		def trial_str_creator(trainable_name, trial_id, config):
		return "{}_{}_123".format(trainable_name, trial_id)

[tune] Custom Logging, Trial Name #3465

[tune] Custom Logging, Trial Name #3465

Conversation

richardliaw commented Dec 4, 2018 • edited Loading

AmplabJenkins commented Dec 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

richardliaw commented Dec 7, 2018

AmplabJenkins commented Dec 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

hartikainen commented Dec 8, 2018

llan-ml commented Dec 8, 2018

AmplabJenkins commented Dec 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 8, 2018

AmplabJenkins commented Dec 9, 2018

AmplabJenkins commented Dec 9, 2018

AmplabJenkins commented Dec 9, 2018

AmplabJenkins commented Dec 10, 2018

richardliaw commented Dec 10, 2018

AmplabJenkins commented Dec 10, 2018

richardliaw commented Dec 11, 2018

AmplabJenkins commented Dec 11, 2018

AmplabJenkins commented Dec 11, 2018

richardliaw commented Dec 4, 2018 •

edited

Loading