[air/output] Collection of small AIR output improvements #37571

krfricke · 2023-07-19T09:20:09Z

Why are these changes needed?

Includes #37531

This PR includes a number of improvements to Ray AIRs context-aware output engine

Remove verbose=x from a number of examples ([AIR output] Use verbose =1 by default in train or tune examples #36911)
Use blocks to logically group outputs/control newlines ([AIR output] Remove the blank line between reported results and checkpoint info. #36815)
Exclude _report_on ([AIR Output] Not sure if some metrics for lightning trainer are unnecessary #36818)
Extend blacklisted keys in result repr ([AIR output] make the best result of tune output more readable #36791)

Related issue number

Closes #36911
Closes #36815
Closes #36818
Closes #36791

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Kai Fricke <kai@anyscale.com>

# Conflicts: # python/ray/tune/experimental/output.py

woshiyyya · 2023-07-20T05:24:20Z

doc/source/ray-air/examples/batch_tuning.ipynb

@@ -688,8 +688,6 @@
    "        # redirect logs to relative path instead of default ~/ray_results/\n",
    "        storage_path=\"my_Tune_logs\",\n",
    "        name=\"batch_tuning\",\n",
-    "        # Set Ray Tune verbosity. Print summary table only with levels 2 or 3.\n",


Do we have a timeline to enable new output by default? Currently the RunConfig docstring is a little confusing.

verbose – 0, 1, 2, or 3. Verbosity mode. 0 = silent, 1 = only status updates, 2 = status and brief results, 3 = status and detailed results. Defaults to 3. If the RAY_AIR_NEW_OUTPUT=1 environment variable is set, uses the new context-aware verbosity settings: 0 = silent, 1 = default, 2 = verbose.

Not needed for now, but would be great to have examples comparing outputs in verbosity level 0, 1, 2

We aim to enable this per default in 2.7

woshiyyya · 2023-07-20T05:29:34Z

python/ray/tune/experimental/output.py

@@ -85,6 +85,7 @@
    "trial_id",
    "experiment_tag",
    "should_checkpoint",
+    "_report_on",  # LIGHTNING_REPORT_STAGE_KEY


Actually this is created by us in LightningTrainer. Yes, we should put it into the blacklist.

woshiyyya · 2023-07-20T05:32:55Z

python/ray/air/result.py

            shown_attributes["metrics"] = {
-                k: v for k, v in self.metrics.items() if k not in AUTO_RESULT_KEYS
+                k: v for k, v in self.metrics.items() if k not in exclude


So BLACKLISTED_KEYS will not be shown in Result and ResultGrid repr, but will still be printed by ProgressReporter on_trial_result, right? I feel like information like time_since_restore and iterations_since_restore are still important for progress tracking.

In this PR we're just removing it from the repr.

Currently the blacklisted keys are also not printed in the results table. Per default, only the most relevant metrics should be printed. If users want to show more metrics, they can customize this.

Maybe customization of this should be a blocker for enabling per default.

matthewdeng

docs stamp

…#37571) This PR includes a number of improvements to Ray AIRs context-aware output engine Remove verbose=x from a number of examples ( [AIR output] Use verbose =1 by default in train or tune examples ray-project#36911) Use blocks to logically group outputs/control newlines ( [AIR output] Remove the blank line between reported results and checkpoint info. ray-project#36815) Exclude _report_on ( [AIR Output] Not sure if some metrics for lightning trainer are unnecessary ray-project#36818) Extend blacklisted keys in result repr ( [AIR output] make the best result of tune output more readable ray-project#36791) Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: NripeshN <nn2012@hw.ac.uk>

…#37571) This PR includes a number of improvements to Ray AIRs context-aware output engine Remove verbose=x from a number of examples ( [AIR output] Use verbose =1 by default in train or tune examples ray-project#36911) Use blocks to logically group outputs/control newlines ( [AIR output] Remove the blank line between reported results and checkpoint info. ray-project#36815) Exclude _report_on ( [AIR Output] Not sure if some metrics for lightning trainer are unnecessary ray-project#36818) Extend blacklisted keys in result repr ( [AIR output] make the best result of tune output more readable ray-project#36791) Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: harborn <gangsheng.wu@intel.com>

…#37571) This PR includes a number of improvements to Ray AIRs context-aware output engine Remove verbose=x from a number of examples ( [AIR output] Use verbose =1 by default in train or tune examples ray-project#36911) Use blocks to logically group outputs/control newlines ( [AIR output] Remove the blank line between reported results and checkpoint info. ray-project#36815) Exclude _report_on ( [AIR Output] Not sure if some metrics for lightning trainer are unnecessary ray-project#36818) Extend blacklisted keys in result repr ( [AIR output] make the best result of tune output more readable ray-project#36791) Signed-off-by: Kai Fricke <kai@anyscale.com>

…#37571) This PR includes a number of improvements to Ray AIRs context-aware output engine Remove verbose=x from a number of examples ( [AIR output] Use verbose =1 by default in train or tune examples ray-project#36911) Use blocks to logically group outputs/control newlines ( [AIR output] Remove the blank line between reported results and checkpoint info. ray-project#36815) Exclude _report_on ( [AIR Output] Not sure if some metrics for lightning trainer are unnecessary ray-project#36818) Extend blacklisted keys in result repr ( [AIR output] make the best result of tune output more readable ray-project#36791) Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

…#37571) This PR includes a number of improvements to Ray AIRs context-aware output engine Remove verbose=x from a number of examples ( [AIR output] Use verbose =1 by default in train or tune examples ray-project#36911) Use blocks to logically group outputs/control newlines ( [AIR output] Remove the blank line between reported results and checkpoint info. ray-project#36815) Exclude _report_on ( [AIR Output] Not sure if some metrics for lightning trainer are unnecessary ray-project#36818) Extend blacklisted keys in result repr ( [AIR output] make the best result of tune output more readable ray-project#36791) Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>

Kai Fricke added 11 commits July 18, 2023 16:45

remove verbose= in examples

4bb58ce

Signed-off-by: Kai Fricke <kai@anyscale.com>

restructure air output handler

ad8e19b

Signed-off-by: Kai Fricke <kai@anyscale.com>

revert train changes

39fd507

Signed-off-by: Kai Fricke <kai@anyscale.com>

time.time

d02fe3a

Signed-off-by: Kai Fricke <kai@anyscale.com>

minor fixes

d65f8bb

Signed-off-by: Kai Fricke <kai@anyscale.com>

revert doc changes

861faaf

Signed-off-by: Kai Fricke <kai@anyscale.com>

Callback

6f1915f

Signed-off-by: Kai Fricke <kai@anyscale.com>

Update examples for verbose=

057188f

Signed-off-by: Kai Fricke <kai@anyscale.com>

Use blocks for newline control

c68d382

Signed-off-by: Kai Fricke <kai@anyscale.com>

LIGHTNING _report_on

c5587d6

Signed-off-by: Kai Fricke <kai@anyscale.com>

blacklisted keys in Result repr

6f7715a

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke mentioned this pull request Jul 19, 2023

[air/output] Add callback hook for trial recovery, only print error table at end #37572

Merged

8 tasks

Kai Fricke added 2 commits July 19, 2023 15:52

add

ef893ca

Signed-off-by: Kai Fricke <kai@anyscale.com>

Merge branch 'master' into air/output/small

af3bae8

# Conflicts: # python/ray/tune/experimental/output.py

krfricke requested a review from woshiyyya July 19, 2023 16:54

krfricke assigned richardliaw and woshiyyya Jul 19, 2023

krfricke marked this pull request as ready for review July 19, 2023 16:55

krfricke requested review from richardliaw, gjoliver, xwjiang2010, amogkam, matthewdeng, Yard1, maxpumperla and a team as code owners July 19, 2023 16:55

woshiyyya approved these changes Jul 20, 2023

View reviewed changes

matthewdeng approved these changes Jul 20, 2023

View reviewed changes

krfricke merged commit bd1b4c9 into ray-project:master Jul 20, 2023
69 of 72 checks passed

krfricke deleted the air/output/small branch July 20, 2023 16:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[air/output] Collection of small AIR output improvements #37571

[air/output] Collection of small AIR output improvements #37571

krfricke commented Jul 19, 2023 •

edited

Loading

woshiyyya Jul 20, 2023

krfricke Jul 20, 2023

woshiyyya Jul 20, 2023

woshiyyya Jul 20, 2023

krfricke Jul 20, 2023

matthewdeng left a comment

[air/output] Collection of small AIR output improvements #37571

[air/output] Collection of small AIR output improvements #37571

Conversation

krfricke commented Jul 19, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

woshiyyya Jul 20, 2023

Choose a reason for hiding this comment

krfricke Jul 20, 2023

Choose a reason for hiding this comment

woshiyyya Jul 20, 2023

Choose a reason for hiding this comment

woshiyyya Jul 20, 2023

Choose a reason for hiding this comment

krfricke Jul 20, 2023

Choose a reason for hiding this comment

matthewdeng left a comment

Choose a reason for hiding this comment

krfricke commented Jul 19, 2023 •

edited

Loading