[air/output] Add callback hook for trial recovery, only print error table at end

[krfricke]

Why are these changes needed?

Includes #37571

This changes the context-aware output handler so that trial errors are immediately reported with their respective error files. The error table is only printed at the end.

This introduces a new callback hook, on_trial_recover which is required so error files are also available in the immediate output when a trial has a transient failure.

Two alternatives here are to either rely on the trial error handling to output the error file or to just not output the error file and wait for a full failure (on_trial_error) or experiment end to show error files for transient errors.

During training:

Trial train_f0b35_00000 errored after 0 iterations at 2023-07-19 12:18:14. Total running time: 5s
Error file: /Users/kai/ray_results/train_2023-07-19_12-18-05/train_f0b35_00000_0_always=True,fail=1_2023-07-19_12-18-09/error.txt

Only at end:

Number of errored trials: 3
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Trial name            # failures   error file                                                                                                             │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ train_f0b35_00004              3   /Users/kai/ray_results/train_2023-07-19_12-18-05/train_f0b35_00004_4_always=True,fail=1_2023-07-19_12-18-09/error.txt  │
│ train_f0b35_00000              3   /Users/kai/ray_results/train_2023-07-19_12-18-05/train_f0b35_00000_0_always=True,fail=1_2023-07-19_12-18-09/error.txt  │
│ train_f0b35_00005             1*   /Users/kai/ray_results/train_2023-07-19_12-18-05/train_f0b35_00005_5_always=False,fail=1_2023-07-19_12-18-09/error.txt │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
* The trial terminated successfully after retrying.

Related issue number

Closes #36854

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[woshiyyya]

Looks great! A few questions:

• Can we print error files both on_trial_error and on_experiment_end?
• Is the error file name informative? Maybe we can print the stack trace (e.g. the last 10 lines)?
• Maybe off the topic, but currently we rewrite the error.txt and only keeps the last one. The users cannot access the previous ones. Do we have any plan to add a suffix and keep all the error_{id}.txt files?

[krfricke]

Looks great! A few questions:

* Can we print error files both `on_trial_error` and  `on_experiment_end`?

On end, we get the error table.

* Is the error file name informative? Maybe we can print the stack trace (e.g. the last 10 lines)?

I should have clarified this - the regular error stacktrace is still streamed from the training worker to stderr!

* Maybe off the topic, but currently we rewrite the `error.txt` and only keeps the last one. The users cannot access the previous ones. Do we have any plan to add a suffix and keep all the `error_{id}.txt` files?

The error.txt is automatically appended, so we keep all previously caught errors automatically.

[xwjiang2010]

Thanks! I am a fan with the footnote :)

[woshiyyya]

Thanks Kai. LGTM!