[air/output] Fix excluded keys in results and config

[krfricke]

Why are these changes needed?

We are currently applying the exclude keys on the flattened results, but this will not work if the excluded keys are dictionaries, such as the config value in the results dict. Instead, we should operate on the original results dict and only flatten from the second level onwards.

Related issue number

Closes #36756

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[justinvyu]

One question: now it's only possible to exclude a full top-level key. Is the exclude key configurable by users or is it just used internally? I remember the trial output table can be configured to only show a certain set of columns, but I think that's a separate path.

[krfricke]

That's a good point - looks like this won't work for nested metrics (e.g. this will break rllib). I'll update the PR early next week

[krfricke]

@justinvyu updated, PTAL

[justinvyu]

Thanks!

A few other points:

1. Can we test the include behavior as well?
2. The only usage of include is here:

   ray/python/ray/tune/experimental/output.py

   Line 1017 in 9f25850

   include=self._metrics,

• The metrics field is populated by: https://github.com/ray-project/ray/blob/9f258506bbcd58763b032b416a90cd00152bfe33/python/ray/tune/tune.py#L958C55-L958C55
• This gets a _progress_metrics attribute from the Trainable, but I don't think this thing is ever set anywhere. Can we remove the usage of include here? We can keep the include functionality for the future (ex: make it customizable as a user config).

[krfricke]

Thanks!

A few other points:

1. Can we test the `include` behavior as well?

2. The only usage of `include` is here: https://github.com/ray-project/ray/blob/9f258506bbcd58763b032b416a90cd00152bfe33/python/ray/tune/experimental/output.py#L1017


* The metrics field is populated by: https://github.com/ray-project/ray/blob/9f258506bbcd58763b032b416a90cd00152bfe33/python/ray/tune/tune.py#L958C55-L958C55

* This gets a `_progress_metrics` attribute from the `Trainable`, but I don't think this thing is ever set anywhere. Can we remove the usage of `include` here? We can keep the include functionality for the future (ex: make it customizable as a user config).

It's actually used in rllib, e.g. here:

https://github.com/ray-project/ray/blob/master/rllib/algorithms/algorithm.py#L211-L218