[Data] Add Runtime Metrics String

[omatthew98]

Why are these changes needed?

This adds an additional set of runtime metrics printed to help identify bottlenecks in the Ray Data code. For example for the following code:

def f(row):
    row["length"] = len(row["text"])
    return row

def g(row):
    return row["length"] > 100

hf_ds = datasets.load_dataset("tweet_eval", "emotion")
ds = ray.data.from_huggingface(hf_ds["train"])
ds = ds.map(f).sort("length").filter(g).materialize()

print(ds.stats())

the following runtime metrics would be printed:

Runtime Metrics:
* ReadParquet->SplitBlocks(24): 1.42s (80.860%)
* Map(f): 290.89ms (16.539%)
* Sort: 0us (0.000%)
* Filter(g): 39.68ms (2.256%)
* Scheduling: 166.04ms (9.440%)
* Total: 1.76s (100%)

Unrelated to the main changes, I noticed that I was using the wrong computation for the total_wall_time used in the dataset throughput, so I fixed that. That unearthed a bug in the DatasetStatsSummary.get_total_wall_time, so I fixed that as well.

Related issue number

Closes #42804

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[c21]

LGTM

Re the example

Runtime Metrics:
* ReadParquet->SplitBlocks(24): 1.42s (80.860%)
* Map(f): 290.89ms (16.539%)
* Sort: 0us (0.000%)
* Filter(g): 39.68ms (2.256%)
* Scheduling: 166.04ms (9.440%)
* Total: 1.76s (100%)

Any reason why Sort took 0us in metrics here?

[omatthew98]

LGTM

Re the example

Runtime Metrics:
* ReadParquet->SplitBlocks(24): 1.42s (80.860%)
* Map(f): 290.89ms (16.539%)
* Sort: 0us (0.000%)
* Filter(g): 39.68ms (2.256%)
* Scheduling: 166.04ms (9.440%)
* Total: 1.76s (100%)

Any reason why Sort took 0us in metrics here?

Hmm I think it might be a quirk of Sort having two sub-operators. From a little pdb investigation it seems like the SortMap and SortReduct OperatorStatsSummarys have their time_total_s set to 0 after execution. I think I have a potential fix for why this would be happening and will put in a separate PR.