forked from ray-project/ray
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Data] Fix throughput time calculations (ray-project#44138)
When doing some documentation for the new stats we added, I noticed some quirks with the dataset throughput reporting. Dataset throughput was often reported as worse through Ray Data than the single node approximation, but the single node approximation should be a floor for how fast it should run through Ray Data (i.e. they should match if a single node / single block was used). We want compute the Ray Data dataset throughput by dividing the total number of rows produced by the total wall time it took for the operation to run (i.e. from starting the process to process completion). We want to compute the single node approximation by dividing the total number of rows produced by the total wall time of all blocks across all operators (i.e. the total amount of time if there were no concurrent processes). Upon digging, I found that the calculation was incorrect. For the Ray Data dataset throughput, we were dividing by `self.time_total_s` which is the total time for the last operator in the chain (sometimes correct, but incorrect with multiple non-fused operators). For the single node approximation we were dividing by `self.get_total_wall_time()`, which is actually what we want for the Ray Data dataset throughput, but incorrect for the single node approximation as it does not total time over the multiple blocks. Signed-off-by: Matthew Owen <mowen@anyscale.com>
- Loading branch information
1 parent
554de41
commit fbb2c6a
Showing
2 changed files
with
177 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters