Please sign in to comment.
Remove the remoteFetchTime metric.
This metric is confusing: it adds up all of the time to fetch shuffle inputs, but fetches often happen in parallel, so remoteFetchTime can be much longer than the task execution time. @squito it looks like you added this metric -- do you have a use case for it? cc @shivaram -- I know you've looked at the shuffle performance a lot so chime in here if this metric has turned out to be useful for you! Author: Kay Ousterhout <email@example.com> Closes #62 from kayousterhout/remove_fetch_variable and squashes the following commits: 43341eb [Kay Ousterhout] Remote the remoteFetchTime metric.
- Loading branch information...
Showing with 0 additions and 14 deletions.
- +0 −1 core/src/main/scala/org/apache/spark/BlockStoreShuffleFetcher.scala
- +0 −7 core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
- +0 −1 core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala
- +0 −4 core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala
- +0 −1 core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala