Skip to content

Commit

Permalink
[SPARK-41327][CORE] Fix SparkStatusTracker.getExecutorInfos by swit…
Browse files Browse the repository at this point in the history
…ch On/OffHeapStorageMemory info

### What changes were proposed in this pull request?

This PR aims to fix `SparkStatusTracker.getExecutorInfos` to return a correct `on/offHeapStorageMemory`.

### Why are the changes needed?

`SparkExecutorInfoImpl` used the following parameter order.
https://github.com/apache/spark/blob/54c57fa86906f933e089a33ef25ae0c053769cc8/core/src/main/scala/org/apache/spark/StatusAPIImpl.scala#L42-L45

SPARK-20659 introduced a bug with wrong parameter order at Apache Spark 2.4.0.
- https://github.com/apache/spark/pull/20546/files#diff-7daca909d33ff8e9b4938e2b4a4aaa1558fbdf4604273b9e38cce32c55e1508cR118-R121

### Does this PR introduce _any_ user-facing change?

Yes.

### How was this patch tested?

Manually review.

Closes apache#38843 from ylybest/master.

Lead-authored-by: Lingyun Yuan <ylybest@gmail.com>
Co-authored-by: ylybest <119458293+ylybest@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 388824c)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
2 people authored and dongjoon-hyun committed Nov 30, 2022
1 parent 443630d commit b20ba7d
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,10 @@ class SparkStatusTracker private[spark] (sc: SparkContext, store: AppStatusStore
port,
cachedMem,
exec.activeTasks,
exec.memoryMetrics.map(_.usedOffHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.usedOnHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.totalOffHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.totalOnHeapStorageMemory).getOrElse(0L))
exec.memoryMetrics.map(_.usedOffHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.totalOnHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.totalOffHeapStorageMemory).getOrElse(0L))
}.toArray
}
}

0 comments on commit b20ba7d

Please sign in to comment.