Skip to content

Commit

Permalink
[SPARK-33283][CORE] Remove useless externalBlockStoreSize from RDDInfo
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
"external block store" API was removed after SPARK-12667,  `externalBlockStoreSize` in `RDDInfo` looks like always 0 and useless. So this pr just to remove this useless variable.

### Why are the changes needed?
remove useless variable.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass the Jenkins or GitHub Action

Closes #30179 from LuciferYang/SPARK-12667-FOLLOWUP.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Oct 29, 2020
1 parent 056b622 commit fa63117
Show file tree
Hide file tree
Showing 3 changed files with 2 additions and 5 deletions.
5 changes: 2 additions & 3 deletions core/src/main/scala/org/apache/spark/rdd/RDD.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1919,9 +1919,8 @@ abstract class RDD[T: ClassTag](

val persistence = if (storageLevel != StorageLevel.NONE) storageLevel.description else ""
val storageInfo = rdd.context.getRDDStorageInfo(_.id == rdd.id).map(info =>
" CachedPartitions: %d; MemorySize: %s; ExternalBlockStoreSize: %s; DiskSize: %s".format(
info.numCachedPartitions, bytesToString(info.memSize),
bytesToString(info.externalBlockStoreSize), bytesToString(info.diskSize)))
" CachedPartitions: %d; MemorySize: %s; DiskSize: %s".format(
info.numCachedPartitions, bytesToString(info.memSize), bytesToString(info.diskSize)))

s"$rdd [$persistence]" +: storageInfo
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -737,7 +737,6 @@ private[spark] class BlockManagerInfo(
if (storageLevel.isValid) {
/* isValid means it is either stored in-memory or on-disk.
* The memSize here indicates the data size in or dropped from memory,
* externalBlockStoreSize here indicates the data size in or dropped from externalBlockStore,
* and the diskSize here indicates the data size in or dropped to disk.
* They can be both larger than 0, when a block is dropped from memory to disk.
* Therefore, a safe way to set BlockStatus is to set its info in accurate modes. */
Expand Down
1 change: 0 additions & 1 deletion core/src/main/scala/org/apache/spark/storage/RDDInfo.scala
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ class RDDInfo(
var numCachedPartitions = 0
var memSize = 0L
var diskSize = 0L
var externalBlockStoreSize = 0L

def isCached: Boolean = (memSize + diskSize > 0) && numCachedPartitions > 0

Expand Down

0 comments on commit fa63117

Please sign in to comment.