New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-10985][CORE] Avoid passing evicted blocks throughout BlockManager #10776
Conversation
Test build #49484 has finished for PR 10776 at commit
|
Option(TaskContext.get()).foreach { tc => | ||
val metrics = tc.taskMetrics() | ||
val lastUpdatedBlocks = metrics.updatedBlocks.getOrElse(Seq[(BlockId, BlockStatus)]()) | ||
metrics.updatedBlocks = Some(lastUpdatedBlocks ++ evictedBlocks.toSeq) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I plan to move this call closer to the eviction site itself (i.e. perform this call inside either the memory store or the block manager itself rather than here).
case Left(arr) => | ||
// We have successfully unrolled the entire partition, so cache it in memory | ||
updatedBlocks ++= |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This update has been pushed further down into the BlockManager code. See the changes to doPut()
. The same holds for the putIterator()
call above.
@andrewor14 @rxin, this should now be ready for review. |
Test build #49581 has finished for PR 10776 at commit
|
@@ -847,7 +854,7 @@ private[spark] class BlockManager( | |||
.format(blockId, Utils.getUsedTimeMs(startTimeMs))) | |||
} | |||
|
|||
updatedBlocks | |||
marked |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not clear what this means, maybe success
or putSuccess
?
Merged into master. |
This patch refactors portions of the BlockManager and CacheManager in order to avoid having to pass
evictedBlocks
lists throughout the code. It appears that these lists were only consumed byTaskContext.taskMetrics
, so the new code now directly updates the metrics from the lower-level BlockManager methods.