New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-1777 (partial)] bugfix: make size of requested memory correctly #1892
Conversation
Can one of the admins verify this patch? |
Hi @liyezhang556520, can you explain your changes? The idea of the existing code is that after requesting, we will end up at |
Hi @andrewor14, |
Ah I see. |
@@ -254,7 +254,7 @@ private[spark] class MemoryStore(blockManager: BlockManager, maxMemory: Long) | |||
} | |||
} | |||
// New threshold is currentSize * memoryGrowthFactor | |||
memoryThreshold = currentSize + amountToRequest | |||
memoryThreshold += amountToRequest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this becomes currentSize * memoryGrowthFactor
if you do the math. Why not just set it to this value? (Then you can remove the comment above)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's better use memoryThreshold += amountToRequest
than memoryThreshold = currentSize * memoryGrowthFactor
, because memoryThreshold += amountToRequest
is more obvious for coder reader how memoryThreshold
increases, and the comment above is more easy to understand.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, that's also fine
add to whitelist |
QA tests have started for PR 1892. This patch merges cleanly. |
QA results for PR 1892: |
LGTM |
Thanks. Merging in master & branch-1.1 |
Author: Zhang, Liye <liye.zhang@intel.com> Closes #1892 from liyezhang556520/lazy_memory_request and squashes the following commits: 335ab61 [Zhang, Liye] [SPARK-1777 (partial)] bugfix: make size of requested memory correctly (cherry picked from commit 2bd8126) Signed-off-by: Reynold Xin <rxin@apache.org>
Author: Zhang, Liye <liye.zhang@intel.com> Closes apache#1892 from liyezhang556520/lazy_memory_request and squashes the following commits: 335ab61 [Zhang, Liye] [SPARK-1777 (partial)] bugfix: make size of requested memory correctly
No description provided.