[SPARK-14055] writeLocksByTask need to be update when removeBlock

[Earne]

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/SPARK-14055

How was this patch tested?

manual tests by running LiveJournalPageRank on a large dataset ( the dataset must larger enough to incure RDD partition eviction).

[JoshRosen]

Jenkins, this is ok to test.

[Earne]

can we always calling blockManager.releaseLock(blockId) no matter newEffectiveStorageLevel is valid or not?

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L360

[JoshRosen]

Can we always calling blockManager.releaseLock(blockId) no matter newEffectiveStorageLevel is valid or not?

No, we have to hold the write lock while calling blockInfoManager.removeBlock(). We have to call BlockInfoManager.removeBlock() to ensure that we don't leak metadata for blocks which are fully removed.

[SparkQA]

Test build #53739 has finished for PR 11875 at commit 679c731.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

Would it maybe be better to have some text explaining the change than just a JIRA link in the PR description?

[JoshRosen]

LGTM. Although a nicer description would be good, I think the patch's title is fairly self-explanatory. It also might be nice to have a regression test, but I can take care of that later during a planned future patch to refactor some locking internals in BlockInfoManager.

[JoshRosen]

Merging this now. Thanks!