[SPARK-30069][CORE][YARN] Clean up non-shuffle disk block manager files following executor exists on YARN

[LantaoJin]

What changes were proposed in this pull request?

Currently we only clean up the local directories on application removed. However, when executors die and restart repeatedly, many temp files are left untouched in the local directories, which is undesired behavior and could cause disk space used up gradually. Especially, in long running service like Spark thrift-server with dynamic resource allocation disabled, it's very easy causes local disk full.
#21390 fixed the same problem on Standalone mode. On YARN, this issue still exists.

From #21390 (comment), YARN only cleans container local dirs when container (executor) is exited. But these files are not in container local dirs.

So this patch is very straightforward:
We create these "temp_xxx " files under the container dirs when the executor is running in YARN container.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add an UT and manually test.

[LantaoJin]

cc @jerryshao @jiangxb1987 @JoshRosen

[SparkQA]

Test build #114626 has finished for PR 26711 at commit c226dd6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[LantaoJin]

retest this please

[SparkQA]

Test build #114690 has finished for PR 26711 at commit c226dd6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

This is actually writing temp shuffle files into YARN container local dirs instead of local dirs? Could there be any performance difference after this PR? (Sorry I'm not familiar with YARN)

[LantaoJin]

This is actually writing temp shuffle files into YARN container local dirs instead of local dirs? Could there be any performance difference after this PR? (Sorry I'm not familiar with YARN)

Yes. I think only this part could have performance difference in YARN mode:
https://github.com/apache/spark/pull/26711/files#diff-b25a153e05a044779df448f024c3a3afR125
But this part already had potentially performance problem. See

spark/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala

Line 215 in 4021354

* is a potentially expensive operation and should only be used for testing.

[LantaoJin]

Ah, DiskBlockManager.stop() also needs to traverse localDirs and containerDirs to delete them when Executor.stop() in YARN mode.
https://github.com/apache/spark/pull/26711/files#diff-b25a153e05a044779df448f024c3a3afR248

But I think it's not a problem.

[LantaoJin]

gentle ping @cloud-fan @jiangxb1987 @dongjoon-hyun @jerryshao

[LantaoJin]

@vanzin @squito

[LantaoJin]

Gentle ping @cloud-fan @jiangxb1987 @dongjoon-hyun @jerryshao @vanzin @squito @srowen @HyukjinKwon

[LantaoJin]

retest this please

[SparkQA]

Test build #119692 has finished for PR 26711 at commit c226dd6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[LantaoJin]

ping @cloud-fan @jiangxb1987 @dongjoon-hyun @srowen @maropu

[tgravescs]

perhaps you can expand on the description as to the approach you are taking? How do you know the temp_ files aren't being used?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[tianshuang]

@LantaoJin Has this problem been solved? We are facing the same problem now

[LantaoJin]

@tianshuang This problem should be resolved by this PR. But it was closed by githut bot. So I think it still exists in 3.0.

[tianshuang]

After thrift-server runs for a long time, you will encounter an error of 'no space left on device'. I hope to reopen this issue to completely solve the problem.

[LantaoJin]

After thrift-server runs for a long time, you will encounter an error of 'no space left on device'. I hope to reopen this issue to completely solve the problem.

It cannot reopen now. Seems I have to re-create a new PR.

[LantaoJin]

@tianshuang recreated as #29378