[SPARK-26570][SQL] Reduce memory consumption when transforming very many filestatus in InMemoryFileIndex.bulkListLeafFiles

[viirya]

What changes were proposed in this pull request?

This PR proposes to wrap the collected SerializableFileStatus with SoftReference, when collecting file statuses in InMemoryFileIndex.bulkListLeafFiles.

Then later when we transform SerializableFileStatus back to Status, if there is memory pressure, traversed SerializableFileStatus can be candidates for GC.

Why are the changes needed?

We get an array of (String, Seq[SerializableFileStatus]) when collecting file status in InMemoryFileIndex.bulkListLeafFiles. Then later we transform it back to sequence of (Path, Seq[FileStatus]).

During the transforming, the items in the array are hold and can not be released by GC. Actually we double memory consumption here.

This change wraps the sequence of SerializableFileStatus with SoftReference in the array. So when we transform SerializableFileStatus back to Status, if there is memory pressure, traversed SerializableFileStatus can be candidates for GC.

If SerializableFileStatus that is not traversed yet is cleared up by GC, we log necessary to users suggesting increasing driver memory. Then we re-list file status for the path.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test.

[SparkQA]

Test build #112001 has finished for PR 26105 at commit 7fc8117.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

First from the stacktrace, I was thinking it is possible that the reason to user OOM is that SerializableFileStatus is too many. Then it is possible to cause OOM when transforming them back to Status. This PR was created for that.

But from users latest posts, from jmap logs, looks like SerializableFileStatus do not hold too much memory. For now I am not sure SerializableFileStatus is the cause of the OOM, so close this first.

[sabyasachinandy]

Did this get resolved?