[SPARK-20255] Move listLeafFiles() to InMemoryFileIndex

[adrian-ionescu]

What changes were proposed in this pull request

Trying to get a grip on the FileIndex hierarchy, I was confused by the following inconsistency:

On the one hand, PartitioningAwareFileIndex defines leafFiles and leafDirToChildrenFiles as abstract, but on the other it fully implements listLeafFiles which does all the listing of files. However, the latter is only used by InMemoryFileIndex.

I'm hereby proposing to move this method (and all its dependencies) to the implementation class that actually uses it, and thus unclutter the PartitioningAwareFileIndex interface.

How was this patch tested?

./build/sbt sql/test

[rxin]

Jenkins, add to whitelist.

[rxin]

LGTM pending Jenkins.

[SparkQA]

Test build #3645 has finished for PR 17570 at commit 1c69820.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Merging in master.