[SPARK-20136][SQL] Add num files and metadata operation timing to scan operator metrics

[rxin]

What changes were proposed in this pull request?

This patch adds explicit metadata operation timing and number of files in data source metrics. Those would be useful to include for performance profiling.

Screenshot of a UI with this change (num files and metadata time are new metrics):

How was this patch tested?

N/A

[rxin]

cc @ericl, @bogdanrdc, @adrian-ionescu, @cloud-fan

[ericl]

Could we add a unit test for this?

[ericl]

This doesn't take into account the time for the initial partition pruning, which is done earlier in logical planning. That's in the PruneFileSourcePartitions rule, so not sure how to connect it with this physical operator though.

[SparkQA]

Test build #75349 has finished for PR 17465 at commit 9efc548.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[adrian-ionescu]

LGTM

I think it's ok that it doesn't take into account partition pruning, since that's not really part of this physical operator. Another thing to note here is that listFiles() may often return from cache. We could consider exposing file cache hits & misses as another metric.

[rxin]

Let me merge this now. I will send a follow-up PR to take the logical planning time into account (otherwise in the vast majority of cases, i.e. pruned partitions, the metadata operation time will be approximately 0).

[rxin]

Merging in master.