Add files_processed and files_scanned metrics to FileStreamMetrics

[adriangb]

Summary

• Add files_processed counter to FileStreamMetrics, incremented for every file assigned to the partition — whether it was opened, pruned (returned an empty stream), or skipped due to a LIMIT. When the stream completes, this equals the total number of files in the partition.
• Add files_opened counter to FileStreamMetrics, incremented as soon as we consider a file for processing (either actually opened, discarded because of a LIMIT or stats, etc.).

Motivation

These metrics enable tracking query progress during long-running scans. Today, there is no way to monitor how far along a file scan is. The existing FileStreamMetrics only provide:

• Timing metrics (time_elapsed_opening, time_elapsed_scanning_total, etc.) — these measure duration but don't indicate progress. You can't tell whether a scan is 10% or 90% done from elapsed time alone.
• Error counters (file_open_errors, file_scan_errors) — these only count failures, not successful progress.
• output_rows or bytes_scanned (from BaselineMetrics) — counts rows emitted, but since we don't know upfront how many rows will be emitted in total this is a poor metric, i.e. it never converges to 100% if there are filters, etc.

In contrast, files_processed and files_opened combined with the known number of files in file_groups give a clear progress indicator: files_processed / total_files. This is the most natural and reliable way to track scan progress since the file count is known at plan time. Depending on what users plan to do with the metric they can pick files_opened / total_files (leading metric) or files_processed / total_files (lagging metric).

Test plan

• Existing file_stream tests pass (8/8)
• cargo check -p datafusion-datasource compiles cleanly

🤖 Generated with Claude Code

[getChan]

Thanks for the improvement.

Small concern: files_processed is incremented on open success (and LIMIT short-circuit), not on completion.
so I considered renaming to better reflect this, but couldn’t find a clearer naming.

[adriangb]

Thanks for the review @getChan. The goal of this metric was precisely to record when we are done with a file, so really it's the implementation that was wrong. I've fixed it now. I also added a files_opened metric to complete the picture.

[adriangb]

@comphead @gabotechs any chance one of you could take a look at this?

[comphead]

                                        // Count this file, the prefetched next file
                                        // (if any), and all remaining files we will
                                        // never open.

I read it twice and prob need some help to understand what files are being counted here? if they never opened how come we need to count them as closed?

[adriangb]

This is if we hit a LIMIT. Maybe files_processed is a better term? I want this to be useful to track the progress of a query, IMO the best metric we have now is "how many files have we completed work on (for any reason) out of all of the files we can possibly look at?".

[comphead]

yeah, processed would be more intuitive I'd say

[adriangb]

updated!

[comphead]

so the file can be opened, but not scanned?

[adriangb]

Yes, e.g. if FilePruner discards it based on stats.

[comphead]

would be that more explanatory to see how many files were skipped instead of closed? Intuitively i would suggest number of open files == number of closed files, unless we debug a connection leak. And skipped by stats would make more sense?

[adriangb]

I think the two invariants are:

• number of open files >= number of closed files
• number of open files == number of closed files at the beginning and end of execution

Regarding files_scanned vs. files_skipped: we already have metrics for file_groups_prunned_statistics. I don't really need the files_scanned metric so let me remove it.

[comphead]

Thanks @adriangb