Describe
Native file scans expose driver-side metrics such as numFiles and numPartitions, but native Iceberg scans currently only expose the default native file scan metrics. This makes it harder to inspect how many Iceberg files and Spark file partitions were planned for a native Iceberg scan.
Solution
Add numFiles and numPartitions SQL metrics to NativeIcebergTableScanExec, consistent with NativeFileSourceScanBase.
The metrics should be populated on the driver after Iceberg FileScanTasks are converted into Spark FilePartitions.
Additional context
Entry points:
NativeIcebergTableScanExec.scalaNativeFileSourceScanBase.scala
Describe
Native file scans expose driver-side metrics such as
numFilesandnumPartitions, but native Iceberg scans currently only expose the default native file scan metrics. This makes it harder to inspect how many Iceberg files and Spark file partitions were planned for a native Iceberg scan.Solution
Add
numFilesandnumPartitionsSQL metrics toNativeIcebergTableScanExec, consistent withNativeFileSourceScanBase.The metrics should be populated on the driver after Iceberg
FileScanTasks are converted into SparkFilePartitions.Additional context
Entry points:
NativeIcebergTableScanExec.scalaNativeFileSourceScanBase.scala