branch-4.0: [enhance](iceberg) Doris Iceberg Scan Metrics Integration #59010#59095
Merged
yiguolei merged 1 commit intobranch-4.0from Dec 18, 2025
Merged
branch-4.0: [enhance](iceberg) Doris Iceberg Scan Metrics Integration #59010#59095yiguolei merged 1 commit intobranch-4.0from
yiguolei merged 1 commit intobranch-4.0from
Conversation
### What problem does this PR solve?
## Overview
This change pipes Apache Iceberg scan metrics directly into Doris query
profiles so operators can inspect per-scan statistics (files, bytes,
manifests, filters, etc.) from the FE profile UI. The integration
consists of three pieces:
1. **Summary Profile Slot** – `SummaryProfile` now includes an `Iceberg
Scan Metrics` entry in the execution summary list so the FE profile
table reserves space for the Iceberg telemetry.
2. **Metrics Reporter** – a new `IcebergMetricsReporter` implementation
formats `ScanReport` data (planning time, file counters, size counters,
delete-file stats, projected columns, metadata) and appends it to the
summary entry whenever an Iceberg scan runs.
3. **Scan Integration** – `IcebergScanNode` calls
`table.newScan().metricsReporter(new IcebergMetricsReporter())`,
ensuring every Iceberg table scan emits the metrics without requiring
catalog-level configuration changes.
All metrics remain scoped to Iceberg scans; other table formats are
untouched and still populate their own runtime-profile sections as
before.
## Implementation Details
### 1. `SummaryProfile`
`fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java`
- Added the constant `ICEBERG_SCAN_METRICS = "Iceberg Scan Metrics"`.
- Inserted the key into `EXECUTION_SUMMARY_KEYS` (with indentation level
3) so the runtime profile tree displays it under the scheduling block
when present.
- No default text is shown unless metrics are actually reported; the
entry stays `N/A` otherwise.
### 2. `IcebergMetricsReporter`
`fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/profile/IcebergMetricsReporter.java`
- Implements `org.apache.iceberg.metrics.MetricsReporter`.
- On each `ScanReport`, it retrieves the current `SummaryProfile` from
`ConnectContext`, grabs the execution summary `RuntimeProfile`, and
appends a human-readable string for that scan.
- Metrics covered:
- Table, snapshot ID, sanitized filter text, projected column list.
- Planning time (pretty-printed duration plus operation count).
- Result/skipped data and delete file counts.
- Total file size / total delete file size (in readable units).
- Manifest counts (scanned/skipped for data and delete manifests).
- Indexed/equality/positional delete file counters.
- Selected metadata keys (currently `scan-state`, `scan-id` if present).
- When multiple Iceberg scans run in one query, each scan’s line is
appended on a new line under the same summary key.
### 3. `IcebergScanNode`
`fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java`
- Replaces `icebergTable.newScan()` with
`icebergTable.newScan().metricsReporter(new IcebergMetricsReporter())`
inside `createTableScan()`.
- This keeps catalog properties untouched and leverages Iceberg’s
per-scan API to attach reporters.
## Example Profile Output
After running a query against `iceberg_docker.test_db.ts_identity`, the
FE profile shows:
```
Iceberg Scan Metrics:
Table Scan (iceberg_docker.test_db.ts_identity):
- table: iceberg_docker.test_db.ts_identity
- snapshot: 6315378011972705169
- filter: true
- columns: id|ts
- planning: 7ms (1 ops)
- data_files: 3
- delete_files: 0
- skipped_data_files: 0
- skipped_delete_files: 0
- total_size: 1.892 KB
- total_delete_size: 0.000
- scanned_manifests: 1
- skipped_manifests: 0
- scanned_delete_manifests: 0
- skipped_delete_manifests: 0
- indexed_delete_files: 0
- equality_delete_files: 0
- positional_delete_files: 0
```
Contributor
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Dec 18, 2025
github-actions
bot
added
the
approved
Contributor
Author
|
PR approved by at least one committer and no changes requested. |
Contributor
Author
|
PR approved by anyone and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #59010