Skip to content

[fix](fe) Report Iceberg scan metrics in batch mode - #66521

Open
suxiaogang223 wants to merge 1 commit into
apache:masterfrom
suxiaogang223:codex/fix-iceberg-batch-scan-metrics
Open

[fix](fe) Report Iceberg scan metrics in batch mode#66521
suxiaogang223 wants to merge 1 commit into
apache:masterfrom
suxiaogang223:codex/fix-iceberg-batch-scan-metrics

Conversation

@suxiaogang223

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Iceberg eager scans attach an SDK metrics reporter and drain the resulting scan profile immediately after planning. Streaming batch scans skipped both steps, so their query profiles showed no Iceberg scan metrics even though the same files and manifests were planned.

This change attaches the existing reporter to streaming scans and drains it after the split source closes. Because streaming split generation runs asynchronously, the query execution summary is captured before dispatch. Profile-group creation is also serialized so concurrent streaming scan nodes cannot replace one another's metric groups.

Release note

Iceberg batch-mode queries now report the same scan metrics as non-batch queries.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Validation performed:

  • IcebergScanPlanProviderTest: 129 tests passed.

  • PluginDrivenScanNodeScanProfileTest: 4 tests passed.

  • Connector SPI test suite: 140 tests passed.

  • Full FE build: ./build.sh --fe succeeded.

  • Behavior changed:

    • No.
    • Yes. Iceberg batch-mode query profiles now include scan metrics.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Iceberg eager scans attach an SDK metrics reporter and drain the resulting scan profile immediately after planning. Streaming batch scans skipped both steps, so their query profiles showed no Iceberg scan metrics even though the same files and manifests were planned. Attach the existing reporter to streaming scans, drain it after the split source closes using the query-scoped execution summary captured before asynchronous dispatch, and serialize profile-group creation across concurrent streaming scans.

### Release note

Iceberg batch-mode queries now report the same scan metrics as non-batch queries.

### Check List (For Author)

- Test: Unit Test
    - IcebergScanPlanProviderTest
    - PluginDrivenScanNodeScanProfileTest
- Behavior changed: Yes. Iceberg batch-mode query profiles now include scan metrics.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/90) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
suxiaogang223 marked this pull request as ready for review August 7, 2026 06:28
@suxiaogang223

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31154055776

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

3 similar comments
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/84) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants