Skip to content

[branch-4.1][fix](iceberg) Backport system table projection fix#65882

Open
hello-stephen wants to merge 1 commit into
branch-4.1from
agent/branch41-iceberg-system-projection
Open

[branch-4.1][fix](iceberg) Backport system table projection fix#65882
hello-stephen wants to merge 1 commit into
branch-4.1from
agent/branch41-iceberg-system-projection

Conversation

@hello-stephen

@hello-stephen hello-stephen commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Backport of #65262 for DORIS-26745.

Current branch-4.1 validation failure: internal build 203691, test_iceberg_system_table_projection, UnsupportedOperationException when deserializing list from a system-table projection.

The patch keeps the branch-4.1 FileQueryScanNode flow and excludes partition keys plus Iceberg/global row-id virtual columns from the projected schema.

Validation:

  • git diff --check: passed
  • Local targeted Maven run could not reach tests because this standalone worktree does not contain generated Thrift sources; the PR compile pipeline is the authoritative validation.

Cherry-picked/adapted from master commit d8e75cf.

@hello-stephen
hello-stephen requested a review from yiguolei as a code owner July 22, 2026 07:46
@hello-stephen

hello-stephen commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei

yiguolei commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

run buildall

@hello-stephen
hello-stephen force-pushed the agent/branch41-iceberg-system-projection branch from 836a849 to 4336f7e Compare July 24, 2026 07:36
@hello-stephen

hello-stephen commented Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

[conflict-resolved-4336f7ec]

Rebased this backport onto current branch-4.1 (192c2dc04873599c75b0c4e125d4b8f5a640b201) and resolved the IcebergScanNode/IcebergScanNodeTest conflicts while preserving both the branch snapshot/scan-semantics changes and the system-table projection implementation/tests.

Attempt history:

  • attempt 1 — old Compile #1003818: first real error was missing SlotDescriptor / Column imports in IcebergScanNode; the rebased head included both.
  • attempt 2 — FE UT #1004684 and Compile #1004685: both finished FAILURE with the same first real compile error, IcebergScanNodeTest.java:[151,84] incompatible types: TupleId cannot be converted to TupleDescriptor.
  • attempt 3 — current head 139c1ff311e45cc5e3c754e6fce191e061abaf84: adapted the branch-4.1 constructor call from new SlotDescriptor(new SlotId(slotId), desc.getId()) to new SlotDescriptor(new SlotId(slotId), desc), matching the branch constructor SlotDescriptor(SlotId, TupleDescriptor).

Attempt-3 validation:

  • conflict markers: none
  • git diff --check: passed
  • FE Maven checkstyle: 0 violations
  • target Groovy syntax compile: passed
  • suite discover/load: test_iceberg_system_table_projection, tags p0,external,iceberg
  • local Maven with JDK 17 passed checkstyle, then stopped before the target module because the isolated worktree lacks generated org.apache.doris.thrift sources; this is not a source verdict, so fresh community FE UT/Compile are required.

[attempt-3-slot-descriptor-parent]

This is the third autonomous adaptation attempt for the same backport. If the fresh head still fails for the same branch-adaptation problem, I will stop further autonomous code changes, document the three attempts and remaining blocker, and hand the implementation back to the developer as agreed.

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30076050993

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@suxiaogang223 @hello-stephen
[fix](iceberg) Project Iceberg system table scans (#65262)
139c1ff 
Iceberg system table scans serialize Iceberg SDK `FileScanTask` objects
and the Java metadata scanner materializes rows from the task schema.
Doris did not pass the actually required system table columns into the
Iceberg SDK scan, so metadata tables such as `$files` and `$data_files`
were planned with the full schema. For files metadata tables, the full
schema can include virtual `readable_metrics` even when the SQL only
requests columns such as `file_size_in_bytes`.

This PR applies SDK top-level column projection only for Iceberg system
table scans. Normal Iceberg data table scans continue to rely on Doris
BE scan range params for column pruning.

- Add system-table-only `TableScan.select(...)` projection in
`IcebergScanNode` before planning the Iceberg SDK scan.
- Skip synthesized/global row id and Iceberg row lineage columns when
building the SDK projection list.
- Add regression coverage for ORC and Parquet Iceberg tables with
`map<boolean, boolean>` columns, verifying projected `$data_files` and
`$files` queries.

(cherry picked from commit d8e75cf)
@hello-stephen
hello-stephen force-pushed the agent/branch41-iceberg-system-projection branch from 4336f7e to 139c1ff Compare July 24, 2026 08:15
@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 30th, 2026 7:58 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30078404523

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

FE UT Coverage Report

Increment line coverage 60.98% (25/41) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

FE Regression Coverage Report

Increment line coverage 58.54% (24/41) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

[attempt-3-buildall-terminal]

Attempt 3 head 139c1ff311e45cc5e3c754e6fce191e061abaf84 has now completed the full required community buildall fan-out successfully:

  • FE UT #1004771: SUCCESS
  • Compile #1004772: SUCCESS
  • P0 #1004798: SUCCESS (6086 passed)
  • External #1004799: SUCCESS (568 passed)
  • cloud_p0 #1004800: SUCCESS (4661 passed)
  • Vault #1004801: SUCCESS
  • NonConcurrent #1004802: SUCCESS (395 passed)

The target occurrence external_table_p0.iceberg.test_iceberg_system_table_projection.test_iceberg_system_table_projection actually executed in External #1004799 and finished SUCCESS in 7057 ms. This closes the three-attempt branch-adaptation validation; remaining blockers are review/merge, followed by the required post-merge exact branch-4.1 rerun.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hello-stephen @yiguolei @suxiaogang223