[feature](iceberg) Support nested column schema change#65329
Conversation
Support nested Iceberg column paths in external schema change operations and wire parser/analyzer metadata ops through ColumnPath.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated review completed for PR 65329.
Requesting changes for two issues:
- Nested Iceberg schema evolution resolves user-supplied paths with exact Iceberg field-name lookup even though Doris exposes Iceberg metadata in lower case, so mixed-case schemas created by Spark or another Iceberg writer cannot be altered through the Doris-visible path.
- The new regression suite validates deterministic
DESCand ordered query output with Groovy assertions only and does not add the generated.outbaseline expected by the Doris regression framework.
Review checkpoints:
- Scope/head verified against PR head
951be9baa362ef7b91d21ae95afcef66736a7432; existing inline review context was empty. - Changed FE parser/command/metadata-op paths, Iceberg nested schema-change flow, and added tests were reviewed.
- Security review was not applicable to this PR.
- Validation is static only: this checkout is missing
thirdparty/installedandthirdparty/installed/bin/protoc, so I did not run FE or regression tests.
Subagent conclusions:
optimizer-rewritereportedNO_NEW_VALUABLE_FINDINGS.tests-session-configproposed TSC-1, which was verified and submitted as the regression.outbaseline inline comment.- Final convergence round completed with both live subagents replying
NO_NEW_VALUABLE_FINDINGSfor the same two-comment final set.
User focus: no additional user-provided review focus was supplied.
| FROM ${tableName} | ||
| ORDER BY id | ||
| """ | ||
| assertTrue(queryRows.size() == 2, queryRows.toString()) |
There was a problem hiding this comment.
Please convert these deterministic result checks to qt_/order_qt_ output checks and add the generated .out file. This suite validates the final DESC schema and the ordered query result only through Groovy assertTrue calls, and there is currently no regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out. That leaves the user-visible output outside the regression framework baseline checks, contrary to the repo rule for determined expected results.
| currentPath += part; | ||
|
|
||
| if (currentType.isStructType()) { | ||
| NestedField field = currentType.asStructType().field(part); |
There was a problem hiding this comment.
Please make nested path resolution case-insensitive and return the canonical Iceberg path before calling UpdateSchema. Doris lower-cases Iceberg metadata when it builds Column/StructField, so a Spark-created schema such as Info STRUCT<Metric: INT> is exposed to users as info.metric. This resolver uses StructType.field(part), which is exact-name lookup, and modifyColumn also calls exact schema.findField(columnPath.getFullPath()); as a result ALTER TABLE ... MODIFY COLUMN info.metric ... and adding a child under info reject the existing field even though that is the only Doris-visible spelling. Please resolve with Iceberg case-insensitive APIs, then pass the canonical path names to validation and updateSchema.*.
2 participants
Support nested Iceberg column paths in external schema change operations and wire parser/analyzer metadata ops through ColumnPath.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)