[improvement](fe) Reuse external scan tasks within a statement - #66473
[improvement](fe) Reuse external scan tasks within a statement#66473924060929 wants to merge 2 commits into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
b848439 to
cecc334
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
|
/review |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
ff12490 to
7327aa2
Compare
|
run buildall |
7327aa2 to
adc7745
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
adc7745 to
01b5a68
Compare
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z. Please trigger /review again after that time. |
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Queries with repeated equivalent external-table relations, such as UNION branches over the same snapshot, plan the same remote files multiple times. This duplicates Iceberg, Paimon, Hudi, and Hive metadata work and can add seconds to FE planning. Reuse connector-native scan tasks with connector-specific semantic keys and single-flight loading within one statement execution. Keep Iceberg streaming batch paths and transactional Hive paths uncached to preserve their memory and transaction semantics. Isolate each prepared-statement execution with a generation handle and invalidate it when execution ends so delayed workers cannot pollute the next execution and idle prepared statements do not retain task lists.
### Release note
Reuse equivalent external-table scan planning results within one statement execution.
### Check List (For Author)
- Test: Unit Test
- StatementContextTest
- IcebergScanNodeTest
- PaimonScanNodeTest
- HiveScanNodeTest
- HudiScanNodeTest
- Behavior changed: Yes, repeated equivalent external scans reuse statement-scoped planning results while transactional and streaming paths retain existing behavior
- Does this need documentation: No
Issue Number: None
Related PR: None
Problem Summary: Statement-scoped external scan task reuse lacked connector-level tests proving that equivalent scans share planning work while snapshots, options, predicates, projections, instants, partitions, reader modes, and transactional or streaming paths remain correctly isolated. Add production-entry coverage for Iceberg, Paimon, Hudi, Hive, and repeated prepared-statement executions, including mutable split isolation and cache cleanup behavior. The tests also exposed a nullable Hudi split weight copy that could unbox null after a cache hit; preserve the nullable field when copying cached splits. Expose the Iceberg manifest loader to a test subclass without changing runtime behavior so the production manifest cache wrapper is exercised directly.
None
- Test: Unit Test
- run-fe-ut.sh: 122 targeted tests passed
- run-fe-ut.sh: 12 strengthened Hive and Hudi tests passed
- Maven Checkstyle: 0 violations
- Behavior changed: No, except preserving the existing standard split-weight fallback when a copied Hudi split has no explicit weight
- Does this need documentation: No
01b5a68 to
8db50d1
Compare
|
run buildall |
FE Regression Coverage ReportIncrement line coverage |
3 participants
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Queries with repeated equivalent external-table relations, such as UNION branches over the same table snapshot, independently plan the same remote files for every branch. On metadata-heavy Iceberg tables this repeats manifest planning and can add seconds to FE planning time; Paimon, Hudi, and Hive have the same class of duplicated split-planning work.
This change introduces a statement-execution-scoped, single-flight external scan-task cache. Each connector builds a semantic key from the facts that affect its split set:
Equivalent scans reuse connector-native tasks within one statement execution. Iceberg streaming/lazy batch paths and transactional Hive paths remain uncached to preserve their memory and transaction semantics. Mutable Hudi/Hive scheduling objects are copied before use so one scan cannot mutate cached state observed by another scan.
The cache is represented by an execution-generation handle captured by each ScanNode. PreparedStatement execution reset swaps in a new generation and invalidates the old one, so delayed workers cannot repopulate or block the next execution. Statement close and binary prepared-execution completion invalidate and clear retained task lists, preventing idle PreparedStatements from retaining split metadata. Loader failures are propagated to all waiters and are not cached, allowing retry.
Release note
Reuse equivalent Iceberg, Paimon, Hudi, and Hive scan-planning results within one statement execution to reduce repeated external metadata work.
Check List (For Author)
IcebergScanNodeTestPaimonScanNodeTestHudiScanNodeTestHiveScanNodeTestStatementContextTestbranch-4.1base