[feat](fe) Support Cloud partition inverted format rollout - #66235
Open
hoshinojyunn wants to merge 1 commit into
Open
[feat](fe) Support Cloud partition inverted format rollout#66235hoshinojyunn wants to merge 1 commit into
hoshinojyunn wants to merge 1 commit into
Conversation
hoshinojyunn
requested review from
924060929,
CalvinKirs,
dataroaring,
englefly,
gavinchou,
luwei16,
morningman,
morrySnow,
mymeiyi and
starocean999
as code owners
July 29, 2026 09:58
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hoshinojyunn
changed the title
Contributor
Author
|
run buildall |
hoshinojyunn
changed the title
Contributor
TPC-H: Total hot run time: 29746 ms |
Contributor
TPC-DS: Total hot run time: 177881 ms |
Contributor
ClickBench: Total hot run time: 25.08 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
airborne12
reviewed
Jul 30, 2026
airborne12
left a comment
airborne12
left a comment
Member
There was a problem hiding this comment.
Two blocking lifecycle and rolling-upgrade issues are called out inline.
| TInvertedIndexFileStorageFormat invertedIndexFileStorageFormat = | ||
| origIdToInvertedIndexFileStorageFormat.get(origPartId); | ||
| if (invertedIndexFileStorageFormat != null) { | ||
| idToInvertedIndexFileStorageFormat.put(newPartId, invertedIndexFileStorageFormat); |
Member
There was a problem hiding this comment.
Blocking: this remaps the partition-owned format in FE metadata, but CloudRestoreJob.createReplicas() still builds every restored Cloud tablet with localTbl.getInvertedIndexFileStorageFormat() and localTbl.getBaseSchemaVersion(). After restoring a mixed V2/V3 table, PartitionInfo can therefore say that a partition is V2 while its new TabletSchema is created from the current table-level V3 format/schema. Please pass the restored partition's resolved format and corresponding backup schema/version into createTabletMetaBuilder() so the physical tablets and partition metadata are restored consistently. This needs coverage for both full-table restore and partition-only restore with mixed V2/V3 partitions.
| TAddOrDropPartitionsResult result; | ||
| try { | ||
| result = masterCallWithRetry(client -> client.addOrDropPartitions(request), | ||
| result = masterCallWithRetry(client -> client.addPartitionsForInsertOverwrite(request), |
Member
There was a problem hiding this comment.
Blocking: this introduces a new Thrift method without a rolling-upgrade compatibility path. During an FE rolling upgrade, a new client can call an old FE that does not implement addPartitionsForInsertOverwrite, resulting in TApplicationException.UNKNOWN_METHOD. Retrying the same method cannot recover, so INSERT OVERWRITE through a remote Doris catalog will fail until every target FE has been upgraded.
Please add explicit capability/version negotiation, or evolve the existing addOrDropPartitions request with an optional source-preserving flag and gate its use on server support. A blind fallback to the old method is not safe because it would create the temporary partition from the current table schema instead of preserving the source partition's physical schema. Please also add a new-client/old-server compatibility test.
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
deardeng
requested changes
Jul 30, 2026
| return partition; | ||
| } | ||
|
|
||
| public OlapFile.TabletMetaCloudPB getTabletMeta(long tabletId) throws DdlException { |
Contributor
There was a problem hiding this comment.
[major/P1] Avoid issuing one synchronous Meta Service RPC per partition while holding the table write lock.
collectPartitionInvertedIndexFileStorageFormats() performs O(partition_count) sequential GetTablet RPCs under olapTable.writeLock. For tables with thousands of partitions, the ALTER can hold the table metadata lock for a long time. Any RPC failure aborts the operation before journaling, and retrying starts the full scan again. Meta Service rate limiting only turns this into latency or failure and does not make the operation scalable. Please batch the tablet-meta reads and perform network I/O outside the table write lock, then reacquire the lock and validate that the table/partition metadata has not changed before committing.
hoshinojyunn
force-pushed
the
feat/partition
branch
4 times, most recently
from
July 31, 2026 08:54
647ffa6 to
1bb5304
Compare
Contributor
Author
|
run buildall |
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn
force-pushed
the
feat/partition
branch
from
July 31, 2026 10:46
1bb5304 to
6ee63cd
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28717 ms |
Contributor
TPC-DS: Total hot run time: 168826 ms |
Contributor
ClickBench: Total hot run time: 23.74 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
1 similar comment
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Cloud tables previously had one table-level
inverted_index_storage_format, so an online V2-to-V3 migration required all partitions to move together. This prevents a gradual rollout and makes it impossible to retain the format selected when an existing partition was created while new partitions use a new default.This PR adds the Cloud-only table property
partition.inverted_index_storage_format. It is the default for partitions created after the property is set. The property supports V2 and V3; V1 is rejected. Existing partitions and their tablet metas are not rewritten by the property update.PartitionInfofor every Cloud partition. The value is durable in journal/replay and is transferred through partition creation, recycle, recovery, and replacement metadata.CloudSchemaChangeJobV2, and creates each partition's base shadow tablet with the matching format and schema version. Non-base indexes keep their normal schema-change behavior.SHOW PARTITIONS,SHOW PARTITION <id>,SHOW PARTITION ID, and the partitions TVF exposeInvertedIndexStorageFormatfromPartitionInfo; they do not read tablet meta from MetaService for this value.partition.inverted_index_storage_formattakes precedence; the legacy table property remains the fallback; the resolved format is persisted per Cloud partition.test_partition_cloud_inverted_index_storage_format_rollout: legacy fallback, initial prefixed property, V2/V3 switching, and existing-partition retention.test_partition_cloud_inverted_index_storage_format_rollout: range, list, auto, and dynamic partition scenarios.test_partition_cloud_inverted_index_storage_format_rollout: mixed-format inverted-index queries, light schema change, rollup, and partition recovery.test_partition_cloud_inverted_index_storage_format_recycle_truncate: partition recycle/recover and table recover with mixed V2/V3 partitions.TRUNCATEandINSERT OVERWRITEcreate partitions from the current base-index schema and resolved partition-format default, rather than reading and preserving the source tablet schema.test_partition_cloud_inverted_index_storage_format_rollout: truncating and overwriting a retained V2 partition after the default changes to V3.test_partition_cloud_inverted_index_storage_format_recycle_truncate: whole-table truncate creates new V3 partitions.PartitionInfo, avoiding per-partition tablet-meta reads forSHOWand TVF output.test_cloud_show_inverted_index_storage_format;test_partitions_tvf(executed against the deployed Cloud FE).updateTablePropertiesno longer scans every partition with synchronous MetaServiceGetTabletRPCs to collect inverted-index formats.TRUNCATE TABLEandINSERT OVERWRITEdo not guarantee that their replacement or temporary partitions inherit the source partition's tablet schema. They create those partitions from the current base-index schema and resolved partition-format default. A replacement can therefore have a different schema, schema version, schema hash, and inverted-index storage format from the source partition. For example, after a V2 partition is retained and the default changes to V3, truncating or overwriting that V2 partition may create its replacement or temporary partition with the current base schema and V3 format.Release note
Cloud OLAP tables can roll out inverted-index file storage formats by partition. New partitions can use V3 while existing V2 partitions remain online and queryable. Replacement and temporary partitions created by truncate or insert overwrite use the current base-index schema and can therefore change format.
Check List (For Author)
Test
test_cloud_show_inverted_index_storage_formattest_partition_cloud_inverted_index_storage_format_rollouttest_partition_cloud_inverted_index_storage_format_recycle_truncatetest_partitions_tvf(against the deployed Cloud FE)Behavior changed:
partition.inverted_index_storage_format; it affects subsequently created partitions and exposes the resolved format in partition metadata commands. Truncate and insert overwrite use the current base-index schema rather than guaranteeing source-schema inheritance.Does this need documentation?
Check List (For Reviewer who merge this PR)