-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[improve](cloud) reduce fe call get_version to meta_service #60467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
a0061e3 to
94cf726
Compare
|
run buildall |
TPC-H: Total hot run time: 31944 ms |
ClickBench: Total hot run time: 27.7 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
94cf726 to
a2edeef
Compare
|
run buildall |
TPC-H: Total hot run time: 33492 ms |
ClickBench: Total hot run time: 28.08 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
a2edeef to
5fda285
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR enhances cloud-mode version management by caching and propagating table/partition visible versions within FE, reducing direct get_version RPCs to the meta-service. It also extends meta-service protocols to return table-level version information and wires FE components to maintain and synchronize these versions across frontends.
Changes:
- Extends Thrift/Proto APIs to carry table and partition version information (including new FE RPC
updateCloudVersionand additional fields inCommitTxnResponse,IndexResponse, andPartitionResponse). - Introduces FE-side tools (
CloudUpdateVersionTool,CloudTableAndPartitionVersionChecker) and related wiring to cache, update, and synchronize table/partition versions in cloud deployments, including changes inOlapTable,CloudPartition, and transaction/catalog paths. - Updates meta-service transaction and partition/index commit flows to compute, persist, and return table versions, with corresponding unit tests validating the new behavior.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| gensrc/thrift/FrontendService.thrift | Adds cloud version-related structs and the updateCloudVersion RPC for FE-to-FE version propagation; note the new API surface. |
| gensrc/proto/cloud.proto | Extends CommitTxnResponse, IndexResponse, and PartitionResponse to include table and partition version metadata used by FE caches. |
| fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java | Implements updateCloudVersion handler to delegate version updates to CloudUpdateVersionTool on non-master FEs. |
| fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | Changes default table/partition version cache TTLs to Long.MAX_VALUE to favor long-lived caches in cloud mode. |
| fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java | Extends afterCreatePartitions signature and call sites to pass OlapTable so cloud code can update and propagate table versions after partition operations. |
| fe/fe-core/src/main/java/org/apache/doris/common/proc/PartitionsProcDir.java | Refactors partition version retrieval to use a helper that, in cloud mode, fetches snapshot partition versions (and logs RPC issues) instead of reading directly from in-memory state. |
| fe/fe-core/src/main/java/org/apache/doris/common/ClientPool.java | Adds a dedicated frontendVersionPool for version-sync RPCs, separate from heartbeat and general FE pools. |
| fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java | Refactors post-commit handling to compute and cache table/partition versions, then propagate them to followers via CloudUpdateVersionTool. |
| fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java | Makes partition/index commit methods return table versions and uses them to update local caches and notify other FEs after create/alter/drop operations. |
| fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudUpdateVersionTool.java | New utility coordinating async version propagation from master to other FEs, including building Thrift requests and using the new frontendVersionPool. |
| fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTableAndPartitionVersionChecker.java | New daemon that periodically checks table versions vs. meta-service and refreshes partition versions where the table has advanced. |
| fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java | Enhances cloud partitions with helper methods to resolve owning tables and adds table-level version locking around cached partition version reads. |
| fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java | Wires in the new checker and update tool and starts the checker in master-only daemon threads. |
| fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java | Passes OlapTable into afterCreatePartitions so cloud catalog can update and sync table versions after dynamic partition creation. |
| fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java | Adds table-version cache helpers and a ReadWriteLock used to coordinate consistent updates of table and partition versions in cloud mode. |
| fe/fe-common/src/main/java/org/apache/doris/common/Config.java | Introduces tunables for cloud version checking intervals, get/update version thread counts, and batch size. |
| cloud/test/txn_lazy_commit_test.cpp | Extends lazy-commit tests to assert correctness of returned table versions and their consistency with get_version. |
| cloud/test/meta_service_versioned_read_test.cpp | Adds tests validating table version values in CommitTxn, commit_index, and commit_partition responses under versioned read/write. |
| cloud/test/meta_service_test.cpp | Expands meta-service tests to cover table version increments and to verify table_versions fields in responses and KV state. |
| cloud/src/meta-service/meta_service_txn.cpp | Updates commit paths to enable versionstamps when needed and to populate per-table version information using either stored table versions or KV transaction versionstamps. |
| cloud/src/meta-service/meta_service_partition.cpp | Extends index/partition commit/drop flows to optionally enable versionstamps and to return updated table versions in the protobuf responses. |
| cloud/src/meta-service/meta_service.h | Adjusts commit_partition_internal signature to accept a PartitionResponse* so that it can populate table versions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
Show resolved
Hide resolved
5fda285 to
46584f1
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 31921 ms |
ClickBench: Total hot run time: 27.74 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
16d67b1 to
2eef113
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 31217 ms |
ClickBench: Total hot run time: 28.99 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
| public static final String CLOUD_CLUSTER = "cloud_cluster"; | ||
| public static final String COMPUTE_GROUP = "compute_group"; | ||
| public static final String DISABLE_EMPTY_PARTITION_PRUNE = "disable_empty_partition_prune"; | ||
| public static final String CLOUD_FORCE_SYNC_VERSION = "cloud_force_sync_version"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
force_sync_version_in_cloud_mode is better?
2eef113 to
5c92b67
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 31058 ms |
ClickBench: Total hot run time: 28.71 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
d6c65c5 to
64e5fd5
Compare
|
run buildall |
64e5fd5 to
22e5f57
Compare
|
run buildall |
TPC-H: Total hot run time: 30468 ms |
ClickBench: Total hot run time: 28.11 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. 🤖 Generated with Claude Code - If this code review was useful, please react with 👍. Otherwise, react with 👎. |
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)