[improve](partition) Increase partition limit defaults to 20000 and add near-limit metrics#61511
Open
dataroaring wants to merge 5 commits intomasterfrom
Open
[improve](partition) Increase partition limit defaults to 20000 and add near-limit metrics#61511dataroaring wants to merge 5 commits intomasterfrom
dataroaring wants to merge 5 commits intomasterfrom
Conversation
… to 20000 and add near-limit metrics Raise the default limits from 500/2000 to 20000 for both max_dynamic_partition_num and max_auto_partition_num to better match modern production workloads. Add warning logs and Prometheus counter metrics (auto_partition_near_limit_count, dynamic_partition_near_limit_count) that fire when partition counts exceed 80% of their configured limits, enabling proactive monitoring before hard failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Pull request overview
This PR updates FE partition-limit defaults and adds observability to help detect near-limit partition growth before hard failures.
Changes:
- Increase default limits:
max_dynamic_partition_numandmax_auto_partition_numto 20000. - Add “near limit” warning logs when partition counts exceed 80% of configured limits.
- Register two new Prometheus counter metrics for near-limit events.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java | Adds near-limit warning + counter increment in auto-partition createPartition limit check. |
| fe/fe-core/src/main/java/org/apache/doris/metric/MetricRepo.java | Declares and registers new near-limit counter metrics. |
| fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java | Adds near-limit warning + counter increment during dynamic partition property analysis. |
| fe/fe-common/src/main/java/org/apache/doris/common/Config.java | Raises default values for dynamic and auto partition limits to 20000. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| result.setStatus(errorStatus); | ||
| LOG.warn("send create partition error status: {}", result); | ||
| return result; | ||
| } else if (partitionNum > autoPartitionLimit * 0.8) { |
Comment on lines
+4407
to
+4409
| LOG.warn("Table {}.{} auto partition count {} is approaching limit {} (>80%)." | ||
| + " Consider increasing max_auto_partition_num.", | ||
| db.getFullName(), olapTable.getName(), partitionNum, autoPartitionLimit); |
Comment on lines
+647
to
+653
| if (expectCreatePartitionNum > Config.max_dynamic_partition_num) { | ||
| throw new DdlException("Too many dynamic partitions: " | ||
| + expectCreatePartitionNum + ". Limit: " + Config.max_dynamic_partition_num); | ||
| } else if (expectCreatePartitionNum > Config.max_dynamic_partition_num * 0.8) { | ||
| LOG.warn("Dynamic partition count {} is approaching limit {} (>80%)." | ||
| + " Consider increasing max_dynamic_partition_num.", | ||
| expectCreatePartitionNum, Config.max_dynamic_partition_num); |
- Fix indentation alignment for multi-line log statements in DynamicPartitionUtil.java - Fix indentation alignment for multi-line log statements in FrontendServiceImpl.java - Ensure string concatenation lines align properly with checkstyle requirements
…eImpl - Align string concatenation line to 24 spaces to match checkstyle requirements - Ensure consistent indentation with DynamicPartitionUtil.java
- Swap MetricRepo/MetaContext imports to fix lexicographic order (CheckStyle) - Remove accidentally committed .orig files
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26983 ms |
TPC-DS: Total hot run time: 169628 ms |
Address Copilot review comments on #61511: - DynamicPartitionUtil: Capture Config.max_dynamic_partition_num into a local variable to avoid inconsistent reads of the mutable config, and use 'dynamicPartitionLimit * 8L / 10' (long arithmetic) instead of '* 0.8' to avoid implicit int-to-double conversion. - FrontendServiceImpl: Use 'autoPartitionLimit * 8 / 10' instead of '* 0.8' to keep the threshold comparison in integer math. The config was already captured to a local variable in the original PR.
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 27021 ms |
TPC-DS: Total hot run time: 169135 ms |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
max_dynamic_partition_numdefault from 500 to 20000 andmax_auto_partition_numfrom 2000 to 20000 to match modern production workloadsauto_partition_near_limit_count,dynamic_partition_near_limit_count) for monitoring/alertingTest plan
/metricsPrometheus endpointrate(doris_fe_auto_partition_near_limit_count[5m]) > 0🤖 Generated with Claude Code