branch-4.1: [feature](fe) Add partition filter sql block rule (#62196)#63724
Open
CalvinKirs wants to merge 1 commit into
Open
branch-4.1: [feature](fe) Add partition filter sql block rule (#62196)#63724CalvinKirs wants to merge 1 commit into
CalvinKirs wants to merge 1 commit into
Conversation
Today SQL_BLOCK_RULE can block SQL by regex/sqlHash or by scan scale
thresholds such as
`partition_num`, `tablet_num`, and `cardinality`, but it cannot directly
reject queries that
scan a partitioned table without using any partition filter.
This PR adds a new scan-based SQL block rule option:
`require_partition_filter`.
When this option is enabled on a rule, Doris rejects scans on supported
partitioned tables if
the query does not contain any usable partition predicate. This helps
prevent accidental full
partition scans caused by missing partition filters.
The rule is intended for workloads where partition filters are mandatory
for safety or cost
control.
### User-visible behavior
Users can now create a SQL block rule like this:
```sql
CREATE SQL_BLOCK_RULE require_partition_filter_rule
PROPERTIES(
"require_partition_filter" = "true",
"global" = "true",
"enable" = "true"
);
Or bind it to specific users:
CREATE SQL_BLOCK_RULE require_partition_filter_rule
PROPERTIES(
"require_partition_filter" = "true",
"global" = "false",
"enable" = "true"
);
SET PROPERTY FOR 'test_user' 'sql_block_rules' =
'require_partition_filter_rule';
If a supported partitioned table is scanned without any partition
predicate, Doris returns an
error like:
sql hits sql block rule: require_partition_filter_rule, missing
partition filter
### Scope
This new rule currently applies to:
- Partitioned internal tables
- Partitioned Hive external tables
This rule does not apply to:
- Non-partitioned internal tables
- Non-partitioned Hive tables
- Iceberg tables
- Other external table types not wired into this rule yet
### Rule semantics
The new property is:
- require_partition_filter = true|false
Behavior:
- The rule only takes effect when require_partition_filter=true
- For supported partitioned tables, the scan is allowed if the query
hits any partition column
in partition pruning predicates
- Filters on non-partition columns do not count
- The rule applies to scan-producing statements, such as:
- SELECT
- INSERT INTO ... SELECT ...
- EXPLAIN is not blocked
Examples:
Blocked:
SELECT * FROM part_tbl;
SELECT * FROM part_tbl WHERE non_partition_col = 1;
INSERT INTO dst SELECT * FROM part_tbl;
Allowed:
SELECT * FROM part_tbl WHERE dt = '2026-04-09';
SELECT * FROM part_tbl WHERE dt = '2026-04-09' AND hh = '10';
INSERT INTO dst SELECT * FROM part_tbl WHERE dt = '2026-04-09';
### Compatibility and validation
require_partition_filter is treated as a scan-based SQL block condition.
It can be used together with existing scan-based limits such as:
- partition_num
- tablet_num
- cardinality
It cannot be mixed with text-based block conditions such as:
- sql
- sqlHash
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Member
Author
|
run feut |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#62196