[Improvement](scan) Update scanner limit controller (#61617)#61962
Merged
yiguolei merged 1 commit intoapache:branch-4.1from Apr 1, 2026
Merged
[Improvement](scan) Update scanner limit controller (#61617)#61962yiguolei merged 1 commit intoapache:branch-4.1from
yiguolei merged 1 commit intoapache:branch-4.1from
Conversation
This pull request introduces a shared atomic limit mechanism in the scanner execution context to more accurately enforce SQL `LIMIT` constraints across multiple concurrent scanners. The changes ensure that the row limit is respected globally, preventing over-scanning and improving resource efficiency. Key updates include the introduction of a thread-safe quota system, modifications to scanner scheduling and task pulling logic, and enhancements to debugging output. **Shared Limit Enforcement and Quota Management:** * Added a new atomic member `_remaining_limit` to `ScannerContext`, initialized with the SQL `LIMIT` value and decremented atomically as rows are scanned. Introduced the `acquire_limit_quota()` method for scanners to claim their share of the remaining quota. [[1]](diffhunk://#diff-0c9a817d45d8130ea3211189e1321d1275e22fd4a9a3fac2bd707b1cfeefa5e5R74) [[2]](diffhunk://#diff-0c9a817d45d8130ea3211189e1321d1275e22fd4a9a3fac2bd707b1cfeefa5e5R99) [[3]](diffhunk://#diff-3049f42cade971254aae07ced700d9a10b2505b03da743efea3270e63bd88dceR224-R229) * Modified the scanner scheduling logic in `ScannerScheduler::_scanner_scan` to check and acquire quota before processing blocks, ensuring that scanners stop or truncate blocks when the shared limit is exhausted. [[1]](diffhunk://#diff-ecdf52f3fb33b9018cc1aff92085e470087071b25d79efed8a849a289215d05fR235-R239) [[2]](diffhunk://#diff-ecdf52f3fb33b9018cc1aff92085e470087071b25d79efed8a849a289215d05fR276-R292) * Updated the pending scanner task pulling logic to avoid scheduling new scanners when the shared limit is depleted. **Scanner Completion and Task Management:** * Enhanced the logic for marking scanner context as finished: now, completion is triggered either when all scanners are done or when the shared limit is exhausted and no scanners are running. **Debugging and Observability:** * Improved the `debug_string()` output in `ScannerContext` to include the current value of `remaining_limit`, aiding in troubleshooting and monitoring. **Performance Optimization:** * Clarified and preserved the per-scanner small-limit optimization, ensuring that when the limit is smaller than the batch size, scanners return early to avoid unnecessary data processing.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Apr 1, 2026
yiguolei
added a commit
that referenced
this pull request
Apr 8, 2026
yiguolei
added a commit
that referenced
this pull request
Apr 8, 2026
BiteTheDDDDt
added a commit
to BiteTheDDDDt/incubator-doris
that referenced
this pull request
Apr 10, 2026
BiteTheDDDDt
added a commit
to BiteTheDDDDt/incubator-doris
that referenced
this pull request
Apr 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick from #61617