Skip to content

branch-4.1: [opt](scan) Schedule scanners up to max concurrency - #66106

Closed
HappenLee wants to merge 1 commit into
apache:branch-4.1from
HappenLee:pick-scan-schedule-branch-4.1
Closed

branch-4.1: [opt](scan) Schedule scanners up to max concurrency#66106
HappenLee wants to merge 1 commit into
apache:branch-4.1from
HappenLee:pick-scan-schedule-branch-4.1

Conversation

@HappenLee

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary: Cherry-pick "Schedule scanners up to max concurrency" from opt_perf_4.1 (commit 2903457) to branch-4.1.

Previously ScannerContext::_get_margin computed two margins (per-scan-operator _min_scan_concurrency guarantee and scheduler-level _min_scan_concurrency_of_scan_scheduler guarantee) and took the max, which under-utilizes scan concurrency. This change schedules scanners directly up to _max_scan_concurrency (_max_scan_concurrency - _num_scheduled_scanners), keeping the low-memory-mode cap, so scanners are scheduled at full concurrency and scan performance improves.

Release note

None

Check List (For Author)

  • Test: No need to test (clean cherry-pick, verified on opt_perf_4.1)
  • Behavior changed: No
  • Does this need documentation: No

@HappenLee
HappenLee requested a review from yiguolei as a code owner July 27, 2026 09:55
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee

Copy link
Copy Markdown
Contributor Author

Close: picked the wrong commit (2903457 has been reverted on opt_perf_4.1). Will re-pick 3d74daa (Reapply "opt Boost scanner concurrency under starvation") instead.

@HappenLee HappenLee closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants