[Enhancement] Support to configure the number of http worker threads on FE #37530
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![starrocks-cr[bot]](https://avatars.githubusercontent.com/in/404334?s=60&v=4){kind=small}
| */ | ||
| @ConfField | ||
| public static int http_worker_threads_num = 0; | ||
|
|
||
| /** | ||
| * The backlog_num for netty http server | ||
| * When you enlarge this backlog_num, you should ensure its value larger than |
There was a problem hiding this comment.
The most risky bug in this code is:
Using the default value of http_worker_threads_num as 0 is risky in some cases.
You can modify the code like this:
@ConfField
public static int http_worker_threads_num = 2 * Runtime.getRuntime().availableProcessors();
github-actions
bot
added
3.2
3.1
3.0
2.5
documentation
banmoy
force-pushed
the
http_worker_size
branch
2 times, most recently
from
97bf519
to
8934980
Compare
gengjun-git
reviewed
Dec 21, 2023
| @@ -935,6 +935,12 @@ Compaction Score 代表了一个表分区是否值得进行 Compaction 的评分 | |||
| - 含义：FE 节点上 HTTP 服务器的端口。 | |||
| - 默认值：8030 | |||
|
|
|||
| #### http_worker_threads_num | |||
|
|
|||
| - 含义：Http Server 用于处理 HTTP 请求的线程数。如果配置为非正数，线程数将设置为 CPU 核数的 2 倍。 | |||
There was a problem hiding this comment.
这样感觉更容易理解一些：
... 如果配置为0，线程数将设置为 CPU 核数的 2 倍。
There was a problem hiding this comment.
这里也想说明一下配置为负数的行为，改成 "如果配置为负数或0"是不是好些？
wyb
reviewed
Dec 21, 2023
banmoy
force-pushed
the
http_worker_size
branch
from
c8bbf19
to
3b77565
Compare
|
@wyb @gengjun-git PTAL again |
wyb
approved these changes
Dec 22, 2023
auto-merge was automatically disabled
December 22, 2023 11:31
Head branch was pushed to by a user without write access
banmoy
force-pushed
the
http_worker_size
branch
from
3b77565
to
4eb8589
Compare
Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
banmoy
force-pushed
the
http_worker_size
branch
from
4eb8589
to
9bd1c94
Compare
|
gengjun-git
approved these changes
Dec 25, 2023
[FE Incremental Coverage Report]✅ pass : 3 / 3 (100.00%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
|
@Mergifyio backport branch-3.2 |
|
@Mergifyio backport branch-3.1 |
|
@Mergifyio backport branch-2.5 |
✅ Backports have been created
|
✅ Backports have been created
|
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
Dec 25, 2023
…on FE (#37530) Why I'm doing: Http worker threads on FE are used to deal with http requests. Some requests may do heavy operations which will block threads, such as transaction stream load commit, and other operations that need to get lock. As a result, http server can't deal with other requests. Before optimizing these heavy operations, we first make the number of http worker threads to be configurable, and increase it as a candidate solution if the problem happens. What I'm doing: Add a configuration to adjust the number of http worker threads on FE Signed-off-by: PengFei Li <lpengfei2016@gmail.com> (cherry picked from commit d24bede)
Merged
18 tasks
mergify bot
pushed a commit
that referenced
this pull request
Dec 25, 2023
…on FE (#37530) Why I'm doing: Http worker threads on FE are used to deal with http requests. Some requests may do heavy operations which will block threads, such as transaction stream load commit, and other operations that need to get lock. As a result, http server can't deal with other requests. Before optimizing these heavy operations, we first make the number of http worker threads to be configurable, and increase it as a candidate solution if the problem happens. What I'm doing: Add a configuration to adjust the number of http worker threads on FE Signed-off-by: PengFei Li <lpengfei2016@gmail.com> (cherry picked from commit d24bede) # Conflicts: # docs/en/administration/Configuration.md # docs/zh/administration/Configuration.md
mergify bot
pushed a commit
that referenced
this pull request
Dec 25, 2023
…on FE (#37530) Why I'm doing: Http worker threads on FE are used to deal with http requests. Some requests may do heavy operations which will block threads, such as transaction stream load commit, and other operations that need to get lock. As a result, http server can't deal with other requests. Before optimizing these heavy operations, we first make the number of http worker threads to be configurable, and increase it as a candidate solution if the problem happens. What I'm doing: Add a configuration to adjust the number of http worker threads on FE Signed-off-by: PengFei Li <lpengfei2016@gmail.com> (cherry picked from commit d24bede) # Conflicts: # docs/en/administration/FE_configuration.md # docs/zh/administration/FE_configuration.md
mergify bot
pushed a commit
that referenced
this pull request
Dec 25, 2023
…on FE (#37530) Why I'm doing: Http worker threads on FE are used to deal with http requests. Some requests may do heavy operations which will block threads, such as transaction stream load commit, and other operations that need to get lock. As a result, http server can't deal with other requests. Before optimizing these heavy operations, we first make the number of http worker threads to be configurable, and increase it as a candidate solution if the problem happens. What I'm doing: Add a configuration to adjust the number of http worker threads on FE Signed-off-by: PengFei Li <lpengfei2016@gmail.com> (cherry picked from commit d24bede) # Conflicts: # docs/en/administration/FE_configuration.md # docs/zh/administration/FE_configuration.md
Closed
18 tasks
banmoy
added a commit
to banmoy/starrocks
that referenced
this pull request
Dec 26, 2023
…on FE (backport StarRocks#37530) Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
Merged
23 tasks
banmoy
added a commit
to banmoy/starrocks
that referenced
this pull request
Dec 26, 2023
…on FE (backport StarRocks#37530) Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
Merged
23 tasks
banmoy
added a commit
to banmoy/starrocks
that referenced
this pull request
Dec 26, 2023
…on FE (backport StarRocks#37530) Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
Merged
23 tasks
Zhangg7723
pushed a commit
to Zhangg7723/starrocks
that referenced
this pull request
Dec 26, 2023
…on FE (StarRocks#37530) Why I'm doing: Http worker threads on FE are used to deal with http requests. Some requests may do heavy operations which will block threads, such as transaction stream load commit, and other operations that need to get lock. As a result, http server can't deal with other requests. Before optimizing these heavy operations, we first make the number of http worker threads to be configurable, and increase it as a candidate solution if the problem happens. What I'm doing: Add a configuration to adjust the number of http worker threads on FE Signed-off-by: PengFei Li <lpengfei2016@gmail.com> Signed-off-by: 张敢 <zhanggan@deepexi.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Http worker threads on FE are used to deal with http requests. Some requests may do heavy operations which will block threads, such as transaction stream load commit, and other operations that need to get lock. As a result, http server can't deal with other requests. Before optimizing these heavy operations, we first make the number of http worker threads to be configurable, and increase it as a candidate solution if the problem happens.
What I'm doing:
Add a configuration to adjust the number of http worker threads on FE
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: