Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set max_final_threads to number of cores by default #47915

Merged
merged 1 commit into from Mar 23, 2023

Conversation

nickitat
Copy link
Member

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Setting max_final_threads would be set to number of cores at server startup (by the same algorithm as we use for max_threads). This improves concurrency of final execution on servers with high number of CPUs.


So the new behaviour is: max_final_threads != 0 => value left untouched, otherwise max_threads = cpus. additionally if max_final_threads > max_threads => max_threads streams will be used for reading.

@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-performance Pull request with some performance improvements label Mar 22, 2023
@KochetovNicolai KochetovNicolai self-assigned this Mar 23, 2023
@nickitat
Copy link
Member Author

Stateless tests (debug) [2/5] -

1103067:2023.03.22 22:50:41.732038 [ 15905 ] {5f8a4909-021c-4dca-8898-62add982d015} <Debug> executeQuery: (from [::1]:44762) (comment: 02232_dist_insert_send_logs_level_hung.sh) insert into function remote('127.2', currentDatabase(), in_02232) select * from numbers(1e6) (stage: Complete)
1103068:2023.03.22 22:50:41.734624 [ 15905 ] {5f8a4909-021c-4dca-8898-62add982d015} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE, REMOTE ON *.*
1103069:2023.03.22 22:50:41.735063 [ 15905 ] {5f8a4909-021c-4dca-8898-62add982d015} <Trace> Connection (127.2:9000): Connecting. Database: (not specified). User: default
1103076:2023.03.22 22:50:41.978349 [ 15905 ] {5f8a4909-021c-4dca-8898-62add982d015} <Trace> Connection (127.2:9000): Connected to ClickHouse server version 23.3.1.
1103100:2023.03.22 22:50:41.982639 [ 24529 ] {5a964b7a-e48a-4b5c-aba7-479855b3f3db} <Debug> executeQuery: (from [::ffff:127.0.0.1]:48458, initial_query_id: 5f8a4909-021c-4dca-8898-62add982d015) (comment: 02232_dist_insert_send_logs_level_hung.sh) DESC TABLE test_lng7d3ph.in_02232 (stage: Complete)
1103108:2023.03.22 22:50:41.992996 [ 15905 ] {5f8a4909-021c-4dca-8898-62add982d015} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
1103110:2023.03.22 22:50:42.000737 [ 24529 ] {093691bb-350a-4bf9-9e00-e374fd24b86a} <Debug> executeQuery: (from [::ffff:127.0.0.1]:48458, initial_query_id: 5f8a4909-021c-4dca-8898-62add982d015) (comment: 02232_dist_insert_send_logs_level_hung.sh) INSERT INTO test_lng7d3ph.in_02232 (key) VALUES (stage: Complete)
1193749:2023.03.22 22:54:50.477081 [ 15905 ] {5f8a4909-021c-4dca-8898-62add982d015} <Information> TCPHandler: Client has dropped the connection, cancel the query.
1193750:2023.03.22 22:54:50.477558 [ 18911 ] {5f8a4909-021c-4dca-8898-62add982d015} <Fatal> : Logical error: 'Pipeline for PushingPipelineExecutor was finished before all data was inserted'.
1193755:2023.03.22 22:54:50.479549 [ 24535 ] {} <Fatal> BaseDaemon: (version 23.3.1.1, build id: E311E03659E90DF873DD1DA003D8FC7273DFFA2F) (from thread 18911) (query_id: 5f8a4909-021c-4dca-8898-62add982d015) (query: insert into function remote('127.2', currentDatabase(), in_02232) select * from numbers(1e6)) Received signal Aborted (6)

looks irrelevant

@nickitat nickitat merged commit 4b635af into ClickHouse:master Mar 23, 2023
138 checks passed
DerekChia added a commit to DerekChia/ClickHouse that referenced this pull request Jun 7, 2023
`max_final_threads` is now set to the number of cores by default. See ClickHouse#47915
@DerekChia DerekChia mentioned this pull request Jun 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-performance Pull request with some performance improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants