New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't fall-back to in-order pool when max_streams = 1
for remote fs
#57334
Don't fall-back to in-order pool when max_streams = 1
for remote fs
#57334
Conversation
This is an automated comment for commit 626668c with description of existing statuses. It's updated for the latest CI running ✅ Click here to open a full report in a separate page Successful checks
|
clickhouse-benchmark -q "WITH RANDOM_SET AS (SELECT rand32() FROM numbers(20)) SELECT distinct _part FROM test_reading WHERE k IN RANDOM_SET and _part = '1_62735_87597_5' SETTINGS allow_experimental_parallel_reading_from_replicas=0, enable_filesystem_cache=0" --cumulative -i 1000
before:
Queries executed: 1000.
localhost:9000, queries: 1000, QPS: 5.409, RPS: 752087.598, MiB/s: 13.627, result RPS: 0.000, result MiB/s: 0.000.
0.000% 0.090 sec.
10.000% 0.142 sec.
20.000% 0.155 sec.
30.000% 0.165 sec.
40.000% 0.175 sec.
50.000% 0.182 sec.
60.000% 0.190 sec.
70.000% 0.200 sec.
80.000% 0.213 sec.
90.000% 0.228 sec.
95.000% 0.240 sec.
99.000% 0.295 sec.
99.900% 0.417 sec.
99.990% 0.443 sec.
after:
Queries executed: 1000.
localhost:9000, queries: 1000, QPS: 6.436, RPS: 891218.317, MiB/s: 16.147, result RPS: 0.000, result MiB/s: 0.000.
0.000% 0.064 sec.
10.000% 0.114 sec.
20.000% 0.127 sec.
30.000% 0.136 sec.
40.000% 0.143 sec.
50.000% 0.151 sec.
60.000% 0.160 sec.
70.000% 0.169 sec.
80.000% 0.181 sec.
90.000% 0.199 sec.
95.000% 0.216 sec.
99.000% 0.259 sec.
99.900% 0.284 sec.
99.990% 0.330 sec. clickhouse-benchmark -q "select * from hits where CounterID < 30 SETTINGS allow_experimental_parallel_reading_from_replicas=0, enable_filesystem_cache=0" --cumulative -i 1000
before:
Queries executed: 1000.
localhost:9000, queries: 1000, QPS: 7.325, RPS: 60009.000, MiB/s: 0.268, result RPS: 109.880, result MiB/s: 0.040.
0.000% 0.061 sec.
10.000% 0.073 sec.
20.000% 0.079 sec.
30.000% 0.083 sec.
40.000% 0.086 sec.
50.000% 0.090 sec.
60.000% 0.094 sec.
70.000% 0.099 sec.
80.000% 0.107 sec.
90.000% 0.124 sec.
95.000% 0.280 sec.
99.000% 1.071 sec.
99.900% 1.082 sec.
99.990% 1.082 sec.
after:
Queries executed: 1000.
localhost:9000, queries: 1000, QPS: 12.266, RPS: 100483.770, MiB/s: 0.449, result RPS: 183.991, result MiB/s: 0.067.
0.000% 0.037 sec.
10.000% 0.045 sec.
20.000% 0.051 sec.
30.000% 0.055 sec.
40.000% 0.058 sec.
50.000% 0.061 sec.
60.000% 0.065 sec.
70.000% 0.070 sec.
80.000% 0.074 sec.
90.000% 0.084 sec.
95.000% 0.091 sec.
99.000% 1.031 sec.
99.900% 1.061 sec.
99.990% 1.068 sec. |
Benchmark results do not look as good as expected. In first case you made 20 consequential reads as I understand before changes. And diff is 30-100ms only. |
it represents a complex case. total number of marks is low (under 20), so there are only 2-3 tasks anyway. so new prefetches don't add match. and mark ranges are sparse within each task, so to get maximum parallelism you need to split the whole set of marks to read on very small tasks. than each one can be prefetched. I did a quick experiment and exec time indeed became less than 50ms. |
86f7af0
to
ffb9fc2
Compare
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Now we use default read pool for reading from external storage when
max_streams = 1
. It is beneficial when read prefetches are enabled.