-
Notifications
You must be signed in to change notification settings - Fork 6.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DB::Exception: Too many parts (600). Merges are processing significantly slower than inserts #3174
Comments
The main requirement about insert to Clickhouse: you should never send too many INSERT statements per second. Ideally - one insert per second / per few seconds. So you can insert 100K rows per second but only with one big bulk INSERT statement. When you send hundreds / thousands insert statements per second to *MergeTree table you will always get some errors, and it can not be changed by adjusting some settings. If you can't combine lot of inserts into one big bulk insert statement outside - then you should create Buffer table before *MergeTree table. |
Hello filimonov, I use the following command to insert the data into ClickHouse. cat *.clickhouse | clickhouse-client --query="INSERT INTO default.MyTable(....")... Since the number of files are large, I don't think many insert statement can be triggered. Question> Based on the above statement, how many insert statement has been issues? Thank you |
Sorry, didn't follow your first post. Can't you just insert your files one by one with tiny delays? for i in *.clickhouse; do
[ -f "$i" ] || break;
echo "Processing $i"
cat $i | clickhouse-client --query="INSERT INTO default.MyTable(....)" ... ;
sleep 0.1;
done |
Hello filimonov, I do have some issues on understanding the meaning of the following config. <merge_tree> Question 1> Why does clickhouse have to throw when the limit of the parts_to_throw_insert has been hit? Question 2> What is the real meaning of parts here? Is it true that the parts are the individual files located on /var/lib/clickhouse/data/default/MyTable/...? Question 3> By decreasing the value in <max_bytes_to_merge_at_max_space_in_pool>107374182400</max_bytes_to_merge_at_max_space_in_pool>, will this cause performance issue? My current workstation has 512GB memory. Since the machine has large memory capacity, I really want to improve the insert/merge performance if that is possible. Question 4> I have increased the value in <max_delay_to_insert>60</max_delay_to_insert>. Thank you |
|
Hi. And what can I do, if inserts are rare and small? I want near realtime processing, So I can't form greater batches. I have up to 1k rows in a batch (3kb-2mb on disc), one insert in 30 seconds. And it seems ClickHouse doesn't merge parts, collect 300 on this table, but it hasn't reached some minimal merge size (even if I stop inserts at all, parts are not merged). So insert process stucks - no more inserts are allowed due to max parts limits, but parts are not merged as total sum of partition sizes is small. |
We also have small inserts every 10 seconds (into dozens of separate tables), so we put a buffer table in front of every MergeTree table. So data is accumulated in memory, and when the buffer gets full it's flushed to disk. This way the number of parts that are created remains small. |
Note to my future self and and also others struggling. In my case there were mutations that looked quite heavy / stale. Check also for them then. @filimonov is there a way to for example list the parts that are waited for to be merged? |
If there are more than one part per partition and their sizes are smaller than maximum part size - that means they will be chosen for merge someday. The smaller part is the bigger chance that it will be merged soon. Check also system.mutations system.merges and system.part_log (last one should be enabled in config). |
@q0987 do you have any further questions? |
i have same problem, even i stop insert about 4 hours, no more part is merged ,most of them are small parts |
@WeiGangqiang also check for mutations |
how to check mutations is Done? |
|
Does that mean, that if the sizes are bigger than the maximum part size, they will never be merged? |
Yes.
Yes. But the merge of too large parts could be unreasonable because it will take a lot of CPU and IO resources for a long time and will provide zero speed-up for future selects. |
So, the same primary key could be in different parts forever? Because it is at least in one part that has reached its max size? |
yes |
So, we have to avoid that at any cost, right? Because the summing, aggregating, replacing trees and sparse indexes would be somehow broken? Perhaps making the partitioning key more granular, adding shards, or whatever? |
Nothing is broken. The opposite. |
|
@filimonov Could you please explain how recommended batch size was calculated?
I tried using less inserts with around 20kk rows and it gave me a good boost. |
ClickHouse client version 18.6.0.
Connected to ClickHouse server version 18.6.0 revision 54401.
Hello all,
I have been working on this issue for almost a week and still see the problem coming back.
The following changes have been made on config.xml:
The input data size: 177GB
The total number of input files: 2050
Average size of each file: 110MB
Command used to import the data:
cat *.clickhouse | clickhouse-client --query="INSERT INTO default.MyTable(....")...
Question 1> Based on the above command, how many insert statement generated?
Is this a bulk insertion?
After the change, the system still reports the same errors as below.
Question 2> What else should I try? Increase the size again?
Thank you very much
2018.09.19 10:25:02.114885 [ 49 ] executeQuery: Code: 252, e.displayText() = DB::Exception: Too many parts (600). Merges are processing significantly slower than inserts., e.what() = DB::Exception (from 127.0.0.1:46902) (in query: INSERT INTO default.MyTable(timestamp, ...) FORMAT RowBinary), Stack trace:
2018.09.19 10:25:02.162124 [ 49 ] ServerErrorHandler: Code: 99, e.displayText() = DB::Exception: Unknown packet 44 from client, e.what() = DB::Exception, Stack trace:
The text was updated successfully, but these errors were encountered: