Exactly-once write semantics #28270
-
Hi Team, I'd like to understand the proper way to ensure exactly-once ingestion into a sharded deployment of ClickHouse (multiple shards, with the same ReplicatedMergeTree tables configured in each shard). My understanding of the process looks as follows (based on the documentation/talks/etc, haven't studied the code though):
This raises two issues (according to my understanding):
Therefore to ensure exactly-once write semantics the following measures needs to be taken:
Please let me know what's wrong with my understanding. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
You can retry the whole INSERT to Distributed table. Moreover, if you use asynchronous inserts to Distributed table (which is by default) and don't enable distributed_directory_monitor_batch_inserts, the Distributed table will perform retries automatically for you, until the data will be inserted exactly once. |
Beta Was this translation helpful? Give feedback.
You can retry the whole INSERT to Distributed table.
The batch will be split between shards and data will be send to shards as usual.
If you have deterministic sharding key, then data will be split between shards deterministically.
Then, on every shard that already have this block of data, it will be skipped, and inserted on shards that don't have data yet.
Moreover, if you use asynchronous inserts to Distributed table (which is by default) and don't enable distributed_directory_monitor_batch_inserts, the Distributed table will perform retries automatically for you, until the data will be inserted exactly once.