-
|
OK. Storage policies are great. Faster volume on front for hot data, slower storage for cold data and all that... What if my shards have different storage speed? Let's say shard 1 is of 3 replicas each with 25 SSD (hot volume on server) and 25 SFF SAS-2 HDD 10k RPM (cold volume on modular storage array - SAS cable attached to second RAID card) storage and my shard 2 is a node of slower LFF HDD volume with incredibly higher capacity. Storage policies could move data from hot to cold disks, but within a node. Could they do it from shard to shard, so as data becomes older and older it gets pushed from hot to cold then to "colder" storage on another shard. which has even slower volume, bur larger capacity? Any advice on how to approach sharding out for larger storage capacity as data grows? Not into S3, really. Would prefer the benefits of optical LAN between nodes instead. What if I add hotter storage of NVMe SSDs on a third RAID card on each node in shard1? This is getting too weird, because weight for shards does not make much sense to me, if I want it to operate in this way described above as weight makes data to be distributed across shards in the amount proportional to the shard weigh, so it kind of implies shards should have same storage speed. What am I missing here? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi @sv3t0sl4v , you should get some ideas from this article ClickHouse does not support automatic shard rebalancing. However, there are ways to rebalance shards in order of preference: If you need to rebalance existing data and you have partitioned your data, consider detaching partitions and manually relocating them to another node before reattaching to the new shard. This is more manual than subsequent techniques but may be faster and less resource-intensive. This is a manual operation and thus needs to consider the rebalancing of the data. |
Beta Was this translation helpful? Give feedback.
Hi @sv3t0sl4v , you should get some ideas from this article
ClickHouse does not support automatic shard rebalancing. However, there are ways to rebalance shards in order of preference:
If you need to rebalance existing data and you have partitioned your data, consider detaching partitions and manually relocating them to another node before reattaching to the new shard. This is more manual than subsequent techniques but may be faster and less resource-intensive. This is a manual operation and thus needs to consider the rebalancing of the data.