Ramifications and status of 'Separation of Storage and Compute' (mentioned in 2022 Roadmap)

[tasdflkjweio]

Per the 2022 Roadmap, a goal is to separate storage from compute. If it's what I think it is, it would mean we can dynamically scale the ClickHouse server horizontally, effectively meaning the compute is stateless. Storage then becomes the point of contention, so I could use some high-throughput, replicated managed disk offering. Is this the goal?

I'm in the process of migrating our environment to Azure, and would love to configure ClickHouse in this way. Has this feature been completed and, if so, is there any documentation on it? To set this up, would it be just another engine type?

As an aside, I'm very excited about the phenomenal work you've been putting into ClickHouse. Thank you for your commitment to making such a high-quality product!

[DerekChia]

hey @tasdflkjweio ClickHouse already support the usage of S3 and HDFS as table engine. By using them, you are effectively separating compute and storage, and you can scale either of them independently.

Reference:

• https://clickhouse.com/docs/en/engines/table-engines/integrations/s3
• https://clickhouse.com/docs/en/engines/table-engines/integrations/hdfs

[tasdflkjweio]

Thank you. It's unclear from what you've shared whether these are related to the 2022 roadmap item or not. Are you able to confirm?

[DerekChia]

You can check out the changelog page for 2022 here and correlate the relevant updates.

For example, I spotted several features, fixes and performance improvements related to "parallel reading" (first point under Separation of Storage and Compute). Just search for the word "parallel" on that page and have fun reading!

[tasdflkjweio]

I'd already looked through the changelog, but it doesn't answer the question of what the intent for the roadmap item is. Like what does separate compute from storage mean specifically in the context of ClickHouse? My question was aimed at what is the goal/feature being developed. It's difficult to gleam that from a bunch of commits, which may be minor steps along the way to some broader objective.

Like if you were in LA and were driving to NYC, someone might be able to figure out your overall destination by taking a look at your many, many turns along the way, but it'd be easier for them to understand if you just told them that you're headed to New York.

Hope the questioning makes sense, I appreciate your responses!

[tasdflkjweio]

I guess no clarity will be given

[chloro-pn]

I guess no clarity will be given

Maybe this can help you, if you can understand Chinese

cloudnativecube/octopus#48