Make materialized view "failable" #34649
jeandeniscosta
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
we are managing a Clickhouse environment which purpose is to have a very small number of "input" tables (currently 2), upon which we would like to build different materialized views suited to different purposes.
raw -> MV1 -> table 1
raw -> MV2 -> table 2
raw -> MV3 -> table 3
etc.
This works quite well (we have more than 20 of them now), however we currently are at risk because any "failing" materialized view will break the whole pipeline. This is not an issue in production where we want to keep a strict control on what happens,
however for multi-team developement fluidity, we'd like to allow for other non-clickhouse-savvy teams to "test" stuff... at the obvious risk of creating bad materialized view with several kinds of errors, which immediately breaks the whole testing environment feed.
Is there a possibility to state that failure to insert into a table via a materialized view should NOT cause the whole INSERT command ? Ideally on a per-MV basis?
This would break the idempotency of inserts, but at the same time give us the opportunity to "open up" an already existing sandbox cluster to teams.
Beta Was this translation helpful? Give feedback.
All reactions