Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using distributed tables with Materialized views #2411

Merged

Conversation

jehuty0shift
Copy link
Contributor

@jehuty0shift jehuty0shift commented May 24, 2018

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

This commit allow to use materialized view to ingest data to distributed tables. The pipeline ingestion is the following:
Kafka consumer node => Materialized View => Distributed Table.
Using Distributed Table instead of local tables has two main advantages:

  1. the underlying database can scale independantly of the kafka consumers nodes. When using only local tables, it is more complicated to achieve this.
  2. the data stays evenly/distributed in case of a kafka consumer node failure. this is not the case when you use only local tables.
    These two points allow a highly available and balanced ingestion pipeline.

Unlike previous attempts to allow this feature, the change in code is minimal (and fix a potential NULL pointer dereferencing) since only StorageDistributed.cpp is modified, however i wrote an integration test (heavily inspired by another one) to ensure that this functionality does not break into the future.

@jehuty0shift jehuty0shift changed the title NULL check when using distributed tables with Materialized views Using distributed tables with Materialized views May 24, 2018
@jehuty0shift jehuty0shift reopened this May 30, 2018
@jehuty0shift jehuty0shift reopened this May 30, 2018
@alexey-milovidov alexey-milovidov merged commit 3d518e4 into ClickHouse:master May 31, 2018
@alexey-milovidov
Copy link
Member

Thank you!

@filimonov filimonov added the comp-matview Materialized views label May 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp-matview Materialized views
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants