Support Open Source ClickHouse Deployments #1496

Pipboyguy · 2024-06-19T20:13:58Z

Description

This will improves support for self-managed ClickHouse open source deployments, while maintaining compatibility with ClickHouse Cloud deployments.

Allow explicitly specifying the desired engine via table_engine_type in clickhouse_adapter. Valid types are merge_tree, replicated_merge_tree, shared_merge_tree, stripe_log, tiny_log
Default to MergeTree if no engine is specified, which now works for both Cloud and self-managed deployments
Update tests to check for the appropriate engine based on annotation
CH cloud has "date_time_input_format" set to "best_effort", while OS deployments don't. This caused some tests to fail for OS deployments. We override setting for clickhouse_connect sessions.
Clarify some networking details for making dlt work with OS deployments in docs

Related Issues

Additional Context

Note that this change does not include support for specifying replication, ZooKeeper or shard details for the ReplicatedMergeTree engine. Users requiring those customizations can continue to specify the full engine definition in their configuration.

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

netlify · 2024-06-19T20:14:14Z

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	`477e815`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/669175d24f3c6800089ff0d4
😎 Deploy Preview	https://deploy-preview-1496--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

…onfiguration setting~ Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

jorritsandbrink

Code itself looks good to me. Just two comments regarding test coverage and the use of sentinel tables.

docs/website/docs/dlt-ecosystem/destinations/clickhouse.md

tests/load/clickhouse/test_clickhouse_adapter.py

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

# Conflicts: # dlt/destinations/impl/clickhouse/clickhouse.py # dlt/destinations/impl/clickhouse/sql_client.py # tests/load/clickhouse/test_clickhouse_adapter.py

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

jorritsandbrink

Thanks for addressing the comments, LGTM!

Just this comment regarding state interaction is still open: #1496 (comment)

This reverts commit 409487c.

This reverts commit e1ac1ce.

rudolfix

@Pipboyguy @jorritsandbrink this is really good. thanks for working on it and all reviews. from my side

pls keep sentinel table. I elaborated quite a bit why
consider adding the table engine in the clickhouse configuration to have easy global swithc

dlt/destinations/impl/clickhouse/clickhouse.py

docs/website/docs/dlt-ecosystem/destinations/clickhouse.md

…ides Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

rudolfix

LGTM! I have just one small suggestion which may not make sense. let's wait for @jorritsandbrink review and we can merge it

rudolfix · 2024-07-05T08:13:30Z

dlt/destinations/impl/clickhouse/sql_client.py

+        self.execute_sql(f"""
+            CREATE TABLE {sentinel_table_name}
+            (_dlt_id String NOT NULL PRIMARY KEY)
+            ENGINE=MergeTree


I'm probably nitpicking but should it use config.table_engine_type here?

@rudolfix Thanks for the spot!

Changed to config table engine:

dlt/dlt/destinations/impl/clickhouse/sql_client.py

Lines 116 to 121 in 9c67c65

sentinel_table_type = cast(TTableEngineType, self.credentials.table_engine_type)

self.execute_sql(f"""

CREATE TABLE {sentinel_table_name}

(_dlt_id String NOT NULL PRIMARY KEY)

ENGINE={TABLE_ENGINE_TYPE_TO_CLICKHOUSE_ATTR.get(sentinel_table_type)}

COMMENT 'internal dlt sentinel table'""")

jorritsandbrink

Overall it looks great, just a comment regarding organization of config attributes.

jorritsandbrink · 2024-07-05T09:55:40Z

dlt/destinations/impl/clickhouse/configuration.py

@@ -36,6 +40,8 @@ class ClickHouseCredentials(ConnectionStringCredentials):
    """Timeout for sending and receiving data. Defaults to 300 seconds."""
    dataset_table_separator: str = "___"
    """Separator for dataset table names, defaults to '___', i.e. 'database.dataset___table'."""
+    table_engine_type: Optional[TTableEngineType] = "merge_tree"


Why do you define this attribute under ClickHouseCredentials?

I think it makes more sense to put it under ClickHouseClientConfiguration. Same for dataset_table_separator and dataset_sentinel_table_name. They aren't credentials.

As a reference: synapse has a default_table_index_type that's similar to table_engine_type, which is defined in the destination client class:

dlt/dlt/destinations/impl/synapse/configuration.py

Line 41 in 60c7327

default_table_index_type: Optional[TTableIndexType] = "heap"

@Pipboyguy yes I agree with @jorritsandbrink
this is probably due to sql_client not seeing config but credentials - but you can pass additional arguments to it
so my take is that we move it and if it is hard then we'll need to change how sql_client is instantiated. I can help in that case

Totally agree. These don't belong in credentials.

Please find latest changes in both code base and docs.

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

…iguration Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

jorritsandbrink

LGTM!

I like the typing module.

Default to for both on-prem and cloud

56b3818

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Pipboyguy linked an issue Jun 19, 2024 that may be closed by this pull request

ClickHouse MergeTree Support #1387

Closed

Pipboyguy self-assigned this Jun 19, 2024

Pipboyguy added enhancement New feature or request tech-debt Leftovers from previous sprint that should be fixed over time labels Jun 19, 2024

Pipboyguy added 2 commits June 19, 2024 22:55

Add documentation for new engine family types

e4e1010

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Typo

d998ac9

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Pipboyguy requested review from rudolfix and sh-rp and removed request for rudolfix June 20, 2024 12:35

Merge branch 'refs/heads/devel' into 1387-clickhouse-mergetree-support

825982b

Pipboyguy changed the title ~~Add support for MergeTree engine in ClickHouse destination~~ Support Open Source ClickHouse Deployments Jun 20, 2024

Pipboyguy added 2 commits June 20, 2024 22:11

Minor doc changes

4912173

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Fix local clickhouse deployment timestamp parsing issue with simple c…

e1ab71d

…onfiguration setting~ Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

jorritsandbrink reviewed Jun 24, 2024

View reviewed changes

docs/website/docs/dlt-ecosystem/destinations/clickhouse.md Show resolved Hide resolved

tests/load/clickhouse/test_clickhouse_adapter.py Show resolved Hide resolved

Pipboyguy added 3 commits June 25, 2024 14:23

Merge branch 'devel' into 1387-clickhouse-mergetree-support

db603a5

Extend support for local deployment time types

8f80391

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Adapt test to check whether CH OSS or cloud

544925a

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

rudolfix added the sprint Marks group of tasks with core team focus at this moment label Jun 26, 2024

Pipboyguy added 9 commits June 26, 2024 20:44

Defend against CH OSS unsupported dbapi datetime parsing

b2a3596

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Fix typo

e2a5c4a

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Add ClickHouse to local destination tests

9ef12c1

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Update ClickHouse test workflow and remove engine types

2388cd6

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Use Python 3.10.x for ClickHouse destination tests

b0e3751

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Merge branch 'devel' into 1387-clickhouse-mergetree-support

d207686

# Conflicts: # dlt/destinations/impl/clickhouse/clickhouse.py # dlt/destinations/impl/clickhouse/sql_client.py # tests/load/clickhouse/test_clickhouse_adapter.py

Add ClickHouse MergeTree support and refactor code

f077346

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Update ClickHouse Docker setup and test workflow

6e44a71

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Refactor ClickHouse tests to cover both OSS and Cloud versions

61bae45

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Pipboyguy added 2 commits July 1, 2024 15:56

Update ClickHouse docker-compose file path in test workflow

c408824

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Cast client to ClickHouseSqlClient in get_deployment_type call

0d306a5

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Pipboyguy requested a review from jorritsandbrink July 1, 2024 14:13

jorritsandbrink reviewed Jul 2, 2024

View reviewed changes

rudolfix removed the sprint Marks group of tasks with core team focus at this moment label Jul 3, 2024

Pipboyguy added 3 commits July 4, 2024 15:56

Merge branch 'refs/heads/devel' into 1387-clickhouse-mergetree-support

d6913cc

Revert "Remove mention of sentinel table for ClickHouse datasets"

28866ee

This reverts commit 409487c.

Revert "Use state instead of sentinel tables"

41a7e1a

This reverts commit e1ac1ce.

rudolfix requested changes Jul 4, 2024

View reviewed changes

dlt/destinations/impl/clickhouse/clickhouse.py Show resolved Hide resolved

docs/website/docs/dlt-ecosystem/destinations/clickhouse.md Show resolved Hide resolved

Pipboyguy added 3 commits July 4, 2024 20:37

Add tests for ClickHouse table engine configuration and adapter overr…

bc7b309

…ides Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Merge branch 'refs/heads/devel' into 1387-clickhouse-mergetree-support

98d2329

Add configurable default table engine type for ClickHouse

f852d44

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Pipboyguy requested review from rudolfix and jorritsandbrink July 4, 2024 20:56

Docs

b0cec6e

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

rudolfix previously approved these changes Jul 5, 2024

View reviewed changes

jorritsandbrink reviewed Jul 5, 2024

View reviewed changes

Pipboyguy added 4 commits July 12, 2024 14:11

Merge branch 'refs/heads/devel' into 1387-clickhouse-mergetree-support

d764734

Fix comments

b2bc5b1

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Add ClickHouse typing module for improved type handling

8d26ef1

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Merge branch 'refs/heads/devel' into 1387-clickhouse-mergetree-support

9c67c65

Pipboyguy dismissed rudolfix’s stale review via 9c67c65 July 12, 2024 14:15

Pipboyguy added 3 commits July 12, 2024 16:59

Move ClickHouse configuration options from credentials to client conf…

3028cf4

…iguration Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Move table_engine_type from credentials to client configuration

8f60436

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Docs

477e815

Signed-off-by: Marcel Coetzee <marcel@mooncoon.com>

Pipboyguy requested review from jorritsandbrink and rudolfix July 12, 2024 18:29

jorritsandbrink approved these changes Jul 13, 2024

View reviewed changes

rudolfix merged commit 9156f44 into devel Jul 15, 2024
52 checks passed

rudolfix deleted the 1387-clickhouse-mergetree-support branch July 15, 2024 19:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Open Source ClickHouse Deployments #1496

Support Open Source ClickHouse Deployments #1496

Pipboyguy commented Jun 19, 2024 •

edited

Loading

netlify bot commented Jun 19, 2024 •

edited

Loading

jorritsandbrink left a comment

jorritsandbrink left a comment

rudolfix left a comment

rudolfix left a comment

rudolfix Jul 5, 2024

Pipboyguy Jul 12, 2024

jorritsandbrink left a comment •

edited

Loading

jorritsandbrink Jul 5, 2024

rudolfix Jul 5, 2024

Pipboyguy Jul 12, 2024

jorritsandbrink left a comment

	sentinel_table_type = cast(TTableEngineType, self.credentials.table_engine_type)
	self.execute_sql(f"""
	CREATE TABLE {sentinel_table_name}
	(_dlt_id String NOT NULL PRIMARY KEY)
	ENGINE={TABLE_ENGINE_TYPE_TO_CLICKHOUSE_ATTR.get(sentinel_table_type)}
	COMMENT 'internal dlt sentinel table'""")

Support Open Source ClickHouse Deployments #1496

Support Open Source ClickHouse Deployments #1496

Conversation

Pipboyguy commented Jun 19, 2024 • edited Loading

Description

Related Issues

Additional Context

netlify bot commented Jun 19, 2024 • edited Loading

✅ Deploy Preview for dlt-hub-docs ready!

jorritsandbrink left a comment

Choose a reason for hiding this comment

jorritsandbrink left a comment

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

rudolfix Jul 5, 2024

Choose a reason for hiding this comment

Pipboyguy Jul 12, 2024

Choose a reason for hiding this comment

jorritsandbrink left a comment • edited Loading

Choose a reason for hiding this comment

jorritsandbrink Jul 5, 2024

Choose a reason for hiding this comment

rudolfix Jul 5, 2024

Choose a reason for hiding this comment

Pipboyguy Jul 12, 2024

Choose a reason for hiding this comment

jorritsandbrink left a comment

Choose a reason for hiding this comment

Pipboyguy commented Jun 19, 2024 •

edited

Loading

netlify bot commented Jun 19, 2024 •

edited

Loading

jorritsandbrink left a comment •

edited

Loading