Include tcp port in Clickhouse destination configuration for normalization #9340

ChristopheDuong · 2022-01-06T15:37:38Z

What

Follow-up to #9072

destination-clickhouse uses HTTP port
normalization-clickhouse uses TCP port

see slack thread: https://airbytehq.slack.com/archives/C01MFR03D5W/p1641468681019500?thread_ts=1641461068.011100&cid=C01MFR03D5W

How

Both ports should be declared in destination configuration

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

…ation

ChristopheDuong · 2022-01-06T15:38:04Z

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1663678967
❌ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1663678967
🐛

ChristopheDuong · 2022-01-06T15:38:21Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1663680016
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1663680016
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     124      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 502    321    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1209    511    58%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     124      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 502    321    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1209    511    58%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     124      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 502     37    93%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     46    69%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1209    186    85%

ChristopheDuong · 2022-01-06T16:07:54Z

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1663797073
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1663797073
No Python unittests run

ChristopheDuong · 2022-01-06T16:19:30Z

/publish connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1663842493
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1663842493

marcosmarxm · 2022-01-06T16:53:52Z

@ChristopheDuong Clickhouse normalization is not working from the connector tests?

ChristopheDuong · 2022-01-06T16:58:50Z

@ChristopheDuong Clickhouse normalization is not working from the connector tests?

I also tried to enable the custom dbt for clickhouse but it didn't work out so I disabled it and the tests are passing now.

I can't seem to be able to run the acceptance tests locally with clickhouse to have more details on the failure though

ChristopheDuong · 2022-01-06T17:02:53Z

Will publish normalization in another PR

Include tcp port in Clickhouse destination configuration for normaliz…

26373f2

…ation

ChristopheDuong requested a review from marcosmarxm January 6, 2022 15:37

github-actions bot added area/connectors Connector related issues normalization labels Jan 6, 2022

ChristopheDuong temporarily deployed to more-secrets January 6, 2022 15:38 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 15:40 Inactive

marcosmarxm approved these changes Jan 6, 2022

View reviewed changes

revert custom dbt on clickhouse

d4ddd0d

ChristopheDuong temporarily deployed to more-secrets January 6, 2022 16:09 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 16:10 Inactive

jrhizor temporarily deployed to more-secrets January 6, 2022 16:21 Inactive

Bumpversion of destination connector

02fafba

github-actions bot added area/platform issues related to the platform area/worker Related to worker labels Jan 6, 2022

ChristopheDuong temporarily deployed to more-secrets January 6, 2022 16:35 Inactive

Format code

be03576

ChristopheDuong merged commit 7727b86 into master Jan 6, 2022

ChristopheDuong deleted the chris/clickhouse-normalization branch January 6, 2022 17:03

ChristopheDuong temporarily deployed to more-secrets January 6, 2022 17:03 Inactive

jrhizor mentioned this pull request Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

karinakuz added the connectors/destinations-warehouse label Jan 14, 2022

karinakuz added the connectors/destination/clickhouse label Jan 14, 2022

marcosmarxm mentioned this pull request Jan 25, 2022

Publish new version of normalization to release clickhouse TCP ports #9794

Closed

mshustov mentioned this pull request Sep 21, 2022

Destination ClickHouse: remove tcp port spec parameter #16970

Closed

37 tasks

grishick mentioned this pull request Oct 1, 2022

Greg/clickhouse polishing #17483

Merged

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include tcp port in Clickhouse destination configuration for normalization #9340

Include tcp port in Clickhouse destination configuration for normalization #9340

ChristopheDuong commented Jan 6, 2022 •

edited

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Jan 6, 2022

ChristopheDuong commented Jan 6, 2022

ChristopheDuong commented Jan 6, 2022

Include tcp port in Clickhouse destination configuration for normalization #9340

Include tcp port in Clickhouse destination configuration for normalization #9340

Conversation

ChristopheDuong commented Jan 6, 2022 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

ChristopheDuong commented Jan 6, 2022 • edited by github-actions bot Loading

ChristopheDuong commented Jan 6, 2022 • edited by github-actions bot Loading

ChristopheDuong commented Jan 6, 2022 • edited by github-actions bot Loading

ChristopheDuong commented Jan 6, 2022 • edited by github-actions bot Loading

marcosmarxm commented Jan 6, 2022

ChristopheDuong commented Jan 6, 2022

ChristopheDuong commented Jan 6, 2022

ChristopheDuong commented Jan 6, 2022 •

edited

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading

ChristopheDuong commented Jan 6, 2022 •

edited by github-actions bot

Loading