Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination Clickhouse: enable normalization for Secure connections #10754

Merged
merged 8 commits into from
Mar 5, 2022
Merged

Destination Clickhouse: enable normalization for Secure connections #10754

merged 8 commits into from
Mar 5, 2022

Conversation

kzzzr
Copy link
Contributor

@kzzzr kzzzr commented Mar 1, 2022

What

Enable normalization for Destination Clickhouse with SSL connection by configuring Secure parameter in profiles.yml
#9837

How

By supplying key "secure": config["ssl"] to method transform_clickhouse

Recommended reading order

  1. transform.py
  2. test_transform_config.py

🚨 User Impact 🚨

No breaking changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the new connector version is published, connector version bumped in the seed directory as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit
  • TestTransformConfig.test_transform_clickhouse
./gradlew :airbyte-integrations:bases:base-normalization:unitTest

@CLAassistant
Copy link

CLAassistant commented Mar 1, 2022

CLA assistant check
All committers have signed the CLA.

@marcosmarxm
Copy link
Member

thanks for the contribution @kzzzr is it possible to you sign the CLA? The team will review your contribution during this week.

Copy link
Member

@marcosmarxm marcosmarxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM only I want to test if the dbt container read correctly the value.

@@ -279,6 +279,7 @@ def transform_clickhouse(config: Dict[str, Any]):
"port": config["port"],
"schema": config["database"],
"user": config["username"],
"secure": config["ssl"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My only concern here is if the config file created by the UI read as boolean and not as a string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, it is boolean:

"ssl": {
"title": "SSL Connection",
"description": "Encrypt data using SSL.",
"type": "boolean",
"default": false,
"order": 6
}

I've managed to run normalization successfully, only I had to change NORMALIZATION_VERSION = "dev", build SUB_BUILD=PLATFORM ./gradlew build and test it with UI locally.

public static final String NORMALIZATION_VERSION = "0.1.68";

However I think it was hard way to test this.

@marcosmarxm marcosmarxm self-assigned this Mar 2, 2022
@kzzzr kzzzr requested a review from a team as a code owner March 2, 2022 22:21
@github-actions github-actions bot added area/api Related to the api area/connectors Connector related issues area/documentation Improvements or additions to documentation area/frontend area/platform issues related to the platform area/scheduler area/server area/worker Related to worker kubernetes labels Mar 2, 2022
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets March 2, 2022 22:23 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets March 2, 2022 23:48 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets March 3, 2022 00:26 Inactive
@marcosmarxm
Copy link
Member

having some problems publishing the new version of normalization.

@github-actions github-actions bot removed area/connectors Connector related issues area/server area/frontend area/api Related to the api kubernetes area/documentation Improvements or additions to documentation labels Mar 5, 2022
@marcosmarxm marcosmarxm merged commit e34c357 into airbytehq:master Mar 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/platform issues related to the platform area/worker Related to worker community normalization
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants