Publish SSL-only version of Postgres Destination #6496

cgardens · 2021-09-28T01:34:34Z

Relates to #6332

What

We want to publish versions of database connectors that only allow SSL connections.

How

Same approach as for postgres source: Exposing SSL-only version of Postgres Source #6362. Create a separate connector that modifies the source.
In addition, I realized that normalization doesn't respect the ssl flag. Now it does.

Pre-merge Checklist

- get normalization published ~~(blocked on github actions outage)~~
- get new destination published ~~(blocked on github actions outage)~~

cgardens · 2021-09-28T01:34:42Z

/publish connector=bases/base-normalization

cgardens · 2021-09-28T02:21:33Z

/publish connector=bases/base-normalization

cgardens · 2021-09-28T02:41:34Z

airbyte-integrations/bases/base-normalization/normalization/transform_config/transform.py

@@ -174,6 +174,10 @@ def transform_postgres(config: Dict[str, Any]):
            "threads": 32,
        }

+        # if unset, we assume true.


hopefully this is all that is needed in normalization. still running tests.

cgardens · 2021-09-28T03:07:58Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1280910626
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1280910626

ChristopheDuong · 2021-09-28T09:46:21Z

airbyte-integrations/bases/base-normalization/normalization/transform_config/transform.py

+        # if unset, we assume true.
+        if config.get("ssl", True):
+            config["sslmode"] = "require"


From DBT docs:
https://docs.getdbt.com/reference/warehouse-profiles/postgres-profile#sslmode

When unset, dbt will connect to databases using the Postgres default, prefer, as the sslmode.

Shouldn't we explicitly set it to "require" if it is unset in the destination config then?

Suggested change

# if unset, we assume true.

if config.get("ssl", True):

config["sslmode"] = "require"

if "ssl" not in config or config.get("ssl", True):

config["sslmode"] = "require"

I just ran /test commands and they work now!

We're just waiting for an answer from this question before publishing then

discussed offline. decided the original syntax is okay. if "config" is not set then get will default to true.

ChristopheDuong · 2021-09-28T09:53:29Z

...irbyte/integrations/destination/postgres/PostgresDestinationStrictEncryptAcceptanceTest.java

+    db = new PostgreSQLContainer<>(DockerImageName.parse("marcosmarxm/postgres-ssl:dev").asCompatibleSubstituteFor("postgres"))
+        .withCommand("postgres -c ssl=on -c ssl_cert_file=/var/lib/postgresql/server.crt -c ssl_key_file=/var/lib/postgresql/server.key");
+    db.start();


A Postgres database container is started for normalization here:

airbyte/airbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.py

Line 38 in 3369237

print("Starting localhost postgres container for tests")

I am guessing, it should be started with SSL using the same image there too?

I pushed this change into normalization integration test on your branch

ChristopheDuong · 2021-09-28T10:00:05Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1282058287
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1282058287
🐛 https://gradle.com/s/lvc2eqylvce7e

…n_encrypt

ChristopheDuong · 2021-09-28T10:23:43Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1282145097
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1282145097
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     35    75%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    390    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  778    432    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     35    75%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    390    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     117      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349     32    91%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         141     46    67%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1037    181    83%

cgardens · 2021-09-28T16:24:12Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1283447591
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1283447591

cgardens · 2021-09-28T17:18:14Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1283614430
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1283614430

ChristopheDuong · 2021-09-30T10:56:58Z

I am merging this for now so it's not blocking others that need to bump version of normalization since the docker image has already been published

I think we still need to publish the postgres strict connector too? @cgardens

cgardens · 2021-09-30T16:46:38Z

thanks. yeah.

try to publish new normalization version

1d53739

github-actions bot added area/connectors Connector related issues area/worker Related to worker normalization labels Sep 28, 2021

cgardens temporarily deployed to more-secrets September 28, 2021 01:35 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 01:37 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 02:23 Inactive

cgardens commented Sep 28, 2021

View reviewed changes

cgardens requested review from sherifnada and ChristopheDuong September 28, 2021 02:41

cgardens changed the title ~~try to publish new normalization version~~ Publish SSL-only version of Postgres Destination Sep 28, 2021

default to using ssl in postgres destinatoin

97fce1b

cgardens temporarily deployed to more-secrets September 28, 2021 02:46 Inactive

cgardens marked this pull request as ready for review September 28, 2021 02:48

tidy up

c17a322

cgardens temporarily deployed to more-secrets September 28, 2021 03:06 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 03:10 Inactive

sherifnada approved these changes Sep 28, 2021

View reviewed changes

ChristopheDuong reviewed Sep 28, 2021

View reviewed changes

ChristopheDuong added 2 commits September 28, 2021 12:00

Run normalization tests using postgres DB with SSL support

890d5a9

Merge remote-tracking branch 'origin/master' into postgres_destinatio…

1fb5a2a

…n_encrypt

jrhizor temporarily deployed to more-secrets September 28, 2021 10:02 Inactive

ChristopheDuong temporarily deployed to more-secrets September 28, 2021 10:25 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 10:26 Inactive

ChristopheDuong approved these changes Sep 28, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets September 28, 2021 16:26 Inactive

bump version

728f5c6

cgardens temporarily deployed to more-secrets September 28, 2021 17:19 Inactive

jrhizor temporarily deployed to more-secrets September 28, 2021 17:20 Inactive

Merge branch 'master' into cgardens/postgres_destination_encrypt

ba0d4ea

cgardens temporarily deployed to more-secrets September 28, 2021 17:22 Inactive

Phlair mentioned this pull request Sep 30, 2021

Normalization: remove ssh errors if ssh is not being used #6504

Merged

ChristopheDuong merged commit 5e75016 into master Sep 30, 2021

ChristopheDuong deleted the cgardens/postgres_destination_encrypt branch September 30, 2021 10:55

jrhizor mentioned this pull request Sep 30, 2021

Bump Airbyte version from 0.30.2-alpha to 0.30.3-alpha #6574

Merged

This was referenced Oct 5, 2021

Create a secure-only Oracle destination #6713

Closed

Create a secure-only MongoDb destination #6714

Closed

etsybaev mentioned this pull request Oct 6, 2021

🎉Destination MySQl - added ssl strict encrypt connector #6763

Merged

33 tasks

alexandr-shegeda mentioned this pull request Oct 6, 2021

Create a secure-only Postgres Destination #6418

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish SSL-only version of Postgres Destination #6496

Publish SSL-only version of Postgres Destination #6496

cgardens commented Sep 28, 2021 •

edited

cgardens commented Sep 28, 2021

cgardens commented Sep 28, 2021

cgardens Sep 28, 2021

cgardens commented Sep 28, 2021 •

edited by github-actions bot

ChristopheDuong Sep 28, 2021

ChristopheDuong Sep 28, 2021

cgardens Sep 28, 2021

ChristopheDuong Sep 28, 2021

ChristopheDuong Sep 28, 2021

cgardens Sep 28, 2021

ChristopheDuong commented Sep 28, 2021 •

edited by github-actions bot

ChristopheDuong commented Sep 28, 2021 •

edited by github-actions bot

cgardens commented Sep 28, 2021 •

edited by github-actions bot

cgardens commented Sep 28, 2021 •

edited by github-actions bot

ChristopheDuong commented Sep 30, 2021 •

edited

cgardens commented Sep 30, 2021 •

edited

Publish SSL-only version of Postgres Destination #6496

Publish SSL-only version of Postgres Destination #6496

Conversation

cgardens commented Sep 28, 2021 • edited

What

How

Recommended reading order

Pre-merge Checklist

cgardens commented Sep 28, 2021

cgardens commented Sep 28, 2021

cgardens Sep 28, 2021

Choose a reason for hiding this comment

cgardens commented Sep 28, 2021 • edited by github-actions bot

ChristopheDuong Sep 28, 2021

Choose a reason for hiding this comment

ChristopheDuong Sep 28, 2021

Choose a reason for hiding this comment

cgardens Sep 28, 2021

Choose a reason for hiding this comment

ChristopheDuong Sep 28, 2021

Choose a reason for hiding this comment

ChristopheDuong Sep 28, 2021

Choose a reason for hiding this comment

cgardens Sep 28, 2021

Choose a reason for hiding this comment

ChristopheDuong commented Sep 28, 2021 • edited by github-actions bot

ChristopheDuong commented Sep 28, 2021 • edited by github-actions bot

cgardens commented Sep 28, 2021 • edited by github-actions bot

cgardens commented Sep 28, 2021 • edited by github-actions bot

ChristopheDuong commented Sep 30, 2021 • edited

cgardens commented Sep 30, 2021 • edited

cgardens commented Sep 28, 2021 •

edited

cgardens commented Sep 28, 2021 •

edited by github-actions bot

ChristopheDuong commented Sep 28, 2021 •

edited by github-actions bot

ChristopheDuong commented Sep 28, 2021 •

edited by github-actions bot

cgardens commented Sep 28, 2021 •

edited by github-actions bot

cgardens commented Sep 28, 2021 •

edited by github-actions bot

ChristopheDuong commented Sep 30, 2021 •

edited

cgardens commented Sep 30, 2021 •

edited