Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image #31486

akashkulk · 2023-10-17T00:10:18Z

Steps taken :

Copy over logic from .*StrictEncrypt.java classes
Update the images in the ported over java test files to use source-postgres instead of the strict encrypt variant
Copy over the expected_spec.json file
Migrate integration tests
Update CAT tests

Future steps :

Remove the source-postgres-strict-encrypt directory
Remove Dockerfile?

-Adds PostgresSourceRunner as a bridge to determine to run strict-encrypt/non-strict-encrypt

vercel · 2023-10-17T00:10:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 18, 2023 6:50pm

github-actions · 2023-10-17T00:10:40Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

…pt-2

airbyte-oss-build-runner · 2023-10-17T00:35:36Z

source-postgres test report (commit `82cd548048`) - ✅

⏲️ Total pipeline duration: 22mn10s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Java Connector Integration Tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

…pt-2

airbyte-oss-build-runner · 2023-10-17T01:27:20Z

source-postgres test report (commit `aec3f12f8f`) - ✅

⏲️ Total pipeline duration: 21mn36s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Java Connector Integration Tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-integrations/connectors/source-postgres/metadata.yaml

bnchrch

So excited to see this go up for review.

From a connector ops perspective it seems good!

Thanks @akashkulk

postamar

Just a bunch of nits on my end, no objections of principle.

postamar · 2023-10-18T00:05:50Z

airbyte-integrations/connectors/source-postgres/build.gradle

@@ -9,7 +9,7 @@ plugins {
 airbyteJavaConnector {
    cdkVersionRequired = '0.1.9'
    features = ['db-sources']
-    useLocalCdk = false
+    useLocalCdk = true


Don't forget to publish the CDK, import the new version and set this to false before merging.

Ah yes! Just wanted to do that to make sure tests were passing!

Am publishing it, making the switch after it's been published and works locally with the newly published version

airbyte-integrations/connectors/source-postgres/metadata.yaml

...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java

docs/integrations/sources/postgres.md

…pt-2

akashkulk · 2023-10-18T17:18:10Z

/publish-java-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/6564341526
❌ Publish Java CDK version=0.1.10 failed!

airbyte-oss-build-runner · 2023-10-18T17:38:56Z

source-postgres test report (commit `85a40ed5b0`) - ❌

⏲️ Total pipeline duration: 19mn41s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	❌
Build source-postgres docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Java Connector Integration Tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

akashkulk · 2023-10-18T17:40:43Z

/publish-java-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/6564577286
✅ Successfully published Java CDK version=0.1.11!

akashkulk · 2023-10-18T17:59:48Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6564746364
✅ Successfully published Java CDK version=0.1.11!

…pt-2

airbyte-oss-build-runner · 2023-10-18T18:39:10Z

source-postgres test report (commit `db2002cd5a`) - ❌

⏲️ Total pipeline duration: 18mn51s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	❌
Build source-postgres docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Java Connector Integration Tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

…pt-2

airbyte-oss-build-runner · 2023-10-18T19:10:52Z

source-postgres test report (commit `e156d376da`) - ✅

⏲️ Total pipeline duration: 18mn36s

Step	Result
Build connector tar	✅
Java Connector Unit Tests	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Java Connector Integration Tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

… make it the default cloud image (airbytehq#31486) Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>

akashkulk and others added 12 commits October 3, 2023 14:02

Create PostgresSourceRunner

114decf

-Adds PostgresSourceRunner as a bridge to determine to run strict-encrypt/non-strict-encrypt

Add strict encrypt spec

20db407

Move over tests

c31c835

Bump version + remove docker override

35ce812

Update the documentation

69c3256

Automated Commit - Formatting Changes

9d6ded9

Revert docs

025d3fd

Merge branch 'master' into variant-remove/source-postgres

9bc81c7

Merge branch 'master' into variant-remove/source-postgres

6fe86b3

Bump dockerfiles + metadata

0baaf6e

Merge branch 'master' into variant-remove/source-postgres

3fb9ed7

Remove strict-encrypt java class

d0ea211

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/source/postgres labels Oct 17, 2023

Merge branch 'master' into akash-variant-remove/source-postgres-attem…

82cd548

…pt-2

akashkulk marked this pull request as ready for review October 17, 2023 00:10

akashkulk requested a review from a team as a code owner October 17, 2023 00:10

octavia-squidington-iv requested review from a team October 17, 2023 00:11

vercel bot deployed to Preview October 17, 2023 00:12 View deployment

Merge branch 'master' into akash-variant-remove/source-postgres-attem…

aec3f12

…pt-2

vercel bot deployed to Preview October 17, 2023 01:05 View deployment

octavia-squidington-iv requested a review from a team October 17, 2023 01:05

akashkulk requested a review from nguyenaiden October 17, 2023 16:11

akashkulk requested review from bnchrch and erohmensing October 17, 2023 16:11

akashkulk changed the title ~~Akash variant remove/source postgres attempt 2~~ Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image Oct 17, 2023

bnchrch reviewed Oct 17, 2023

View reviewed changes

airbyte-integrations/connectors/source-postgres/metadata.yaml Show resolved Hide resolved

bnchrch approved these changes Oct 17, 2023

View reviewed changes

postamar reviewed Oct 18, 2023

View reviewed changes

akashkulk and others added 2 commits October 18, 2023 10:17

Address code review comments

4706bfc

Merge branch 'master' into akash-variant-remove/source-postgres-attem…

85a40ed

…pt-2

vercel bot deployed to Preview October 18, 2023 17:20 View deployment

bazarnov approved these changes Oct 18, 2023

View reviewed changes

akashkulk added 2 commits October 18, 2023 10:39

bump version

f30289e

Update Dockerfile

63ae48f

akashkulk and others added 3 commits October 18, 2023 11:01

Merge branch 'master' into akash-variant-remove/source-postgres-attem…

249e082

…pt-2

Bump cdk versions + tests for source-postgres

b6ab9e3

Merge branch 'master' into akash-variant-remove/source-postgres-attem…

db2002c

…pt-2

vercel bot deployed to Preview October 18, 2023 18:20 View deployment

Merge branch 'master' into akash-variant-remove/source-postgres-attem…

e156d37

…pt-2

akashkulk merged commit 52520d0 into master Oct 18, 2023
23 of 26 checks passed

akashkulk deleted the akash-variant-remove/source-postgres-attempt-2 branch October 18, 2023 19:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image #31486

Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image #31486

akashkulk commented Oct 17, 2023

vercel bot commented Oct 17, 2023 •

edited

Loading

github-actions bot commented Oct 17, 2023

airbyte-oss-build-runner commented Oct 17, 2023

airbyte-oss-build-runner commented Oct 17, 2023

bnchrch left a comment

postamar left a comment

postamar Oct 18, 2023

akashkulk Oct 18, 2023

akashkulk Oct 18, 2023

akashkulk commented Oct 18, 2023 •

edited by github-actions bot

Loading

airbyte-oss-build-runner commented Oct 18, 2023

akashkulk commented Oct 18, 2023 •

edited by github-actions bot

Loading

akashkulk commented Oct 18, 2023 •

edited by github-actions bot

Loading

airbyte-oss-build-runner commented Oct 18, 2023

airbyte-oss-build-runner commented Oct 18, 2023

Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image #31486

Migrate source-postgres-strict-encrypt logic into source-postgres and make it the default cloud image #31486

Conversation

akashkulk commented Oct 17, 2023

vercel bot commented Oct 17, 2023 • edited Loading

github-actions bot commented Oct 17, 2023

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 17, 2023

source-postgres test report (commit 82cd548048) - ✅

airbyte-oss-build-runner commented Oct 17, 2023

source-postgres test report (commit aec3f12f8f) - ✅

bnchrch left a comment

Choose a reason for hiding this comment

postamar left a comment

Choose a reason for hiding this comment

postamar Oct 18, 2023

Choose a reason for hiding this comment

akashkulk Oct 18, 2023

Choose a reason for hiding this comment

akashkulk Oct 18, 2023

Choose a reason for hiding this comment

akashkulk commented Oct 18, 2023 • edited by github-actions bot Loading

airbyte-oss-build-runner commented Oct 18, 2023

source-postgres test report (commit 85a40ed5b0) - ❌

akashkulk commented Oct 18, 2023 • edited by github-actions bot Loading

akashkulk commented Oct 18, 2023 • edited by github-actions bot Loading

airbyte-oss-build-runner commented Oct 18, 2023

source-postgres test report (commit db2002cd5a) - ❌

airbyte-oss-build-runner commented Oct 18, 2023

source-postgres test report (commit e156d376da) - ✅

vercel bot commented Oct 17, 2023 •

edited

Loading

source-postgres test report (commit `82cd548048`) - ✅

source-postgres test report (commit `aec3f12f8f`) - ✅

akashkulk commented Oct 18, 2023 •

edited by github-actions bot

Loading

source-postgres test report (commit `85a40ed5b0`) - ❌

akashkulk commented Oct 18, 2023 •

edited by github-actions bot

Loading

akashkulk commented Oct 18, 2023 •

edited by github-actions bot

Loading

source-postgres test report (commit `db2002cd5a`) - ❌

source-postgres test report (commit `e156d376da`) - ✅