Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SSL option for Postgres source/destination #2757

Merged
merged 19 commits into from
Apr 16, 2021

Conversation

marcosmarxm
Copy link
Member

@marcosmarxm marcosmarxm commented Apr 5, 2021

What

Enable SSL connection. Resolves #1806

  • Source Postgres
  • Source Redshift
  • Source Mysql
  • Source Mssql
  • Destination Postgres

How

Add spec SSL and edit JDBC URL to request SSL

Pre-merge Checklist

  • Run integration tests
  • Publish Docker images

Recommended reading order

  1. test.java
  2. component.ts
  3. the rest

@marcosmarxm
Copy link
Member Author

marcosmarxm commented Apr 12, 2021

For running, Postgres enabled with SSL I need to send the certificates and keys to Docker container.

  1. I tried to hard-coded the files inside the resources folder but after the build, the ownership and permissions are changed not allowing the container to access the file.
  2. Also hard-coded the files and execute de commands with testcontainer, but I cannot execute anything before the container start and Postgresql need to enable SSL at starting.
  3. Create a sh script that generate the keys and send to /tmp/airbyte_crt/, users needed to run using sudo... nothing easy and automatic
  4. Solution: Create a custom container and use it (at this moment I used my own account in docker hub to host postgres SSL image).

the proposal is to create an airbyte-docker-ssl repository where the Dockerfiles and script for creating these customized containers will be placed. @jrhizor WDYT?

@marcosmarxm
Copy link
Member Author

Screenshot from 2021-04-12 21-43-15

Copy link
Contributor

@jrhizor jrhizor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good % the few small remaining comments.

@marcosmarxm
Copy link
Member Author

@jrhizor I added the Dockerfile and script to generate SSL credentials inside tools/integrations-test-ssl. I'll create an issue to work on an additional CI workflow to build and release these custom images.
Also, add logic to append additional parameters can you check this?

Is OK to merge only Postgres for now? I'll divide and conquer the others with individual issues/PR to be easy to discuss implementations.

@marcosmarxm marcosmarxm changed the title Add SSL option for JDBC source/destinations Add SSL option for Postgres source/destination Apr 14, 2021
@jrhizor jrhizor merged commit ca8f304 into airbytehq:master Apr 16, 2021
@sherifnada
Copy link
Contributor

amazing, thanks @marcosmarxm !! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support SSL for database connections (Sources & Destinations)
4 participants