Skip to content

add SSL options to mssql - #33071

Merged
Marius Posta (postamar) merged 9 commits into
masterfrom
stephane/12-02-fix_MssqlSource.java
Jan 4, 2024
Merged

Marius Posta (postamar) merged 9 commits into
masterfrom
stephane/12-02-fix_MssqlSource.java

Conversation

@stephane-airbyte

@stephane-airbyte Stephane Geneix (stephane-airbyte) commented Dec 4, 2023 •

Copy link
Copy Markdown
Contributor

This is adding extra support for SSL options in our MS SQL source.

The MSSQL options are a bit different than postgres and MySql,so I'm not using their common logic. The flip side is that this change is not breaking compatibility!
We're basically adding the ability to send a certificate. (of the server or of the CA that signed the server certificate), and the ability to override the hostname used to connect in order to match the hostname of the server certificate.

While doing those changes, I also noticed that the password was actually mandatory in MsSqlSource, even though it wasn't marked as such in the spec.jsob.

The tests try to go the extra mile by connecting to the SQL server by IP rather than hostname, which triggers a failure to validate the certificate's host. And There's an extra test that connects by IP and passes the certificate's hostname in the spec, so the loop is closed.

On minor thing to note is that the MS SQL JDBC driver expects a keystore (technically a keystore file path), but we can't pass those through the current UI (because a key store is not a text file). So instead, we're passing a certificate, and the MsSqlSource converts that into a keystore file that is materialized in the container and its path is passed to the JDBC driver

tests are passing, connector version is bumpted

Resolves #30004

@vercel

vercel Bot commented Dec 4, 2023 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Jan 4, 2024 7:20pm

Stephane Geneix (stephane-airbyte) commented Dec 4, 2023 •

Copy link
Copy Markdown
Contributor Author

@github-actions

github-actions Bot commented Dec 4, 2023 •

Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only added for clarity. By default, encrypt=false implies trustServerCertificate=true.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trustServerCertificate is trust all essentially.

This will trust any server certificate. I assume for the exchange of credentials which is still encrypted event when encrypt=false (you mentioned this offline)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added for clarity. By default, encrypt=true implies trustServerCertificate=false

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was really not doing anything. If anything, it'd guarantee that the connection would fail (because with this, the sever certificate would have to be signed by one of the root authorities in the default trust store)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the old behavior is equivalent to not pushing anything as a trustStorelocation JDBC parameer

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MSSQL driver takes a trust store file location as a parameter. Unfortunately, such a file is binary, which means it can't be uploaded from the current UI. It really doesn't matter, since a certificate is just as good, as long as we have either the server certificate that of the CA that signed the server certificate

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the password is required in the parameter parsing logic in MssqlSource.java. Making it mandatory in the UI as well

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a breaking change? What is the expected behavior here?
I'm thinking of existing configs for example saved without a password

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unless I'm missing something, the previous required fields didn't even exist in the spec.json. Removing them...

@stephane-airbyte
Stephane Geneix (stephane-airbyte) force-pushed the stephane/12-02-fix_MssqlSource.java branch 3 times, most recently from bd13cbd to 2bee629 Compare December 6, 2023 22:00

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that the password is a mandatory field, it needs to be added here too

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this allows us to reduce the test running times from 1m to 2seconds for the tests that are expecting a failure to connect to the DB!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we always want to set this for all test cases, though? I don't have a problem with it. I took a more conservative approach with maybeSetShorterConnectionTimeout but that conservatism may not have been warranted.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no problems with it so far...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trustServerCertificate is trust all essentially.

This will trust any server certificate. I assume for the exchange of credentials which is still encrypted event when encrypt=false (you mentioned this offline)

@postamar Marius Posta (postamar) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand all of this but for what I do, I don't have any significant objections.

connectorType: source
definitionId: b5ea17b1-f170-46dc-bc31-cc744ca984c1
dockerImageTag: 3.1.0
dockerImageTag: 3.1.1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI is going to complain about a missing release note.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably go to the latest 3.5.0

}

@ParameterizedTest
@EnumSource(CertificateKey.class)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yesss! Though perhaps having two enums, one for valid and another for invalid certificates, might be preferable for expressing these tests.

.build();
try {
AirbyteCatalog catalog = new MssqlSource().discover(config);
assertTrue(certificateKey.isValid);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it's just me but I would find some branching logic if (certificateKey.isValid) { ... easier to read here. Do I understand correctly that we expect discover to throw iff !isValid?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no we expect discover to work for valid certificates, and fail for invalid ones. I will split this test in 2 to be clearer

public void withSslCertificates(MSSQLServerContainer<?> container) {
// yes, this is uglier than sin. The reason why I'm doing this is because there's no command to
// reload a SqlServer config. So I need to create all the necessary files before I start the
// SQL server. Hence this horror

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whatever it takes!

EOF
} && /opt/mssql/bin/sqlservr
""",
"{hostName}", container.getHost());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine by me but perhaps using an env var to inject the host name would be simpler? Then again perhaps not, I don't know. Just a thought.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I guess that would work too. Even though I have more refactors in mind.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we always want to set this for all test cases, though? I don't have a problem with it. I took a more conservative approach with maybeSetShorterConnectionTimeout but that conservatism may not have been warranted.

@stephane-airbyte
Stephane Geneix (stephane-airbyte) force-pushed the stephane/12-02-fix_MssqlSource.java branch 2 times, most recently from e9d0acb to 19bf49f Compare December 19, 2023 17:27
@stephane-airbyte
Stephane Geneix (stephane-airbyte) changed the base branch from master to stephane/12-19-move_mssql_to_latest_CDK_remove_LEGACY_state December 19, 2023 18:19

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure mssql points to the latest - 0.8.2(?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done (0.10.4)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Stephane.
Just make sure all versions are aligned before merging,

@katmarkham
Kat Wilson (katmarkham) removed the request for review from a team January 3, 2024 18:17
Base automatically changed from stephane/12-07-don_t_use_MsSQLConfigBuilder.withSsl_with_strings_but_instead_properly_named_and_typed_methods to master January 4, 2024 17:52
@postamar

Marius Posta (postamar) commented Jan 4, 2024 •

Copy link
Copy Markdown
Contributor

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7413665676
✅ Successfully published Java CDK version=0.10.4!

@postamar
Marius Posta (postamar) merged commit 15490f6 into master Jan 4, 2024
@postamar
Marius Posta (postamar) deleted the stephane/12-02-fix_MssqlSource.java branch January 4, 2024 19:50
@postamar

Copy link
Copy Markdown
Contributor

Updated and merged as discussed during team standup earlier this week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/source/mssql

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Source MSSQL (SQL Server) SSL implementation not complete

4 participants