[SAT] [JAVA] Add DISCOVER, CHECK, and READ SAT tests for Java sources with local instances

[evantahler]

From Add SAT to all Source Connectors - Tech Spec

This will involve determining how to boot up test source docker containers. To complete the initial story, we should get CHECK and READ testing working for one Java source (source-postgres), and then create and assign additional stories to the Java Sources team to complete.

[evantahler]

@DoNotPanicUA You mentioned that you had a prototype branch for this work perhaps?

[DoNotPanicUA]

@DoNotPanicUA You mentioned that you had a prototype branch for this work perhaps?

@evantahler
There are two types of sources: local, global
Local sources like Postgres require local instances for test running. That is why we have steps 2 and 3 in the tech spec execution plan.
I implemented a few POC for step 3, and my conclusion is that it might not the best idea to settle local env as part of the source. We will increase the complexity of the container handling and get a lot of work maintaining integration with the Python code, with is also required. Also, a problem is how to shut it down properly.
So, I propose to make it simple - keep small source instances in the AWS (actually, we already have) for all local sources and use them in the same way as we test global sources.

Local Source:

• clickhouse
• cockroachdb
• db2
• mssql
• mysql
• oracle
• postgres
• tidb
• elasticsearch
• kafka
• sftp

Global sources:

• bigquery
• snowflake
• mongodb-v2
• redshift

Here is what I propose to do next:

1. Add DESCOVER, CHECK and READ SAT tests to the global sources as they already have test instances.
2. Create test instances for the local sources in the AWS.
3. Add the tests to the local source using the AWS creds

[evantahler]

In the spirit of "Done is better than Perfect", I agree that if we can skip the need for setting up and tearing down test databases, than that's probably better! Since these are read-only tests, it should be OK to run them regularly and in parallel. We also already have the required tooling for setting up secrets with SAT, so 👍!

keep small source instances in the AWS (actually, we already have)
Does every local source already have a hosted test database/server? Are any missing?

[pedroslopez]

Turned this issue into an EPIC to track doing this work for the local sources. The global sources are tracked in the epic #19708

The postgres-specific implenentationo that this was pointed for has been moved into a new issue #19899

[alafanechere]

@evantahler I created the issues for other java sources. I tagged the DB/DW connectors team.

Edit:
https://github.com/airbytehq/airbyte/projects/24#card-87305727
and the SAT-on-database-source label

[alafanechere]

source-postgres is passing SAT in high strictness level mode on this PR #20549