Java CDK 'no-op': v0.0.1, v0.0.2 #28687

aaronsteers · 2023-07-25T19:04:05Z

Note
Part of this epic.

(Replaces #28446.)

What

New Java CDK for Sources DB. For an overview, you can read a preview of the developer-facing docs here:

📄 Airbyte CDK Readme (Preview)

This should be considered a "no-op" version of the CDK. The PR is non-invasive so that it can be merged without issue and without merge conflicts.

In this PR, we will:

Create a CDK test flow.
Create a CDK publish flow (MavenLocal).
Map a dependency from at least one connector to the CDK.
Create a CDK publish flow (public Maven).
Create CI flows to manage the test and publish lifecycle tasks.
Point all Java connectors to depend on the CDK.

Working and timely tests are critical to use having confidence to deploy the next time around.

The airbyte-ci test command was previously skipping execution of unit tests. This PR brings back unit tests.
Posgres had two flaky unit tests which I've disabled. I've logged an issue for follow-up here:
- Flaky unit tests for source-postgres: CdcSourceTest.testExistingData() and verifyCheckpointStatesBySeconds() #29411
Added --fail-fast, --code-tests-only, and --fast-tests-only to the airbyte-ci connectors test command. If no arguments are set, then the default behaviors are unchanged.
- As of now, the 'fast tests' are just all of the unit tests. We can add handling for 'slow' unit tests in the future if needed.
- "Code tests" are everything except pre-release checklist and metadata items - including format checks, unit tests, and integration tests. This mode allows you to see your code is working before investing in metadata, version bumps, changelogs, etc.
- "Fail fast" is a setting we can pass down to gradle, but it's also a behavior we can implement on our side. The current behavior is to tell Gradle to abort tests once it sees its first failure, and also then in our control flow, we return without running more tests.
- Sample usage when debugging, using all three flags:
  - airbyte-ci connectors --name=source-postgres test --fail-fast --code-tests-only --fast-tests-only

Immediate fast-follow PR can be merged as soon as this one does:

Add cdk 0.0.1 dependency to all java connectors #29219
1. Adds the CDK 0.0.1 dependency to all connectors.

And this v0.0.1 PR moves code into the CDK:

Moved shared Java base classes to Java CDK (base-java) #29224
1. Move directories that should be included in the CDK to the CDK directory.
2. Update dependency package names as needed to prevent breakage on connectors.

How

Describe the solution

🚨 User Impact 🚨

(No breaking changes.)

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

github-actions · 2023-08-21T16:49:19Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	69.48%	🍏

github-actions · 2023-08-21T16:49:47Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	69.48%	🍏

aaronsteers · 2023-08-21T16:52:09Z

Pre-release test publish for Postgres (all succeeded):

✅ https://github.com/airbytehq/airbyte/actions/runs/5929151204
- ✅ https://github.com/airbytehq/airbyte/actions/runs/5929151204/job/16076223824

For BigQuery:

✅ https://github.com/airbytehq/airbyte/actions/runs/5929187976

For Source-Faker (not touched, but is used as a random python check):

✅ https://github.com/airbytehq/airbyte/actions/runs/5929173327

github-actions · 2023-08-21T16:59:49Z

destination-bigquery test report (commit `f53263334d`) - ✅

⏲️ Total pipeline duration: 33mn56s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-21T17:08:47Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	69.48%	🍏

github-actions · 2023-08-21T17:12:17Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	69.48%	🍏

github-actions · 2023-08-21T17:32:00Z

source-postgres test report (commit `f53263334d`) - ❌

⏲️ Total pipeline duration: 32mn00s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-08-21T18:05:48Z

destination-bigquery test report (commit `676b3c96a3`) - ✅

⏲️ Total pipeline duration: 31mn26s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-21T18:36:25Z

source-postgres test report (commit `676b3c96a3`) - ❌

⏲️ Total pipeline duration: 30mn26s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

aaronsteers · 2023-08-21T19:00:56Z

/approve-and-merge reason="postgres metadata test failure is unrelated, will be addressed in #27872"

octavia-approvington · 2023-08-21T19:01:30Z

This code is at peace with itself

github-actions · 2023-08-21T19:09:17Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	69.48%	🍏

github-actions · 2023-08-21T19:11:13Z

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	69.48%	🍏

github-actions · 2023-08-21T19:21:33Z

source-postgres test report (commit `f744342be0`) - ✅

⏲️ Total pipeline duration: 34mn46s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-08-21T19:29:39Z

destination-bigquery test report (commit `f744342be0`) - ❌

⏲️ Total pipeline duration: 07mn50s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-21T20:04:32Z

destination-bigquery test report (commit `5510cd1751`) - ❌

⏲️ Total pipeline duration: 32mn30s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-08-21T20:08:43Z

source-postgres test report (commit `5510cd1751`) - ❌

⏲️ Total pipeline duration: 04mn00s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

Co-authored-by: aaronsteers <aaronsteers@users.noreply.github.com> Co-authored-by: Conor <cpdeethree@users.noreply.github.com> Co-authored-by: cpdeethree <conor@airbyte.io> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Joe Bell <joseph.bell@airbyte.io>

aaronsteers added 2 commits July 25, 2023 11:52

checkpoint: working build and test

e5518c5

Merge remote-tracking branch 'origin/master' into 28445-java-cdk-001

db9b1c6

aaronsteers requested a review from a team as a code owner July 25, 2023 19:04

octavia-squidington-iii added the CDK Connector Development Kit label Jul 25, 2023

aaronsteers mentioned this pull request Jul 25, 2023

Draft: Java Source DBs CDK #28446

Closed

aaronsteers marked this pull request as draft July 25, 2023 19:06

aaronsteers removed the request for review from a team July 25, 2023 19:06

aaronsteers and others added 4 commits July 25, 2023 19:09

Automated Commit - Formatting Changes

17455a7

chore: update build.gradle config

0635b44

chore: pin to CDK dependency within source-mysql

1c65795

docs: add a simple cdk readme

37e1a94

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mssql labels Jul 25, 2023

aaronsteers and others added 12 commits July 25, 2023 14:49

fix: gradle runnable from repo root

60b0003

add a stub to test a java cdk workflow (#28703)

db7590e

fix: gradle runnable from repo root

b83ed58

Merge remote-tracking branch 'origin/master' into 28445-java-cdk-001

6b6f201

build airbyte java cdk with dagger

554ca1f

Merge branch '28445-java-cdk-001' into cdk-publish-flow

6d9c91c

fix path

417d14e

update poetry lock file

bbb256b

point at cdk package instead of default airbyte-cdk

ede7ee4

add ci helper script

c25e03e

mv readme.md

fb0b0f1

revert mssql, apply sdk on mysql source

a41ed27

prateekmukhedkar requested review from rodireich, nguyenaiden, subodh1810 and akashkulk July 31, 2023 06:05

chore: add cdk test to mysql source

dbfc89b

aaronsteers added 6 commits August 21, 2023 09:02

ci: remove retry action for build step

7742a35

fix: move strictness level to acceptance test yaml

d2abd30

Merge remote-tracking branch 'origin/master' into 28445-java-cdk-001

f532633

chore: remove 'snapshot' suffix, publish CDK as v0.0.2

b6757c5

chore: pre-release checklist: version bumps + changelog

87cd409

docs: clarify connector publish process, fix changelog date

676b3c9

aaronsteers added 2 commits August 21, 2023 11:44

revert with comment: test_strictness_level: high

6baaac4

Merge branch 'master' into 28445-java-cdk-001

f744342

octavia-approvington approved these changes Aug 21, 2023

View reviewed changes

octavia-approvington merged commit d094b15 into master Aug 21, 2023
18 of 22 checks passed

octavia-approvington deleted the 28445-java-cdk-001 branch August 21, 2023 19:01

aaronsteers changed the title ~~Java CDK 'no-op': v0.0.1~~ Java CDK 'no-op': v0.0.1, v0.0.2 Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Java CDK 'no-op': v0.0.1, v0.0.2 #28687

Java CDK 'no-op': v0.0.1, v0.0.2 #28687

aaronsteers commented Jul 25, 2023 •

edited

Loading

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

aaronsteers commented Aug 21, 2023 •

edited

Loading

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

aaronsteers commented Aug 21, 2023

octavia-approvington commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

Java CDK 'no-op': v0.0.1, v0.0.2 #28687

Java CDK 'no-op': v0.0.1, v0.0.2 #28687

Conversation

aaronsteers commented Jul 25, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

github-actions bot commented Aug 21, 2023

Coverage report for source-postgres

github-actions bot commented Aug 21, 2023

Coverage report for source-postgres

aaronsteers commented Aug 21, 2023 • edited Loading

github-actions bot commented Aug 21, 2023

destination-bigquery test report (commit f53263334d) - ✅

github-actions bot commented Aug 21, 2023

Coverage report for source-postgres

github-actions bot commented Aug 21, 2023

Coverage report for source-postgres

github-actions bot commented Aug 21, 2023

source-postgres test report (commit f53263334d) - ❌

github-actions bot commented Aug 21, 2023

destination-bigquery test report (commit 676b3c96a3) - ✅

github-actions bot commented Aug 21, 2023

source-postgres test report (commit 676b3c96a3) - ❌

aaronsteers commented Aug 21, 2023

octavia-approvington commented Aug 21, 2023

github-actions bot commented Aug 21, 2023

Coverage report for source-postgres

github-actions bot commented Aug 21, 2023

Coverage report for source-postgres

github-actions bot commented Aug 21, 2023

source-postgres test report (commit f744342be0) - ✅

github-actions bot commented Aug 21, 2023

destination-bigquery test report (commit f744342be0) - ❌

github-actions bot commented Aug 21, 2023

destination-bigquery test report (commit 5510cd1751) - ❌

github-actions bot commented Aug 21, 2023

source-postgres test report (commit 5510cd1751) - ❌

aaronsteers commented Jul 25, 2023 •

edited

Loading

aaronsteers commented Aug 21, 2023 •

edited

Loading

destination-bigquery test report (commit `f53263334d`) - ✅

source-postgres test report (commit `f53263334d`) - ❌

destination-bigquery test report (commit `676b3c96a3`) - ✅

source-postgres test report (commit `676b3c96a3`) - ❌

source-postgres test report (commit `f744342be0`) - ✅

destination-bigquery test report (commit `f744342be0`) - ❌

destination-bigquery test report (commit `5510cd1751`) - ❌

source-postgres test report (commit `5510cd1751`) - ❌