java CDK: hoist top-level gradle projects into CDK

[postamar]

Fulfilling this TODO item that's been there since the inception of the Java CDK should help speed up building java connectors.

Not all projects have been hoisted, only those which connectors depend on, directly or transitively.

Fixes #31400

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 30, 2023 6:31pm

[postamar]

According to airbyte-ci connectors --support-level=certified --language=java list we have:

┃                                                       6 selected connectors                                                                                   
┃ ┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓                             
┃ ┃ Modified ┃ Connector             ┃ Language ┃ Release stage ┃ Version ┃ Folder                                                ┃                             
┃ ┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩                             
┃ │ X        │ destination-bigquery  │ java     │ certified     │ 2.3.1   │ airbyte-integrations/connectors/destination-bigquery  │                             
┃ │ X        │ destination-s3        │ java     │ certified     │ 0.5.1   │ airbyte-integrations/connectors/destination-s3        │                             
┃ │ X        │ destination-snowflake │ java     │ certified     │ 3.4.2   │ airbyte-integrations/connectors/destination-snowflake │                             
┃ │ X        │ source-mongodb-v2     │ java     │ certified     │ 1.0.3   │ airbyte-integrations/connectors/source-mongodb-v2     │                             
┃ │ X        │ source-mysql          │ java     │ certified     │ 3.1.3   │ airbyte-integrations/connectors/source-mysql          │                             
┃ │ X        │ source-postgres       │ java     │ certified     │ 3.2.14  │ airbyte-integrations/connectors/source-postgres       │                             
┃ └──────────┴───────────────────────┴──────────┴───────────────┴─────────┴───────────────────────────────────────────────────────┘       

I will run the connectors test manually on each of there:

• destination-bigquery: https://github.com/airbytehq/airbyte/actions/runs/6694703815
• destination-s3: https://github.com/airbytehq/airbyte/actions/runs/6694708978
• destination-snowflake: https://github.com/airbytehq/airbyte/actions/runs/6694713214
• source-mongodb-v2: https://github.com/airbytehq/airbyte/actions/runs/6695322794
• source-mysql: https://github.com/airbytehq/airbyte/actions/runs/6694729594
• source-postgres: https://github.com/airbytehq/airbyte/actions/runs/6694736590

all ✅

[stephane-airbyte]

this is a dangerous file. won't even display the diff in github web UI... And even if it did, it would be unreadable since it's just a single line... Would you be open to remerging to main, if I reformat this file first?

[stephane-airbyte]

unless this is automatically generated by some test, in which case we should modify that to add proper formatting

[stephane-airbyte]

actually, this needs to be added to the .gitignore. Or rather replace the current entry

[postamar]

This file is even more dangerous than you think, in this project's gradle.build you'll find a downloadConnectorRegistry task that's a dependency of processResources and this task HTTP GETs this json file from one of our services. Yeah...

Still, this file shouldn't have been touched in this PR, only moved. I'll look into this. Thanks for noticing

[postamar]

Sorry I responded to your top-level comment without refreshing the page. You're right, this needs a gitignore update. Thanks!

[stephane-airbyte]

I tend to be paranoid about very large changes. In a previous job, someone fixed spotbugs in a 3k+ files at once and at the same time buried a change to the javac command line. So thank you for not doing that :)

[postamar]

Same here. I try to have each change either change form or function but not both. This is a "form" change for sure.

[stephane-airbyte]

looks like we're setting useLocalCdk to true in a lot of gradle files. Do we understand the ins and outs of doing that?

1. does that mean that any changes to the CDK will always be pushed to the individual connectors, instead of each connector pinning their CDK version?
2. are we OK with that change of behavior?
3. if so, why wasn't it done to begin with?
4. if everything becomes useLocalCdk, why not make that the top level default and remove all the override-to-default?

[postamar]

The useLocalCdk = true will be reverted once this PR is approved. The current workflow is to publish a new version of the CDK by writing a github comment containing publish-cdk ... and then update each connector's gradle file with the new CDK version and setting useLocalCdk back to false.

This workflow is veeeery much a work in progress and not something we intend to settle on. The CDK is fairly new as a concept, we haven't fleshed everything out.

[postamar]

Thanks for the review! I'll wait for the github actions to complete and then I'll publish the CDK with version 0.2.0, and so forth, before merging.

[postamar]

/publish-java-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/6696909893
❌ Publish Java CDK version=0.2.0 failed!

[postamar]

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6697169665
✅ Successfully published Java CDK version=0.2.0!

[postamar]

/approve-and-merge reason="bump CDK deps for all connectors"

[postamar]

Approve-and-merge failed with client_payload is too large., merging manually instead.

[edgao]

I missed this PR earlier - we intentionally excluded the typing-deduping modules from the initial CDK release because they're still under very active development and going through a lot of breaking changes. Is there a technical barrier to unhoisting those two specific projects?

[postamar]

I didn't realize that. There's no obstacle to unhoisting these projects and they're pretty far down the dependency chain anyway. Alternatively, we can simply ignore the published jars for the time being and redefine the dependencies to typing-deduping as being from source, like previously. At that point, the only difference will be the gradle project name. I'll open a PR and add you as a reviewer.