java CDK: build no longer downloads files from connector registry

[postamar]

These files were part of the CDK build and formed a circular dependency: their content is updated based on the connector metadata when a connector is published. Incidentally, removing these also allows removing the dependencies on micronaut.

This PR is best reviewed commit-by-commit:

• 95f6e26 removes the whole init-oss module and changes DestinationAcceptanceTest, which is the only downstream dependency, to rely on the connector's metadata.yaml file instead of the connector metadata json blob.
• 40bf905 removes all references to micronaut in the repo; this is made possible by the previous commit.
• b25fe6f removes the dependency on the specs_secrets_mask.yaml file which was used by a custom log4j plugin to scrub secrets from the logs. The log config in this repo is now only used for testing and since tests are run via airbyte-ci the secrets will already be scrubbed by dagger.
• the final two commits are version accounting boilerplate and also ensure the changes are validated on a couple of connectors.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 25, 2024 7:01pm

[github-actions]

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage	71.63%	🍏

[postamar]

Confirmed with @alafanechere that the test logs will be scrubbed in github, but it's possible that the test logs end up in some other report, like a gradle scan, and remain unscrubbed. I'll investigate further.

[postamar]

@alafanechere could you review my last commit 4e1a328 please?
This should ensure that the java logs never leak any secrets. I've tested this manually.

[alafanechere]

To what extent would it be harder / simpler to create this pattern within the containers under test instead of building the pattern in the Python logic?

I'm also wondering if we could declare this logic within the GradleTask step.

Slightly out of scope:
Would it also be worth configuring the log scrubbing at the connector image level at runtime: when the config is parsed

[alafanechere]

Could you please add a unit test for this? 🙏

[postamar]

Yes! sorry it slipped my mind

[postamar]

Thanks for taking a look! I'll add the unit test.

To what extent would it be harder / simpler to create this pattern within the containers under test instead of building the pattern in the Python logic?

Unfortunately, the scrubbing needs to be applied to all logs, including the test framework, and not just the logs emitted by the connector container. Also, not all secrets are equal: the scrubbing should only apply to those secrets downloaded by ci_credentials, we don't want to scrub passwords for testcontainer instances (which furthermore are often dumb strings like "password" or "test").

I'm also wondering if we could declare this logic within the GradleTask step.

The integration test task class would be its proper place but for some reason, dagger secrets are defined by the PipelineContext, which isn't something I'm a fan of. This is why I moved the pattern generation logic to util instead of inlining it.

Slightly out of scope:
Would it also be worth configuring the log scrubbing at the connector image level at runtime: when the config is parsed

Indeed. @stephane-airbyte and I are also thinking about how to properly scrub secrets in prod. It's a bit more involved in that case.

[postamar]

I added a unit test and checked again that the scrubbing works (it does).

[postamar]

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7658937523
✅ Successfully published Java CDK version=0.15.0!