Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-ci: burst gradle task cache on new java cdk release #36480

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Mar 26, 2024

We want to burst the gradle dependency cache when a new CDK version is published.

Copy link

vercel bot commented Mar 26, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Apr 3, 2024 8:07am

Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite Graphite

@alafanechere alafanechere marked this pull request as ready for review March 26, 2024 10:26
@alafanechere alafanechere requested a review from a team as a code owner March 26, 2024 10:26
@alafanechere alafanechere force-pushed the augustin/03-26-airbyte-ci_burst_gradle_task_cache_on_new_java_cdk_release branch from deafcff to df5f805 Compare March 26, 2024 10:27
@alafanechere alafanechere force-pushed the augustin/03-26-airbyte-ci_burst_gradle_task_cache_on_new_java_cdk_release branch from df5f805 to 12cb59a Compare March 26, 2024 10:39
@alafanechere alafanechere force-pushed the augustin/03-26-airbyte-ci_burst_gradle_task_cache_on_new_java_cdk_release branch from 12cb59a to 4fddfa7 Compare March 26, 2024 10:52
@postamar
Copy link
Contributor

This seems like overkill for a very transient problem. The airbyte-ci codebase is already exceedingly complex and I'm not keen to add mysterious code to handle edge cases like these.

Can we instead bust the cache by having the user check a box when running the workflow manually in the github actions page?

@alafanechere
Copy link
Contributor Author

This seems like overkill for a very transient problem. The airbyte-ci codebase is already exceedingly complex and I'm not keen to add mysterious code to handle edge cases like these.

Can we instead bust the cache by having the user check a box when running the workflow manually in the github actions page?

The implementation would be slightly more complex and would only work on workflow dispatch...
I believe my approach is cheaper and correct.
An alternative would be to eagerly download the airbyte-cdk without caching - can we do it easily in a single gradle command?

Copy link
Contributor

Why is the CDK special? This failure mode can occur with literally any dependency. It's more likely to happen with the CDK because we're hosting it in our own (rather shitty) maven repo instead of pulling from maven central.

@alafanechere
Copy link
Contributor Author

Why is the CDK special? This failure mode can occur with literally any dependency. It's more likely to happen with the CDK because we're hosting it in our own (rather shitty) maven repo instead of pulling from maven central.

I think the CDK is special as developers reference its version in connectors before actually publishing it. This is what's causing the problem.

@alafanechere alafanechere force-pushed the augustin/03-26-airbyte-ci_burst_gradle_task_cache_on_new_java_cdk_release branch from 4fddfa7 to 89e1dbd Compare April 3, 2024 08:06
@alafanechere alafanechere enabled auto-merge (squash) April 3, 2024 08:08
@alafanechere alafanechere merged commit 4288dcb into master Apr 3, 2024
30 checks passed
@alafanechere alafanechere deleted the augustin/03-26-airbyte-ci_burst_gradle_task_cache_on_new_java_cdk_release branch April 3, 2024 08:21
@postamar
Copy link
Contributor

postamar commented Apr 3, 2024

I think the CDK is special as developers reference its version in connectors before actually publishing it.

Sorry, I'd overlooked this very important point!

msaffitz added a commit to Encore-Post-Sales/airbyte-magnify that referenced this pull request Apr 3, 2024
* master: (1562 commits)
  ✨ source-google-drive: migrate to poetry  (airbytehq#36581)
  enable spotbugs for CDK core and dependencies submodule (airbytehq#36612)
  ✨ Source Salesforce, Shopify: add maxSecondsBetweenMessages in metadata (airbytehq#36723)
  java-cdk: re-export airbyte-api dependency (airbytehq#36759)
  Source Google Sheets: address dependency conflict and update CDK (airbytehq#36515)
  Airbyte CI: rename incorrectly named pipelines (airbytehq#36722)
  Source Azure Blob Storage: add integration tests (airbytehq#36542)
  Salesforce: retry on download_data and create_stream_job (airbytehq#36385)
  ✨Source Monday: Bumped CDK version dependency (airbytehq#36746)
  airbyte-ci: burst gradle task cache on new java cdk release (airbytehq#36480)
  chore(connectors): remove tasks.py and top-level requirements.txt (airbytehq#36738)
  airbyte-ci: fix pull-request-number option for migrate_to_base_image (airbytehq#36779)
  🤖 Bump patch version of Python CDK
  add backward compatibility for an old close slice logic (airbytehq#36774)
  Bump Airbyte version from 0.57.0 to 0.57.1
  🤖 Bump patch version of Python CDK
  low-code: Fix cursor pagination instantiation if the stop_condition is a string (airbytehq#36760)
  fix rabbitmq icon and simplify docs registry code (airbytehq#36767)
  Update azure-entra-id.md (airbytehq#36758)
  re-enable rabbitmq on OSS (airbytehq#36749)
  ...
nurikk pushed a commit to nurikk/airbyte that referenced this pull request Apr 4, 2024
markcusack pushed a commit to markcusack/airbyte that referenced this pull request Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants