airbyte-ci: fix pip / cdk caching #31974

alafanechere · 2023-10-30T18:00:46Z

What

We want to install the freshest CDK version on connectors that are not pinning their CDK version.
But the current approach was not valid because:

We picked the latest CDK version from Pypi API, and there's a lag between PyPi API package version availability and availability on pip install, which caused Investigate pip install race condition between cdk publish and connector publish #31793
The shared pip cache volume was not mounted on connector build.

On connector build: always install the latest CDK, guaranteed by a cachebuster env var and a --force-reinstall
Mount pip and poetry cache during image build
Write a test to make sure the airbyte-cdk is correctly installed from the cache

vercel · 2023-10-30T18:00:54Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 31, 2023 8:46am

alafanechere · 2023-10-30T18:00:58Z

Current dependencies on/for this PR:

alafanechere marked this pull request as ready for review October 30, 2023 18:12

alafanechere requested a review from a team October 30, 2023 18:12

bnchrch approved these changes Oct 30, 2023

View reviewed changes

alafanechere added 4 commits October 31, 2023 09:46

airbyte-ci: fix pip / cdk caching

d6028f3

bump version

04edf0c

bump version

f8c016a

bump version

031f613

alafanechere force-pushed the augustin/10-30-airbyte-ci_fix_pip_/_cdk_caching branch from aec217d to 031f613 Compare October 31, 2023 08:46

alafanechere enabled auto-merge (squash) October 31, 2023 08:46

alafanechere merged commit df98303 into master Oct 31, 2023
19 checks passed

alafanechere deleted the augustin/10-30-airbyte-ci_fix_pip_/_cdk_caching branch October 31, 2023 08:58