airbyte-ci: Add pypi publishing logic

[flash1293]

Add pypi publish functionality as a standalone command and as step within connector publish:

Standalone command is in a group poetry:

This is how to publish a poetry project like airbyte lib:

airbyte-ci poetry --package-path=airbyte-lib publish --pypi-token="api-token" --test-pypi

It can also publish legacy projects (this is the functionality used by the connector publish pipeline):

airbyte-ci poetry --package-path=airbyte-integrations/connectors/source-apify-dataset publish --publish-name=airbyte-source-apify-dataset --publish-version="2.1.2" --pypi-token="api-token" --test-pypi

During connector publish, it checks whether the pypi remote registry is enabled - if yes, it checks whether the current version is published already. If no, it will run the same logic as the poetry publish command from above, with the package name set to airbyte-{connector-name} and version set to the docker image tag.

Internally, the package name and version provided to the step will override whatever is defined in the setup.py or pyproject.toml file of the package.

Open questions:

• How to pass the pypi api token to the connector publish step? Currently, it's just reading from os.environ deep in the code, a cleaner approach would probably be nice, but putting this very specific thing on the top level of airbyte-ci options also feels wrong. Happy for expert opinions here
• Is the basic setup any good or should it be structured differently?
• What image should be used to perform the publish? I just went with an image that has poetry pre-installed, but maybe we have some more defined opinions on it. Ideally I would like to not block on this for the initial release of this feature to unblock the functionality first as this is a critical part
• Integration of passing the pypi api token via CI - I need someone with admin rights to configure this on the repo I think
• Testing - what are the best strategies to test this? We can of course mock stuff away but it would be nice to stay as close as possible to the "real" thing. One thing that's difficult with this is that we can't publish the same version multiple times, pypi doesn't allow this

Test connector publish pipeline via https://github.com/airbytehq/airbyte/actions/workflows/publish_connectors.yml

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 24, 2024 3:12pm

[flash1293]

@alafanechere

I addressed your comments, it's ready for another look.

During testing I think I noticed a weird error with the caching. To test the connector publish, I added the remoteRegistries section for pokeapi and ran the workflow

• On the first try it skipped over the pypi section, even though it was enabled in the metadata: https://github.com/airbytehq/airbyte/actions/runs/7594166736/job/20685401062
• Then, I added a debug statement to the airbyte-ci code and ran again. This time, it correctly recognized that pypi remote publishing is enabled for the connector: https://github.com/airbytehq/airbyte/actions/runs/7594192174

The only thing I changed was the airbyte-ci code, because of this I think there is something cached that shouldn't be cached. It's possible this is only an issue during development because it's running airbyte-ci in dev mode, but I wanted to highlight it - could you take a look at this please to make sure it will work correctly in production?

[alafanechere]

Nothing blocking but additional feedback following your latest changes.
I'm wondering to what extent we could remove pypi occurences from the code and replace them by python_registry. I'd love to understand which logic is pypi specific. If we have some pypi specific logic I'd appreciate clearly isolating them so that supporting private / other python registry is easy.
I'll check your caching issue now.

[alafanechere]

@alafanechere

I addressed your comments, it's ready for another look.

During testing I think I noticed a weird error with the caching. To test the connector publish, I added the remoteRegistries section for pokeapi and ran the workflow

• On the first try it skipped over the pypi section, even though it was enabled in the metadata: https://github.com/airbytehq/airbyte/actions/runs/7594166736/job/20685401062
• Then, I added a debug statement to the airbyte-ci code and ran again. This time, it correctly recognized that pypi remote publishing is enabled for the connector: https://github.com/airbytehq/airbyte/actions/runs/7594192174

The only thing I changed was the airbyte-ci code, because of this I think there is something cached that shouldn't be cached. It's possible this is only an issue during development because it's running airbyte-ci in dev mode, but I wanted to highlight it - could you take a look at this please to make sure it will work correctly in production?

I'm pretty sure this problems comes from an issue on this switch.
On your first run the airbyte-ci binary built on master was used, explaining why your new logic was not called.
On the second run airbyte-ci is installed from the branch code.
I believe the "install airbyte-ci dev" works only if you modified airbyte-ci code on your head commit, not on the branch.

[alafanechere]

@flash1293 I successfully tested a source-pokeapi pre-release with both Pypi disabled and enabled.