Add CDK publishing setup.

[davinchia]

What

Allow us to publish CDK versions on PyPi via Github workflows.

This lets us:

1. Version CDK moving forward
2. Publish via Github actions to both PyPi and test PyPi for test releases.

Note, this will not work on this branch as Github only detects new workflows after they are checked into master, so I was forced to develop and test this from a fork. Browse this PR or this PR to see this command in action.

See an example published package here. This was published from my fork. I also published an initial 0.0.1 on actual PyPi to demonstrate how this works with an actual package.

Added publishing instructions to the Release Document.

Pre-merge Checklist

• Remove exchange rates changes
• Create Airbyte PyPi account
• Add PyPi account credentials to Github

How

• Add the required python set up files to properly configure and build the Python package.
• Add the publish-cdk workflow. This operates off the slash command we currently use and uses the mariamrf/py-package-publish-action@v1.1.0 action to build and publish the built package to PyPi/TestPyPi
• The slash command will always publish to Test PyPi unless the dry-run=false flag is passed in as part of the command e.g. /publish-cdk dry-run=false.

Recommended reading order

1. setup.py - read through to understand how the package is configured.
2. publish-cdk-command.yml - understand the publish command
3. slash-commands.yml - understand how this is linked
4. the rest

[davinchia]

@sherifnada finally done! one question: what's Airbyte email should I use to set up an Airbyte PyPi account? Should I set up a new email for this?

[jrhizor]

looks good % comments

[keu]

not sure if this is the right ticket, but please makes sure the package works before publishing to PyPI

[davinchia]

@keu yeap. The CDK is working right now. Thanks for checking.

[keu]

@keu yeap. The CDK is working right now. Thanks for checking.

sorry, I think I didn't make my point clear there, right now CDK is working because we install it from monorepo,
when we change this and will install it from PYPI it should be tested before publishing, i.e. in the CI to prevent packaging issues. I see it like this:

• build package
• go to the temp folder and install package (prevent using local version)
• run dummy test or tests included in the package
• publish

[davinchia]

@keu yeap I get you. I should also be clearer - I tested on the source-exchange-rates package from Test PyPi before publishing on PyPi. The required source-exchange-rates changes are checked in if you are interested, although I am going to remove them before merging since they currently don't play well with the docker entry points.

Agree with you on tests and package-stability. Right now, I've published to PyPi (and Test PyPi) only to demonstrate the publishing mechanism works since Github actions are tricky. It was also a sanity test to see if source-exchange-rates works with the published PyPi package. Moving forward, publishing will follow the flow you described. In fact, tests are the next thing Sherif and I are adding to the CDK.

[davinchia]

Created PyPi and Test PyPi accounts for us. Credentials are in Lastpass. Same password and user for both. Added to secrets.