Automate publishing release to pypi

[aleksandr-kotlyar]

Problem

Selene doesn't have auto-deploy script for its releases. All work is doing manually through console.

What

Automate release building and publishing to pypi when new tag has been pushed into repo.

ToDo

• 1. Write GitHub-Action workflow publish.yml.
• 2. Test build job by publishing new release on github.

Definition of done

• When new release is published on github - publishing job starts automatically.
• Publishing job publishes new selene release to pypi.
• Selene release is available to install from pypi for example by pip install selene.

Guide

https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

Publishing package distribution releases using GitHub Actions CI/CD workflows

Publishing package distribution releases using GitHub Actions CI/CD workflows¶

GitHub Actions CI/CD allows you to run a series of commands whenever an event occurs on the GitHub platform. One popular choice is having a workflow that’s triggered by a push event. This guide shows you how to publish a Python distribution whenever a tagged commit is pushed. It will use the pypa/gh-action-pypi-publish GitHub Action.

Attention

This guide assumes that you already have a project that you know how to build distributions for and it lives on GitHub.

Saving credentials on GitHub¶

In this guide, we’ll demonstrate uploading to both PyPI and TestPyPI, meaning that we’ll have two separate sets of credentials. And we’ll need to save them in the GitHub repository settings.

Let’s begin! 🚀

1. Go to https://pypi.org/manage/account/#api-tokens and create a new API token. If you have the project on PyPI already, limit the token scope to just that project. You can call it something like GitHub Actions CI/CD — project-org/project-repo in order for it to be easily distinguishable in the token list. Don’t close the page just yet — you won’t see that token again.

2. In a separate browser tab or window, go to the Settings tab of your target repository and then click on Secrets in the left sidebar.

3. Create a new secret called PYPI_API_TOKEN and copy-paste the token from the first step.

4. Now, go to https://test.pypi.org/manage/account/#api-tokens and repeat the steps. Save that TestPyPI token on GitHub as TEST_PYPI_API_TOKEN.

   Attention

   If you don’t have a TestPyPI account, you’ll need to create it. It’s not the same as a regular PyPI account.

Creating a workflow definition¶

GitHub CI/CD workflows are declared in YAML files stored in the .github/workflows/ directory of your repository.

Let’s create a .github/workflows/publish-to-test-pypi.yml file.

Start it with a meaningful name and define the event that should make GitHub run this workflow:

name: Publish Python 🐍 distributions 📦 to PyPI and TestPyPI
on: push

Defining a workflow job environment¶

Now, let’s add initial setup for our job. It’s a process that will execute commands that we’ll define later. In this guide, we’ll use Ubuntu 18.04:

jobs:
  build-n-publish:
    name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
    runs-on: ubuntu-18.04

Checking out the project and building distributions¶

Then, add the following under the build-n-publish section:

    steps:
    - uses: actions/checkout@master
    - name: Set up Python 3.7
      uses: actions/setup-python@v1
      with:
        python-version: 3.7

This will download your repository into the CI runner and then install and activate Python 3.7.

And now we can build dists from source. In this example, we’ll use build package, assuming that your project has a pyproject.toml properly set up (see PEP 517/PEP 518).

Tip

You can use any other method for building distributions as long as it produces ready-to-upload artifacts saved into the dist/ folder.

So add this to the steps list:

    - name: Install pypa/build
      run: >-
        python -m
        pip install
        build
        --user
    - name: Build a binary wheel and a source tarball
      run: >-
        python -m
        build
        --sdist
        --wheel
        --outdir dist/
        .

Publishing the distribution to PyPI and TestPyPI¶

Finally, add the following steps at the end:

    - name: Publish distribution 📦 to Test PyPI
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
        repository_url: https://test.pypi.org/legacy/
    - name: Publish distribution 📦 to PyPI
      if: startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

These two steps use the pypa/gh-action-pypi-publish GitHub Action: the first one uploads contents of the dist/ folder into TestPyPI unconditionally and the second does that to PyPI, but only if the current commit is tagged.

That’s all, folks!¶

Now, whenever you push a tagged commit to your Git repository remote on GitHub, this workflow will publish it to PyPI. And it’ll publish any push to TestPyPI which is useful for providing test builds to your alpha users as well as making sure that your release pipeline remains healthy!

https://github.com/pypa/gh-action-pypi-publish

PyPI publish GitHub Action

PyPI publish GitHub Action

This action allows you to upload your Python distribution packages
in the dist/ directory to PyPI.
This text suggests a minimalistic usage overview. For more detailed
walkthrough check out the PyPA guide.

Usage

To use the action add the following step to your workflow file (e.g.
.github/workflows/main.yml)

- name: Publish a Python distribution to PyPI
  uses: pypa/gh-action-pypi-publish@master
  with:
    user: __token__
    password: ${{ secrets.PYPI_API_TOKEN }}

Pro tip: instead of using branch pointers, like master, pin versions of
Actions that you use to tagged versions or sha1 commit identifiers. This will
make your workflows more secure and better reproducible, saving you from sudden
and unpleasant surprises.

A common use case is to upload packages only on a tagged commit, to do so add a
filter to the step:

  if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')

So the full step would look like:

- name: Publish package
  if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
  uses: pypa/gh-action-pypi-publish@master
  with:
    user: __token__
    password: ${{ secrets.PYPI_API_TOKEN }}

The example above uses the new API token feature of
PyPI, which is recommended to restrict the access the action has.

The secret used in ${{ secrets.PYPI_API_TOKEN }} needs to be created on the
settings page of your project on GitHub. See Creating & using secrets.

Non-goals

This GitHub Action has nothing to do with building package
distributions. Users are responsible for preparing dists for upload
by putting them into the dist/ folder prior to running this Action.

Advanced release management

For best results, figure out what kind of workflow fits your
project's specific needs.

For example, you could implement a parallel workflow that
pushes every commit to TestPyPI or your own index server,
like devpi. For this, you'd need to (1) specify a custom
repository_url value and (2) generate a unique version
number for each upload so that they'd not create a conflict.
The latter is possible if you use setuptools_scm package but
you could also invent your own solution based on the distance
to the latest tagged commit.

You'll need to create another token for a separate host and then
save it as a GitHub repo secret.

The action invocation in this case would look like:

- name: Publish package to TestPyPI
  uses: pypa/gh-action-pypi-publish@master
  with:
    user: __token__
    password: ${{ secrets.TEST_PYPI_API_TOKEN }}
    repository_url: https://test.pypi.org/legacy/

Customizing target package dists directory

You can change the default target directory of dist/
to any directory of your liking. The action invocation
would now look like:

- name: Publish package to PyPI
  uses: pypa/gh-action-pypi-publish@master
  with:
    user: __token__
    password: ${{ secrets.PYPI_API_TOKEN }}
    packages_dir: custom-dir/

Disabling metadata verification

It is recommended that you run twine check just after producing your files,
but this also runs twine check before upload. You can also disable the twine
check with:

   with:
     verify_metadata: false

Tolerating release package file duplicates

Sometimes, when you publish releases from multiple places, your workflow
may hit race conditions. For example, when publishing from multiple CIs
or even having workflows with the same steps triggered within GitHub
Actions CI/CD for different events concerning the same high-level act.

To facilitate this use-case, you may use skip_existing (disabled by
default) setting as follows:

   with:
     skip_existing: true

Pro tip: try to avoid enabling this setting where possible. If you
have steps for publishing to both PyPI and TestPyPI, consider only using
it for the latter, having the former fail loudly on duplicates.

For Debugging

Sometimes, twine upload can fail and to debug use the verbose setting as follows:

   with:
     verbose: true

License

The Dockerfile and associated scripts and documentation in this project
are released under the BSD 3-clause license.

[aleksandr-kotlyar]

@yashaka I have prepared the concept of release process with automatic publishing to pypi:

Release process.

Create new release on GitHub.
Choose new tag or take not released yet.
Describe release notes.
Select pre-release checkbox if not stable.
Publish the release on GitHub.

Then GitHub action will automatically build and publish release to pypi with selected tag automatically. Also it will commit the tag semver into __init__.py and pyproject.toml before building if it has not been there yet.

[yashaka]

Great, let's reflect this in corresponding documentation place, like contribution guide or something else