Python packaging and uploading to PyPI

[udosson]

This PR contains the building and publishing of the Python package of Plenum to PyPI.
A successful test tun of the PR can be found in this GHA run. For testing purposes, the indy-plenum package was published to TestPyPi. The indy-plenum package of the test can be installed via python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple indy-plenum==1.13.0.dev133.

@ryjones could you create a secret called PYPI_API_TOKEN according to PyPI docs, please? This is needed to publish the package to PyPI. Thanks!

Signed-off-by: udosson r.klemens@yahoo.de

[sovbot]

Can one of the admins verify this patch?

[udosson]

Could be removed here and upstreamVer could be used from the job build_release. However, setting the build version here as well allows for parallel processing

[WadeBarnes]

@udosson, .github/actions/publish-pypi/prepare-package.sh is an adaptation of build-scripts\ubuntu-1604\prepare-package.sh. Why not just update build-scripts\ubuntu-1604\prepare-package.sh? Don't the dependencies still need to be updated as in build-scripts\ubuntu-1604\prepare-package.sh; https://github.com/hyperledger/indy-plenum/blob/master/build-scripts/ubuntu-1604/prepare-package.sh#L21-L24

[udosson]

@WadeBarnes thanks for your comment.
I tried to use build-scripts\ubuntu-1604\prepare-package.sh without modifying it like in this GHA run #127 https://github.com/udosson/indy-plenum/blob/5d2cb7b07ebbda5f297a39e5780011d0dceed4ea/.github/workflows/build.yaml#L442.

However, if I try to install python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple indy-plenum==1.13.0.dev127 I run into the following error:

Collecting ujson==1.33-1build1 (from indy-plenum==1.13.0.dev127)
  Could not find a version that satisfies the requirement ujson==1.33-1build1 (from indy-plenum==1.13.0.dev127) (from versions: 0.1.dev1, 1.4, 1.6, 1.8, 1.9, 1.15, 1.18, 1.19, 1.21, 1.22, 1.23, 1.30, 1.33, 1.34, 1.35, 1.36.dev64, 1.36.dev65, 1.36.dev66, 1.36.dev74, 1.36.dev78, 1.36.dev80, 1.36.dev83, 1.36.dev85, 1.36.dev87, 2.0.0, 2.0.1.dev8, 2.0.1, 2.0.2.dev3, 2.0.2.dev11, 2.0.2, 2.0.3.dev28, 2.0.3.dev30, 2.0.3, 2.0.4.dev1, 2.0.4.dev2, 2.0.4.dev4, 2.0.4.dev7, 2.0.4.dev14, 2.0.4.dev47, 2.0.4.dev58, 2.0.4.dev96, 2.0.4.dev103, 2.1, 2.1.10, 2.1.11, 2.1.15, 2.1.16, 2.1.17, 2.1.19, 2.1.22, 2.1.23, 3.0.0, 3.0.1.dev3, 3.0.1.dev10, 3.0.1.dev17, 3.0.1.dev20, 3.1.0, 3.1.1.dev2, 3.1.1.dev4, 3.1.1.dev7, 3.2.0, 3.2.1.dev2, 3.2.1.dev4)
No matching distribution found for ujson==1.33-1build1 (from indy-plenum==1.13.0.dev127)

https://github.com/hyperledger/indy-plenum/blob/master/build-scripts/ubuntu-1604/prepare-package.sh#L21-L24 sets the python packages according to the versions of the packages in Launchpad. E.g. ujson==1.33-1build1 for xenial https://launchpad.net/ubuntu/+source/ujson/1.33-1build1.
But there is no PyPI packages called ujson==1.33-1build1. It's just ujson==1.33 https://pypi.org/project/ujson/1.33/

I think the reason for this pinning is because ujson, prompt_toolkit, and msgpack-python are already pre-installed in Ubuntu 16:04. That's why the dependencies for the Debian package are changed to the version of the Canonical archive. Thus, there is no need for distributing the three python packages. Also, you'll notice that these three packages aren't listed in build-scripts\ubuntu-1604/build-3rd-parties.sh (https://github.com/hyperledger/indy-plenum/blob/master/build-scripts/ubuntu-1604/build-3rd-parties.sh#L88)

###Plain ubuntu:1604 container
root@d33a149f3931:/# apt list | grep ujson  
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

python-ujson/xenial 1.33-1build1 amd64
python-ujson-dbg/xenial 1.33-1build1 amd64
python3-ujson/xenial 1.33-1build1 amd64
python3-ujson-dbg/xenial 1.33-1build1 amd64

For the PyPI package, we need to install the dependencies as such as ujson==1.33 and we cannot change the version tag to the version of the Canonical archive.

[WadeBarnes]

I just came to the same conclusion. I dug a bit deeper and I see that it is only needed for the deb package build, and the version of prepare-package.sh you added is the one used by the Jenkins shared libraries for the PyPi package build.

I think we should merge the two scripts together so there isn't the confusion over having two almost identical scripts. I can take a pass at that.

[WadeBarnes]

@udosson, My other recommendation would be to separate the PyPi package build from the publishing to keep it consistent with how the plenum and 3rd-party-dependency packages are being managed. That would allow the publish_artifacts job to be completely responsible for publishing all packages. Do you want to take a pass at that?

That way we're always excising the package builds, and then only publishing them when appropriate.

[udosson]

@WadeBarnes thanks for your suggestion.
So we'll have three builds build_release, build_3rd_party_dependencies, build-python-package running in parallel and then we either publish everything or nothing. Thanks for the clarification via rocker chat. I'll update this PR prior to our meeting tomorrow.

[ryjones]

@ryjones could you create a secret called PYPI_API_TOKEN according to PyPI docs, please? This is needed to publish the package to PyPI. Thanks!

Done!

[andrewwhitehead]

As of #1548 the pip version does not need to be overridden

[udosson]

Thanks for the contribution, @andrewwhitehead. I update the PR and removed the pinned pip version.

[ryjones]

@udosson any chance for a squash-like commit to tidy up and fix the DCO issue?

[udosson]

@ryjones, done!

[WadeBarnes]

(ci) test this please