Automatically build and upload PyPI packages on release #69

dralley · 2020-02-17T17:56:29Z

Using Github Actions

This will build and upload both source and binary Python packages automatically every time a new release is published. The binary packages should enable Python users to use libcomps across any (linux) platform without needing the compile-time dependencies needed by Python source distributions.

We (current maintainers of the PyPI libcomps package) will need to add your PyPI account as a maintainer, and then you will need to add your credentials to the github repo secrets. Afterwards it should be automated and we will ensure it continues working if any problems arise in the future.

Here's a snapshot of using the binary package built on Centos 7 on a clean Debian VM with pip 20 (older versions can't use the binary package)

vagrant@debian10:~/devel/community/libcomps$ python3 -m pip --version
pip 20.0.2 from /home/vagrant/.local/lib/python3.7/site-packages/pip (python 3.7)
vagrant@debian10:~/devel/community/libcomps$ python3 -m pip install wheelhouse/libcomps-0.1.14.post1-cp37-cp37m-manylinux2014_x86_64.whl 
Defaulting to user installation because normal site-packages is not writeable
Processing ./wheelhouse/libcomps-0.1.14.post1-cp37-cp37m-manylinux2014_x86_64.whl
Installing collected packages: libcomps
Successfully installed libcomps-0.1.14.post1
vagrant@debian10:~/devel/community/libcomps$ cd libcomps/src/python/tests/
vagrant@debian10:~/devel/community/libcomps/libcomps/src/python/tests$ python3 -m pytest __test.py 
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.7.3, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/vagrant/devel/community/libcomps
collected 82 items                                                                                                                                                                                                                           

__test.py ..................................................................................                                                                                                                                           [100%]

============================================================================================================= 82 passed in 0.73s =============================================================================================================
vagrant@debian10:~/devel/community/libcomps/libcomps/src/python/tests$

dralley · 2020-11-09T18:04:40Z

If you're not interested in this, let me know and I can just close the PRs.

lukash · 2020-11-13T13:29:58Z

@dralley I'm planning to attempt to set this up in the next few days, sorry about the long silence.

dralley · 2020-12-23T06:56:51Z

@lukash I have added building documentation and running the C tests with check to the CI

lukash

Hi @dralley, sorry about keeping you waiting. I've created an account on PyPI for the team: rpm-software-management. You can add us as maintainers 🙂

Some comments below, in addition, could you please split this into three commits:

python build CI workflow
python release workflow
the rest of the updates along with a bit of description (possibly split it into more commits if there are more unrelated fixes, but no need to go too fine-grained if you don't want to)

Thanks!

lukash · 2021-03-12T15:30:20Z

.github/workflows/ci.yml

+on: [push, pull_request]
+
+jobs:
+  test_standard_build:


Could you drop this job, testing rpm build is part of our standard CI workflow now?

lukash · 2021-03-12T15:31:53Z

.github/workflows/ci.yml

+          make tests
+          make test
+
+  test_python_build:


Could you update this job to be identical to the createrepo_c one, except for necessary differences?

lukash · 2021-03-12T15:32:51Z

.github/workflows/release.yml

+    types: [created]
+
+jobs:
+  deploy:


Is it possible to test this job without making a release?

We could trigger it on push and tell it to use the credentials for the "test PyPI" instance rather than the production one.

I'm not really sure it's necessary though, we can check everything up to point of the upload pretty easily. https://github.com/rpm-software-management/createrepo_c/pull/256/checks?check_run_id=2099124911

https://github.com/rpm-software-management/libcomps/runs/2100839496?check_suite_focus=true

lukash · 2021-03-12T15:33:21Z

.github/workflows/release.yml

+    - name: Publish wheels to PyPI
+      env:
+        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}


Is it necessary to use the password here, or could an API token be used instead?

It's a little awkward to do but it looks like it's possible

lukash · 2021-03-12T15:44:50Z

One more thing, could you name the workflows ci-python.yml and release-python.yml? I've renamed the createrepo_c one, for consistency.

dralley · 2021-03-12T15:59:26Z

@lukash This is great! Yes, I can update the PR.

lukash · 2021-03-15T10:13:56Z

.github/workflows/ci-python.yml

+        run: |
+          sudo dnf -y install dnf-plugins-core
+          sudo dnf -y builddep libcomps.spec
+          pip install --upgrade pip


At this stage, nitpicking 🙂 but is the upgrade call necessary? (Just trying to keep the variance between workflows minimal, we don't have this in createrepo_c, either it's a different case or we can be uniform?)

It's not necessary on Fedora, it is necessary on images with older versions of pip. Without it you need to manually install "skbuild" because pip doesn't know how to read pyproject.toml

With that said, if you would prefer to either drop it or do the manual install, either of those are fine with me.

Well, the workflow is running on the latest fedora, so it's always going to work here IIUC? (if so, I'd prefer to drop it)

.github/workflows/ci-python.yml

lukash · 2021-03-15T10:16:03Z

.github/workflows/release-python.yml

+
+    - name: Install and Test Python universal binary wheel package
+      run: |
+        pip install --user dist/libcomps-0.1.15.post1-cp39-cp39-manylinux2014_x86_64.whl


Hardcoded version of the .whl file here.

Since we have to manually specify the versions of Python to build wheels for, I figured this was OK, because if Fedora bumps their Python version then we need to edit the file anyway.

However, it wouldn't be ideal to only discover this when a new release is attempted. Maybe we should freeze the container image to Fedora 33?

I'd leave it on fedora:latest.

But apart from the python version (for which I don't have a solution), there's also the libcomps version itself (0.1.15), that's the main problem I was pointing out 🙂

Oh, right. I was looking at the cp39-cp39 section because I spent half an hour trying to get glob patterns to work.

But a glob pattern for the libcomps version would be fine.

@dralley any plans to address this? It's the main outstanding issue with the PR, the rest is mostly nitpicks...

Sorry, I've been really busy. I just pushed the commit.

Okay, no worries. But can you squash the fix with the commit that it fixes?

Also, looking at this, another question: It only releases the binary package for the python listed here, that is, python 3.9 (inferring from cp39), correct? So this won't work for distros with any other python than what we decide to do the release for?

No, it only tests that package, but it releases all of them.

(It's the only one it can test, because you can't use a wheel built for 3.7 with Python 3.9, etc).

Oh, right, sorry 🙂

lukash · 2021-03-15T10:16:54Z

MANIFEST.in

@@ -0,0 +1,7 @@
+graft libcomps*


Is adding this file intentional?

Yes, it ensures that source packages can be built properly, because otherwise not all the necessary files are included by default. Granted that source packages are only a "backup" if for some reason there isn't a wheel available that is compatible with your platform.

There are attempts in the Python ecosystem to get rid of all the separate configuration files that are needed (setup.py, manifest.in, etc.) and standardize on pyproject.toml, but it will be a while until they can be reliably avoided.

That's a bit annoying, I don't have thorough knowledge of the python build system, but since this only needs to be added now, I assume it wasn't needed until now and thus is used only in this special case. That makes it quite hard to maintain for someone who doesn't know all the details.

lukash · 2021-03-15T10:26:11Z

Ah, also: I assume this is still WIP because of the __TOKEN__ placeholder. Just a heads-up, it seems you are using this PR branch for testing, which is generating quite a lot of spam mail. Could you possibly test on a different branch and push here when finished? 😁 Cheers!

dralley · 2021-03-15T13:44:20Z

@lukash Apologies for the spam! It might be configurable, because we're also using Github Actions in our org and I don't get spammed by everyone else's failed PR runs.

Re: __token__, apparently that is how it's actually supposed to be done. I agree, it's awkward and "feels" wrong.

https://packaging.python.org/specifications/pypirc/#using-a-pypi-token

lukash · 2021-03-15T16:30:44Z

@lukash Apologies for the spam! It might be configurable, because we're also using Github Actions in our org and I don't get spammed by everyone else's failed PR runs.

I didn't mean the spam of the failed workflow runs, just the spam of repeatedly force-pushing to the branch. I don't want to turn off those notifications, I use them.

IMO people should only create PRs when the code is ready to be reviewed and same for updating PRs. If you push your WIPs, besides spamming it's actually not clear when your PR is ready for a review (I don't want to review WIP unless explicitly asked for feedback on it). Just my PoV, not sure how others see this.

Re: __token__, apparently that is how it's actually supposed to be done. I agree, it's awkward and "feels" wrong.

https://packaging.python.org/specifications/pypirc/#using-a-pypi-token

Oh, that's actually not terrible, so the __TOKEN__ basically tells it the password is a token, sorry, I just missed that!

dralley · 2021-03-15T17:25:51Z

IMO people should only create PRs when the code is ready to be reviewed and same for updating PRs. If you push your WIPs, besides spamming it's actually not clear when your PR is ready for a review (I don't want to review WIP unless explicitly asked for feedback on it). Just my PoV, not sure how others see this.

I understand. In this case, I was just testing the release CI (temporarily changed the trigger to "pull_request" in order to do so). Both PRs are reviewable at the present moment.

Manylinux wheels include libython statically, so trying to dynamically link the lib will cause the build to fail.

lukash · 2021-03-24T12:25:21Z

@dralley I just force-pushed your master 😬 hope it's ok. I made a few cosmetic adjustments, as well as renaming the PYPI_API_TOKEN secret and changed the Python CI to run on pull_request only (same as I did on createrepo_c). Let's see how it works, we'll need to keep an eye on the release workflow once we actually do a release, libcomps is released very infrequently.

dralley force-pushed the master branch from cab2939 to 117c3b1 Compare February 17, 2020 20:41

dralley force-pushed the master branch from 117c3b1 to afa3e36 Compare November 9, 2020 18:04

dralley force-pushed the master branch 5 times, most recently from acdd7dc to 2c8b988 Compare November 13, 2020 02:58

dralley force-pushed the master branch 4 times, most recently from f72a8ed to e24a1aa Compare December 23, 2020 06:53

dralley force-pushed the master branch 4 times, most recently from be1cc67 to 7a438e0 Compare December 24, 2020 05:09

dralley force-pushed the master branch from 7a438e0 to b5bd699 Compare January 6, 2021 14:05

lukash reviewed Mar 12, 2021

View reviewed changes

dralley force-pushed the master branch 8 times, most recently from 2a80d70 to 6c0f962 Compare March 13, 2021 02:49

dralley force-pushed the master branch 4 times, most recently from 5ca1905 to 8a54129 Compare March 13, 2021 03:28

Miscellaneous docs & metadata updates

e750b2e

dralley force-pushed the master branch 12 times, most recently from 877f5ef to a6ff43f Compare March 13, 2021 05:43

lukash reviewed Mar 15, 2021

View reviewed changes

lukash self-assigned this Mar 23, 2021

dralley force-pushed the master branch from 5292864 to a6a51f4 Compare March 23, 2021 17:25

dralley added 3 commits March 24, 2021 13:14

Add CI which tests installation of the Python package via pip

a20784c

Add CI to publish Python packages to PyPI on new releases

7ec53b2

Adjustments to Python wheels work properly

d73d27e

Manylinux wheels include libython statically, so trying to dynamically link the lib will cause the build to fail.

lukash force-pushed the master branch from a6a51f4 to d73d27e Compare March 24, 2021 12:22

lukash merged commit 78aaa6d into rpm-software-management:master Mar 24, 2021

Automatically build and upload PyPI packages on release #69

Automatically build and upload PyPI packages on release #69

dralley commented Feb 17, 2020 •

edited

dralley commented Nov 9, 2020

lukash commented Nov 13, 2020 •

edited

dralley commented Dec 23, 2020

lukash left a comment

lukash Mar 12, 2021

dralley Mar 12, 2021

lukash Mar 12, 2021

dralley Mar 12, 2021

lukash Mar 12, 2021

dralley Mar 12, 2021

dralley Mar 13, 2021

lukash Mar 12, 2021

dralley Mar 12, 2021

lukash commented Mar 12, 2021

dralley commented Mar 12, 2021

lukash Mar 15, 2021

dralley Mar 15, 2021

lukash Mar 15, 2021

lukash Mar 15, 2021

dralley Mar 15, 2021

lukash Mar 15, 2021

dralley Mar 16, 2021

lukash Mar 23, 2021

dralley Mar 23, 2021

lukash Mar 23, 2021

dralley Mar 23, 2021

lukash Mar 24, 2021

lukash Mar 15, 2021

dralley Mar 15, 2021

lukash Mar 15, 2021

lukash commented Mar 15, 2021

dralley commented Mar 15, 2021

lukash commented Mar 15, 2021

dralley commented Mar 15, 2021 •

edited

lukash commented Mar 24, 2021

Automatically build and upload PyPI packages on release #69

Automatically build and upload PyPI packages on release #69

Conversation

dralley commented Feb 17, 2020 • edited

dralley commented Nov 9, 2020

lukash commented Nov 13, 2020 • edited

dralley commented Dec 23, 2020

lukash left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukash commented Mar 12, 2021

dralley commented Mar 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukash commented Mar 15, 2021

dralley commented Mar 15, 2021

lukash commented Mar 15, 2021

dralley commented Mar 15, 2021 • edited

lukash commented Mar 24, 2021

dralley commented Feb 17, 2020 •

edited

lukash commented Nov 13, 2020 •

edited

dralley commented Mar 15, 2021 •

edited