Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add wheel building and deployment via GitHub Actions and cibuildwheel #601

Merged
merged 13 commits into from Oct 24, 2021
182 changes: 182 additions & 0 deletions .github/workflows/wheel_tests_and_release.yml
@@ -0,0 +1,182 @@
name: Build Wheels and Release
on:
push:
branches:
- cibuildwheel
tags:
- 'v*'
- 'buildwheels*'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this for? There seems to be no guard for this kind of tag anywhere, so does it upload a release for them to PyPI without a normal version number?

Same question for the cibuildwheel branch name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove the cibuildwheel branch condition before merging. That was just to make it run everytime I push an update while testing.

Pushing a buildwheels* tag should not trigger deployment. I think the following conditions requires the tag start with v. I assume the repository_owner check is to prevent it running on forks.

  deploy:
    name: Release
    needs: [build_linux_37_and_above_wheels, build_macos_wheels, build_windows_wheels]
    if: github.repository_owner == 'PyWavelets' && startsWith(github.ref, 'refs/tags/v') && always()

I think the needs section ensures that it only gets run if the wheel builds all succeeded.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be removed in a follow-up PR, it's harmless to have in here.

env:
CIBW_BUILD_VERBOSITY: 2
# CIBW_BEFORE_BUILD: pip install cython
CIBW_TEST_REQUIRES: pytest
CIBW_TEST_COMMAND: pytest --pyargs pywt


jobs:
build_linux_37_and_above_wheels:
name: Build python ${{ matrix.cibw_python }} wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-18.04]
cibw_python: [ "cp37-*", "cp38-*" ]
rgommers marked this conversation as resolved.
Show resolved Hide resolved
cibw_manylinux: [ manylinux1 ]
include:
- os: ubuntu-18.04
cibw_python: "cp39-*"
cibw_manylinux: manylinux2010
- os: ubuntu-18.04
cibw_python: "cp310-*"
cibw_manylinux: manylinux2014
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
with:
platforms: arm64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me: we're going to need macOS arm64 too soon. Best to leave for a separate PR. Although would be good to check that it's possible with cibuildwheel or in the works (the multibuild support is available now).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like macOS arm64 should be possible for Python 3.8, 3.9 and 3.10: https://github.com/pypa/cibuildwheel#what-does-it-do

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added universal2 and arm64 wheels on MacOS and it succeeded (see: https://github.com/grlee77/pywt/actions/runs/1334259199). Do you recommend providing the universal2 wheels in addition to the separate x86_64 and arm64 ones?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, universal2 is useful I think, since it's the default for the Python.org installers and because it's the best wheel to grab if you don't know beforehand what you need.

- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel
- name: Build the wheel
run: |
python -m cibuildwheel --output-dir dist
env:
CIBW_BUILD: ${{ matrix.cibw_python }}
CIBW_ARCHS_LINUX: auto aarch64
CIBW_MANYLINUX_X86_64_IMAGE: ${{ matrix.cibw_manylinux }}
CIBW_MANYLINUX_I686_IMAGE: ${{ matrix.cibw_manylinux }}
- uses: actions/upload-artifact@v2
with:
name: wheels
path: ./dist/*.whl

build_macos_wheels:
name: Build wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [macos-latest]
env:
MACOSX_DEPLOYMENT_TARGET: "10.13"
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'

- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel
- name: Build wheels for CPython (MacOS)
run: |
python -m cibuildwheel --output-dir dist
env:
CIBW_BUILD: "cp3?-*" # cp310-*
CIBW_SKIP: "cp35-* cp36-*"
CIBW_MANYLINUX_X86_64_IMAGE: manylinux1
CIBW_MANYLINUX_I686_IMAGE: manylinux1
CC: /usr/bin/clang
CXX: /usr/bin/clang++
# CPPFLAGS: "-Xpreprocessor -fopenmp"
# CFLAGS: "-Wno-implicit-function-declaration -I/usr/local/opt/libomp/include"
# CXXFLAGS: "-I/usr/local/opt/libomp/include"
# LDFLAGS: "-Wl,-rpath,/usr/local/opt/libomp/lib -L/usr/local/opt/libomp/lib -lomp"

- uses: actions/upload-artifact@v2
with:
name: wheels
path: ./dist/*.whl

build_windows_wheels:
name: Build wheels on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [windows-latest]

steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0

- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'
grlee77 marked this conversation as resolved.
Show resolved Hide resolved

- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel
- name: Build Windows wheels for CPython
run: |
python -m cibuildwheel --output-dir dist
env:
CIBW_BUILD: "cp3?-*" # cp310-*
CIBW_SKIP: "cp35-* cp36-*"
CIBW_MANYLINUX_X86_64_IMAGE: manylinux1
CIBW_MANYLINUX_I686_IMAGE: manylinux1

- uses: actions/upload-artifact@v2
with:
name: wheels
path: ./dist/*.whl


deploy:
name: Release
needs: [build_linux_37_and_above_wheels, build_macos_wheels, build_windows_wheels]
if: github.repository_owner == 'PyWavelets' && startsWith(github.ref, 'refs/tags/v') && always()
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: actions/setup-python@v2
name: Install Python
with:
python-version: '3.7'

- name: Install Twine
run: |
python -m pip install --upgrade pip
pip install -r requirements/build.txt
pip install twine

- uses: actions/download-artifact@v2
id: download
with:
name: wheels
path: ./dist

- name: Publish the source distribution on PyPI
run: |
PYWT_VERSION=$(git describe --tags)
python setup.py sdist
ls -la ${{ github.workspace }}/dist
# We prefer to release wheels before source because otherwise there is a
# small window during which users who pip install pywt will require compilation.
twine upload ${{ github.workspace }}/dist/*.whl
twine upload ${{ github.workspace }}/dist/pywt-${PYWT_VERSION:1}.tar.gz
env:
TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }}
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}

- name: Github release
uses: softprops/action-gh-release@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
37 changes: 37 additions & 0 deletions pyproject.toml
@@ -0,0 +1,37 @@
# Note that in maintenance branches, all build dependencies should
# have an upper bound equal to the most recent already-released version
# of the dependency. This to prevent that a future backwards-incompatible
# release will break the source build of a PyWavelets release.


[build-system]
requires = [
"wheel",
"setuptools",
"Cython>=0.29.18",

# NumPy dependencies - to update these, sync from
# https://github.com/scipy/oldest-supported-numpy/, and then
# update minimum version to match our install_requires min version
# ----------------------------------------------------------------

# numpy 1.19 was the first minor release to provide aarch64 wheels, but
# wheels require fixes contained in numpy 1.19.2
"numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64'",
# aarch64 for py39 is covered by default requirement below

# arm64 on Darwin supports Python 3.8 and above requires numpy>=1.20.0
"numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin'",
"numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin'",

# default numpy requirements
"numpy==1.17.3; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy'",
"numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy'",

# For Python versions which aren't yet officially supported,
# we specify an unpinned NumPy which allows source distributions
# to be used and allows wheels to be used as soon as they
# become available.
"numpy; python_version>='3.10'",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add 3.10 now, or leave that to a quick follow-up PR? We need 3.10 wheels, forgot about those till this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a look at this now.

"numpy; python_version>='3.8' and platform_python_implementation=='PyPy'",
]