Skip to content

Commit

Permalink
CI: Add a Cygwin run to GHA CI.
Browse files Browse the repository at this point in the history
This involved a good bit of debugging of the build process, especially
for the bundled FreeType.  Eventually removing the debugging flags let
it just work.

There are still fork failures, but I've done everything I know to do.
CYGWIN=detect_bloda may reveal more, but may not.

parent 48e5934
author DWesl <22566757+DWesl@users.noreply.github.com> 1651882104 -0400
committer DWesl <22566757+DWesl@users.noreply.github.com> 1670678538 -0500

parent 48e5934
author DWesl <22566757+DWesl@users.noreply.github.com> 1651882104 -0400
committer DWesl <22566757+DWesl@users.noreply.github.com> 1670678484 -0500

CI: Add a Cygwin run to GHA CI.

Let's see if I translated the workflow even close to correctly.

CI: Pin Cygwin setuptools<60 and NumPy !=1.21

setuptools>60 breaks Pillow install, and may cause problems for NumPy.

NumPy 1.21 has intermittent segfaults on GHA runners, so let's just skip that.

CI: Pin Cygwin setuptools<60 and avoid problem upgrades

setuptools>=60 runs into python-pillow/Pillow#6216 and pypa/setuptools#3304.

NumPy==1.21.* has intermittent segfaults due to SIMD attempts on unaligned memory.

kiwisolver, numpy, and pillow have problems installing, so just verify that they are present, don't try to upgrade them.

CI: Upgrade Cygwin pip

CI: Install g++ and print info on Python.h

Hopefully one of these will convince NumPy to compile.

CI: Install libjpeg-devel for Pillow install

Let's see if it keeps setuptools old enough for this to work.

CI: BLD: Use system freetype and qhull

Given that I'm checking a build problem at the moment, this should allow that to show up faster.

This should probably be reverted before merging.

CI: Specify shell for Cygwin pytest step.

CI: Rebase Matplotlib DLLs before running pytest

Try to avoid "BlockingIOError: Resource Unavailable"
with a fork() error on stderr.

CI: Compile against bundled freetype.

See if this fixes the various test errors.

It will take a few tries to get freetype to actually compile, I think.

CI: Install make for libfreetype build

Let's see if this lets the build run to completion.

BLD: Run autoconf before installing bundled freetype

Let's see if this fixes the install problems.

CI: Try to get Cygwin build of FreeType working.

It doesn't fail the same way locally.

CI: Revert to actually-configured python alternatives.

BLD: Fix autoconf call on Cygwin.

BLD: Fix directory for Cygwin autoreconf.

CI: Fix directory for Cygwin autoconf in FreeType.

BLD: Fix autoreconf command line.

BLD: Run only autoconf on Cygwin, not autoreconf.

libtoolize still fails; let's see if this at least gets through configure now.

BLD: Run verbose libtoolize on bundled FreeType

Let's see if this produces useful information.

BLD: Copy the libtool files and ensure write permissions

Copy instead of symlinking, and make sure the current user has write permissions for the target files.

BLD: Add print statements to update on Autotools progress

Let's see if I this narrows down where the error happens.

BLD: Add permissions for everybody to fix libtoolize

It's the only thing I can think of that would make the copy fail.

CI: Try to clean up error reporting.

Errors importing matplotlib are distinct from test failures; pip's logs of compile failures have a lot of extra stuff not related to the actual compile failure

CI: Skip broken FreeType build on Cygwin

I'll have to update many tests, but that should be doable.

CI: Add FreeType and QHull to Cygwin CI

I'm going the hard way through the testing process.

BLD: Use auto-typed result from dlsym

Try to avoid the Cygwin-vs-Linux dlsym signature incompatibility.
(Cygwin returns FARPROC aka `long long int (*)()`, Linux returns void *)

CI: Add Cygwin dependencies and try to rebase more

Let's hope this sidesteps the BlockingIOErrors in subprocess.run

CI: Split Cygwin rebase step

One step for finding the files.
a separate step for rebasing, using only rebase.

CI: Only rebase files in /usr/ and /usr/local/

GHA gives permissions to write to /usr/local, so pip puts the new files there.

CI: Add premissions to the image comparisons

See if this lets the artifact upload step work.

CI: Use the bundled FreeType on Cygwin again

CI: Try to change rebase to avoid fork failures

BLD: Fix definition of FT_CONFIG_CONFIG_H on Cygwin

The default definition doesn't actually define everything assumed to be defined by that header.
It was changed a few versions before, but they seem to have only checked whether it worked a few places.

BLD: Add autoconf and libtoolize back in

I'd forgotten configure crashed.

BLD: Continue on libtoolize failure

CI: Try to get more information from make

CI: Uses Cygwin FreeType package again.

CI: Try doubled backslashes for result directory.

Let's see if this allows upload-artifact to succeed.

BUG: Rebase DLLs in ediable install

I skipped those earlier, which doesn't do anything useful.

CI, BUG: Install git on Cygwin runner.

Some of the tests use setuptools_scm, which needs git.

CI, BUG: Run actions/checkout before Cygwin install

Otherwise git panics about someone else owning the repository midway through cloning.

CI, BUG: Mark the repo safe again

Git is apparently picky about how I specify Windows paths.

CI, DBG: Print Windows permissions for image dir

Inspired by
https://social.technet.microsoft.com/wiki/contents/articles/31139.powershell-how-to-get-folder-permissions.aspx

Apparently Windows "dir" doesn't have an option to output permissions, that's a separate utility.

CI, DBG: Print more permissions for result_images

Try printing permissions of the images that fail to upload.  The directory seems to deny only write and delete permissions, which shouldn't cause problems.

CI, DBG: Give everyone all permissions to result img dir

Maybe this will let upload-artifact succeed.

CI: Use dash for /bin/sh on Cygwin

This seems to help libtoolize finish without crashing.

CI: Try to compile bundled freetype on Cygwin again

This should get farther.

CI: FIX: Specify proper shell when setting shell.

CI, FIX: Spell shell options correctly to set /bin/sh

CI, FIX: Use dash to set dash as /bin/sh on Cygwin

BLD: Specify /bin/dash explicitly for Cygwin FreeType build

TODO: Remember to revert the configure change.

CI: Keep bash around when making dash /bin/sh

CI: Make /bin/sh a symlink to /bin/dash

CI: Set Cygwin tempdir to /tmp

CI: Explicitly install m4 on Cygwin.

CI, DBG: Clarify what /bin/sh is around def change.

I want to see how they change before and after I change it from /bin/bash to /bin/dash

CI, FIX: Use bash to change /bin/sh to /bin/dash

CI, FIX: Clean up syntax to set sh to dash.

Also avoid duplicate work for a failure.

BLD: Reduce verbosity of FreeType setup on Cygwin

Use autoreconf instead of autoconf --verbose

Should help me figure out where the new failure is.

BLD: Stop running autoreconf on Cygwin

It doesn't seem to help much

CI, DBG: Only run upload-artifact debuggers if tests ran

Don't try to examine images if there hasn't been anything to create them.

CI, FIX: Use single quotes for condition.

Double quotes cause problems

CI: Allow Cygwin CI to run on patch-2

BLD: Revert most changes to setupext.py

Still unconditionally rebuild ./configure on Cygwin, to try to update build system.

BLD: Revert the rest of the changes to setupext.py

FreeType seems to build on unmodified matplotlib locally, so this shouldn't be needed.

CI: Install matplotlib in Cygwin CI

Hopefully this pulls in the last dependencies and keeps  the FreeType build from failing.

CI: Specify MAKEFLAGS, not MAKE

https://www.gnu.org/software/make/manual/make.html#Options_002fRecursion
suggests specifying MAKEFLAGS to ensue options are passed to make, not MAKE.  This might help with the weird "directory not found" errors.

CI, BUG: Specify --coverage in CPPFLAGS and LDFLAGS

Specifying in CPPFLAGS only causes link to fail.
LDFLAGS=-lgcov should also work, but this should be more general.

STY: Remove trainling whitespace

Get pre-commit to stop sending me emails about trailing whitespace.

CI,BUG: Make sure coverage library gets linked

There may be a problem in the configtest procedure.

CI: Cut separate build step

It's failing in the pip install step, and I want to be sure that's not because there's already a build tree.

CI: Cygwin: Don't add coverage tracing to matplotlib

It causes pip install to fail.

CI: Install importlib_resources

Apparently not listed in the requirements*.txt

BUG: Fix spelling

CI: Cygwin: Drop LaTeX packages

The PGF tests mostly fail.

CI: Set ffmpeg path so Cygwin tests don't use Windows executable

CI: Append to mplrc, don't overwrite.

Let's see if this fixes the key lookup error on import

CI: Cygwin: Create new mplrc file instead of overwriting old

Appending creates load errors in forked tests; replacing the line should fix that, and creating a new version should fix the remaining problems.

CI,FIX: Cygwin: Create mpl config directories

CI: Cygwin: Don't install WX, do install ipython

WX imports keep failing.

One test uses IPython and fails, let's make sure that's not because IPython isn't installed.

CI: Cygwin: Uninstall wxPython

It shouldn't be installed, so this should be fast.

CI: Cygwin: Try to get remaining tests working.

Rebase executables in /bin as well as those in /usr/bin
Remove default ACLs on /tmp

CI,FIX: Fix syntax and style of previous commit

//bin is not POSIX-compliant and doesn't work on Cygwin.
Delete trailing whitespace.

CI: Cygwin: Set username to root and uid to 0

This should let checks for "can I access all files regardless of permissions" work the way Linux expects.

CI,FIX: Cygwin: Try to fix sed expression for euid=0

Sed expression #2, character 0: error: no previous regex.
Not really useful, sed.

CI: Cygwin: Stop changing username, just change uid

CI: Stop rebasing binaries in /bin

They are already covered in /usr/bin, and probably the important information is in the file, not a database somewhere (I mean, it's also in a database, but you can turn that off).

CI: Cygwin: Upload code coverage data
  • Loading branch information
DWesl committed Dec 10, 2022
1 parent 48e5934 commit 2f708ef
Showing 1 changed file with 223 additions and 0 deletions.
223 changes: 223 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,229 @@ jobs:
name: "${{ matrix.python-version }} ${{ matrix.os }} ${{ matrix.name-suffix }} result images"
path: ./result_images

test-cygwin:
runs-on: windows-latest
name: Python 3.${{ matrix.python-minor-version }} on Cygwin
if: "(github.repository == 'matplotlib/matplotlib' || github.repository == 'DWesl/matplotlib') && !contains(github.event.head_commit.message, '[ci skip]') && !contains(github.event.head_commit.message, '[skip ci]') && !contains(github.event.head_commit.message, '[skip github]')"
env:
SHELLOPTS: igncr
CYGWIN_NOWINPATH: 1
CHERE_INVOKING: 1
TMP: /tmp
TEMP: /tmp
strategy:
matrix:
python-minor-version: [8]

steps:
- name: Fix line endings
run: git config --global core.autocrlf input

- uses: actions/checkout@v3
with:
fetch-depth: 0

- uses: cygwin/cygwin-install-action@v2
with:
packages: >-
ccache gcc-g++ gdb git graphviz libcairo-devel libffi-devel
libgeos-devel libQt5Core-devel pkgconf libglib2.0-devel
noto-cjk-fonts
python3${{ matrix.python-minor-version }}-devel
python3${{ matrix.python-minor-version }}-pip
python3${{ matrix.python-minor-version }}-wheel
python3${{ matrix.python-minor-version }}-setuptools
python3${{ matrix.python-minor-version }}-cycler
python3${{ matrix.python-minor-version }}-dateutil
python3${{ matrix.python-minor-version }}-fonttools
python3${{ matrix.python-minor-version }}-imaging
python3${{ matrix.python-minor-version }}-kiwisolver
python3${{ matrix.python-minor-version }}-numpy
python3${{ matrix.python-minor-version }}-packaging
python3${{ matrix.python-minor-version }}-pyparsing
python3${{ matrix.python-minor-version }}-sip
python3${{ matrix.python-minor-version }}-sphinx
python-cairo-devel
python3${{ matrix.python-minor-version }}-cairo
python3${{ matrix.python-minor-version }}-gi
python3${{ matrix.python-minor-version }}-matplotlib
xorg-server-extra libxcb-icccm4 libxcb-image0
libxcb-keysyms1 libxcb-randr0 libxcb-render-util0
libxcb-xinerama0
make autoconf autoconf2.5 automake automake1.10 libtool m4
libqhull-devel libfreetype-devel
libjpeg-devel libwebp-devel
- name: Set runner username to root and id to 0
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
/bin/mkpasswd.exe -c | sed -e "s/$(id -u)/0/" >/etc/passwd
- name: Mark test repo safe
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
git.exe config --global --add safe.directory /proc/cygdrive/d/a/matplotlib/matplotlib
git config --global --add safe.directory /cygdrive/d/a/matplotlib/matplotlib
C:/cygwin/bin/git.exe config --global --add safe.directory D:/a/matplotlib/matplotlib
/usr/bin/git config --global --add safe.directory /cygdrive/d/a/matplotlib/matplotlib
- name: Use dash for /bin/sh
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
ls -l /bin/sh.exe /bin/bash.exe /bin/dash.exe
/bin/rm -f /bin/sh.exe || exit 1
cp -sf /bin/dash.exe /bin/sh.exe || exit 1
ls -l /bin/sh.exe /bin/bash.exe /bin/dash.exe
- name: Cache pip
uses: actions/cache@v3
with:
path: C:\cygwin\home\runneradmin\.cache\pip
key: Cygwin-py3.${{ matrix.python-minor-version }}-pip-${{ hashFiles('requirements/*/*.txt') }}
restore-keys: |
${{ matrix.os }}-py3.${{ matrix.python-minor-version }}-pip-
- name: Cache ccache
uses: actions/cache@v3
with:
path: C:\cygwin\home\runneradmin\.ccache
key: Cygwin-py3.${{ matrix.python-minor-version }}-ccache-${{ hashFiles('src/*') }}
restore-keys: Cygwin-py3.${{ matrix.python-minor-version }}-ccache-

- name: Cache Matplotlib
uses: actions/cache@v3
with:
path: |
C:\cygwin\home\runneradmin/.cache/matplotlib
!C:\cygwin\home\runneradmin/.cache/matplotlib/tex.cache
!C:\cygwin\home\runneradmin/.cache/matplotlib/test_cache
key: 1-Cygwin-py3.${{ matrix.python-minor-version }}-mpl-${{ github.ref }}-${{ github.sha }}
restore-keys: |
1-Cygwin-py3.${{ matrix.python-minor-version }}-mpl-${{ github.ref }}-
1-Cygwin-py3.${{ matrix.python-minor-version }}-mpl-
- name: Ensure correct python version
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
/usr/sbin/alternatives --set python /usr/bin/python3.${{ matrix.python-minor-version }}
/usr/sbin/alternatives --set python3 /usr/bin/python3.${{ matrix.python-minor-version }}
- name: Check missing files
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
ls -l /usr/include/python*/Python.h
file /usr/include/python*/Python.h
ls -l /usr/share/libtool/build-aux/ltmain.sh
file /usr/share/libtool/build-aux/ltmain.sh
- name: Install Python dependencies
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
python -m pip install --upgrade pip 'setuptools<60' wheel
python -m pip install kiwisolver 'numpy!=1.21.*' pillow importlib_resources
grep -v -F -e psutil requirements/testing/all.txt >requirements_test.txt
python -m pip install --upgrade 'contourpy>=1.0.1' cycler fonttools \
packaging pyparsing python-dateutil setuptools-scm \
-r requirements_test.txt sphinx ipython
python -m pip install --upgrade pycairo 'cairocffi>=0.8' PyGObject &&
python -c 'import gi; gi.require_version("Gtk", "3.0"); from gi.repository import Gtk' &&
echo 'PyGObject is available' ||
echo 'PyGObject is not available'
python -m pip install --upgrade pyqt5 &&
python -c 'import PyQt5.QtCore' &&
echo 'PyQt5 is available' ||
echo 'PyQt5 is not available'
python -mpip install --upgrade pyside2 &&
python -c 'import PySide2.QtCore' &&
echo 'PySide2 is available' ||
echo 'PySide2 is not available'
yes | python -m pip uninstall wxpython || echo 'wxPython already uninstalled'
- name: Install matplotlib
shell: bash.exe -eo pipefail -o igncr "{0}"
env:
AUTOCONF: /usr/bin/autoconf-2.69
MAKEFLAGS: dw
run: |
ccache -s
git describe
# Set flag in a delayed manner to avoid issues with installing other
# packages
cat <<EOT >> mplsetup.cfg
[rc_options]
backend=Agg
[libs]
system_freetype = False
system_qhull = True
EOT
cat mplsetup.cfg
# All dependencies must have been pre-installed, so that the minver
# constraints are held.
python -m pip install --no-deps -ve .
- name: Find DLLs to rebase
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
find {/usr,/usr/local}/{bin,lib/python3.?/site-packages} /usr/lib/lapack . -name \*.exe -o -name \*.dll -print >files_to_rebase.txt
- name: Rebase DLL list
shell: ash.exe "{0}"
run: "rebase --database --filelist=files_to_rebase.txt"

- name: Check that Matplotlib imports
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
/usr/bin/python -c "import matplotlib as mpl; import matplotlib.pyplot as plt"
- name: Set ffmpeg path
shell: 'bash.exe -eo pipefail -o igncr "{0}"'
run: |
oldmplrc=$(python -c "from matplotlib import matplotlib_fname as mplrc_file; print(mplrc_file())")
echo "${oldmplrc}"
awk '/animation.ffmpeg_path/ { print "animation.ffmpeg_path: /usr/bin/ffmpeg.exe"; next } { print $0; }' "${oldmplrc}" >matplotlibrc
mkdir -p ~/.config/matplotlib/
cp matplotlibrc ~/.config/matplotlib/matplotlibrc
mkdir -p ~/.matplotlib/
cp matplotlibrc ~/.matplotlib/matplotlibrc
- name: Remove default ACLs on /tmp
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
cd /tmp && setfacl -b .
- name: Run pytest
shell: bash.exe -eo pipefail -o igncr "{0}"
id: cygwin-run-pytest
run: |
xvfb-run python -mpytest -raR -n auto \
--maxfail=50 --timeout=300 --durations=25 \
--cov-report=xml --cov=lib --log-level=DEBUG --color=yes
- name: Add permissions for image comparisons
if: failure() && steps.cygwin-run-pytest.conclusion == 'failure'
shell: bash.exe -eo pipefail -o igncr "{0}"
run: |
chmod a+rwx -R ./result_images
chmod a+rwx $(find ./result_images -type d)
chmod o+rw -R ./result_images
- name: List result image directory contents
if: failure() && steps.cygwin-run-pytest.conclusion == 'failure'
run: |
dir ./result_images
Get-ACL ./result_images/* | Format-List
Get-ACL ./result_images/test_agg/* | Format-List
- uses: actions/upload-artifact@v3
if: failure()
with:
name: "3.${{ matrix.python-minor-version }} Cygwin result images"
path: ".\\result_images"

- name: Upload code coverage
uses: codecov/codecov-action@v3

# Separate dependent job to only upload one issue from the matrix of jobs
create-issue:
if: ${{ failure() && github.event_name == 'schedule' }}
Expand Down

0 comments on commit 2f708ef

Please sign in to comment.