JPEG not detected when installing Pillow under Cygwin64 #6216
Labels
Comments
radarhere
changed the title
|
@DWesl do you have any thoughts on this one? |
|
I know the Pillow tests require jpeg command-line utilities in addition to the jpeg and jpeg-2000 libraries. From #5878, the relevant Cygwin packages seem to be If installing the packages mentioned before doesn't fix the issue, it would help to have a list of installed Cygwin packages, with |
|
Thank you for your responses! The list of installed packages is attached. |
|
You may want Are you installing using |
|
Yes, everything is installed using And: as mentioned above, adding a static libjpeg. a to /usr/local allows installation to (sort of) work. |
|
Well that's weird. What's your |
|
I can reproduce this on 32-bit Cygwin. This wasn't an issue a few months ago, so you should be able to get pillow from a version or three back. I haven't figured out where things are breaking yet. |
|
Try installing |
|
BINGO! Installation of Pillow worked under Cygwin64 worked some time ago. For Python 3.8, I have a dist-info directory with files with modification date Oct 24, 2021. |
|
Cygwin setuptools is still at version 59.5.0, so install should work on a new Cygwin64 installation. Cygwin isn't likely to upgrade until NumPy and setuptools>60 regain compatibility Do you want to report this to setuptools, or should I? |
|
It will probably have a greater impact if you do... |
|
Thanks very much for figuring this one out. Closing. pypa/setuptools#3302 can now be followed instead. |
This was referenced May 4, 2022
DWesl
added a commit
to DWesl/matplotlib
that referenced
this issue
May 7, 2022
setuptools>=60 runs into python-pillow/Pillow#6216 and pypa/setuptools#3304. NumPy==1.21.* has intermittent segfaults due to SIMD attempts on unaligned memory. kiwisolver, numpy, and pillow have problems installing, so just verify that they are present, don't try to upgrade them.
6 tasks
|
pypa/distutils#139 has merged a fix into upstream of setuptools, so that fix should be propagating into setuptools in time. For anyone who needs a fix before then and cannot downgrade to |
|
Thank you for helping with this! |
This was referenced Nov 25, 2022
DWesl
added a commit
to DWesl/matplotlib
that referenced
this issue
Dec 10, 2022
This involved a good bit of debugging of the build process, especially for the bundled FreeType. Eventually removing the debugging flags let it just work. There are still fork failures, but I've done everything I know to do. CYGWIN=detect_bloda may reveal more, but may not. parent 48e5934 author DWesl <22566757+DWesl@users.noreply.github.com> 1651882104 -0400 committer DWesl <22566757+DWesl@users.noreply.github.com> 1670678538 -0500 parent 48e5934 author DWesl <22566757+DWesl@users.noreply.github.com> 1651882104 -0400 committer DWesl <22566757+DWesl@users.noreply.github.com> 1670678484 -0500 CI: Add a Cygwin run to GHA CI. Let's see if I translated the workflow even close to correctly. CI: Pin Cygwin setuptools<60 and NumPy !=1.21 setuptools>60 breaks Pillow install, and may cause problems for NumPy. NumPy 1.21 has intermittent segfaults on GHA runners, so let's just skip that. CI: Pin Cygwin setuptools<60 and avoid problem upgrades setuptools>=60 runs into python-pillow/Pillow#6216 and pypa/setuptools#3304. NumPy==1.21.* has intermittent segfaults due to SIMD attempts on unaligned memory. kiwisolver, numpy, and pillow have problems installing, so just verify that they are present, don't try to upgrade them. CI: Upgrade Cygwin pip CI: Install g++ and print info on Python.h Hopefully one of these will convince NumPy to compile. CI: Install libjpeg-devel for Pillow install Let's see if it keeps setuptools old enough for this to work. CI: BLD: Use system freetype and qhull Given that I'm checking a build problem at the moment, this should allow that to show up faster. This should probably be reverted before merging. CI: Specify shell for Cygwin pytest step. CI: Rebase Matplotlib DLLs before running pytest Try to avoid "BlockingIOError: Resource Unavailable" with a fork() error on stderr. CI: Compile against bundled freetype. See if this fixes the various test errors. It will take a few tries to get freetype to actually compile, I think. CI: Install make for libfreetype build Let's see if this lets the build run to completion. BLD: Run autoconf before installing bundled freetype Let's see if this fixes the install problems. CI: Try to get Cygwin build of FreeType working. It doesn't fail the same way locally. CI: Revert to actually-configured python alternatives. BLD: Fix autoconf call on Cygwin. BLD: Fix directory for Cygwin autoreconf. CI: Fix directory for Cygwin autoconf in FreeType. BLD: Fix autoreconf command line. BLD: Run only autoconf on Cygwin, not autoreconf. libtoolize still fails; let's see if this at least gets through configure now. BLD: Run verbose libtoolize on bundled FreeType Let's see if this produces useful information. BLD: Copy the libtool files and ensure write permissions Copy instead of symlinking, and make sure the current user has write permissions for the target files. BLD: Add print statements to update on Autotools progress Let's see if I this narrows down where the error happens. BLD: Add permissions for everybody to fix libtoolize It's the only thing I can think of that would make the copy fail. CI: Try to clean up error reporting. Errors importing matplotlib are distinct from test failures; pip's logs of compile failures have a lot of extra stuff not related to the actual compile failure CI: Skip broken FreeType build on Cygwin I'll have to update many tests, but that should be doable. CI: Add FreeType and QHull to Cygwin CI I'm going the hard way through the testing process. BLD: Use auto-typed result from dlsym Try to avoid the Cygwin-vs-Linux dlsym signature incompatibility. (Cygwin returns FARPROC aka `long long int (*)()`, Linux returns void *) CI: Add Cygwin dependencies and try to rebase more Let's hope this sidesteps the BlockingIOErrors in subprocess.run CI: Split Cygwin rebase step One step for finding the files. a separate step for rebasing, using only rebase. CI: Only rebase files in /usr/ and /usr/local/ GHA gives permissions to write to /usr/local, so pip puts the new files there. CI: Add premissions to the image comparisons See if this lets the artifact upload step work. CI: Use the bundled FreeType on Cygwin again CI: Try to change rebase to avoid fork failures BLD: Fix definition of FT_CONFIG_CONFIG_H on Cygwin The default definition doesn't actually define everything assumed to be defined by that header. It was changed a few versions before, but they seem to have only checked whether it worked a few places. BLD: Add autoconf and libtoolize back in I'd forgotten configure crashed. BLD: Continue on libtoolize failure CI: Try to get more information from make CI: Uses Cygwin FreeType package again. CI: Try doubled backslashes for result directory. Let's see if this allows upload-artifact to succeed. BUG: Rebase DLLs in ediable install I skipped those earlier, which doesn't do anything useful. CI, BUG: Install git on Cygwin runner. Some of the tests use setuptools_scm, which needs git. CI, BUG: Run actions/checkout before Cygwin install Otherwise git panics about someone else owning the repository midway through cloning. CI, BUG: Mark the repo safe again Git is apparently picky about how I specify Windows paths. CI, DBG: Print Windows permissions for image dir Inspired by https://social.technet.microsoft.com/wiki/contents/articles/31139.powershell-how-to-get-folder-permissions.aspx Apparently Windows "dir" doesn't have an option to output permissions, that's a separate utility. CI, DBG: Print more permissions for result_images Try printing permissions of the images that fail to upload. The directory seems to deny only write and delete permissions, which shouldn't cause problems. CI, DBG: Give everyone all permissions to result img dir Maybe this will let upload-artifact succeed. CI: Use dash for /bin/sh on Cygwin This seems to help libtoolize finish without crashing. CI: Try to compile bundled freetype on Cygwin again This should get farther. CI: FIX: Specify proper shell when setting shell. CI, FIX: Spell shell options correctly to set /bin/sh CI, FIX: Use dash to set dash as /bin/sh on Cygwin BLD: Specify /bin/dash explicitly for Cygwin FreeType build TODO: Remember to revert the configure change. CI: Keep bash around when making dash /bin/sh CI: Make /bin/sh a symlink to /bin/dash CI: Set Cygwin tempdir to /tmp CI: Explicitly install m4 on Cygwin. CI, DBG: Clarify what /bin/sh is around def change. I want to see how they change before and after I change it from /bin/bash to /bin/dash CI, FIX: Use bash to change /bin/sh to /bin/dash CI, FIX: Clean up syntax to set sh to dash. Also avoid duplicate work for a failure. BLD: Reduce verbosity of FreeType setup on Cygwin Use autoreconf instead of autoconf --verbose Should help me figure out where the new failure is. BLD: Stop running autoreconf on Cygwin It doesn't seem to help much CI, DBG: Only run upload-artifact debuggers if tests ran Don't try to examine images if there hasn't been anything to create them. CI, FIX: Use single quotes for condition. Double quotes cause problems CI: Allow Cygwin CI to run on patch-2 BLD: Revert most changes to setupext.py Still unconditionally rebuild ./configure on Cygwin, to try to update build system. BLD: Revert the rest of the changes to setupext.py FreeType seems to build on unmodified matplotlib locally, so this shouldn't be needed. CI: Install matplotlib in Cygwin CI Hopefully this pulls in the last dependencies and keeps the FreeType build from failing. CI: Specify MAKEFLAGS, not MAKE https://www.gnu.org/software/make/manual/make.html#Options_002fRecursion suggests specifying MAKEFLAGS to ensue options are passed to make, not MAKE. This might help with the weird "directory not found" errors. CI, BUG: Specify --coverage in CPPFLAGS and LDFLAGS Specifying in CPPFLAGS only causes link to fail. LDFLAGS=-lgcov should also work, but this should be more general. STY: Remove trainling whitespace Get pre-commit to stop sending me emails about trailing whitespace. CI,BUG: Make sure coverage library gets linked There may be a problem in the configtest procedure. CI: Cut separate build step It's failing in the pip install step, and I want to be sure that's not because there's already a build tree. CI: Cygwin: Don't add coverage tracing to matplotlib It causes pip install to fail. CI: Install importlib_resources Apparently not listed in the requirements*.txt BUG: Fix spelling CI: Cygwin: Drop LaTeX packages The PGF tests mostly fail. CI: Set ffmpeg path so Cygwin tests don't use Windows executable CI: Append to mplrc, don't overwrite. Let's see if this fixes the key lookup error on import CI: Cygwin: Create new mplrc file instead of overwriting old Appending creates load errors in forked tests; replacing the line should fix that, and creating a new version should fix the remaining problems. CI,FIX: Cygwin: Create mpl config directories CI: Cygwin: Don't install WX, do install ipython WX imports keep failing. One test uses IPython and fails, let's make sure that's not because IPython isn't installed. CI: Cygwin: Uninstall wxPython It shouldn't be installed, so this should be fast. CI: Cygwin: Try to get remaining tests working. Rebase executables in /bin as well as those in /usr/bin Remove default ACLs on /tmp CI,FIX: Fix syntax and style of previous commit //bin is not POSIX-compliant and doesn't work on Cygwin. Delete trailing whitespace. CI: Cygwin: Set username to root and uid to 0 This should let checks for "can I access all files regardless of permissions" work the way Linux expects. CI,FIX: Cygwin: Try to fix sed expression for euid=0 Sed expression #2, character 0: error: no previous regex. Not really useful, sed. CI: Cygwin: Stop changing username, just change uid CI: Stop rebasing binaries in /bin They are already covered in /usr/bin, and probably the important information is in the file, not a database somewhere (I mean, it's also in a database, but you can turn that off). CI: Cygwin: Upload code coverage data
DWesl
added a commit
to DWesl/matplotlib
that referenced
this issue
Jan 4, 2023
This involved a good bit of debugging of the build process, especially for the bundled FreeType. Eventually removing the debugging flags let it just work. There are still fork failures, but I've done everything I know to do. CYGWIN=detect_bloda may reveal more, but may not. parent 48e5934 author DWesl <22566757+DWesl@users.noreply.github.com> 1651882104 -0400 committer DWesl <22566757+DWesl@users.noreply.github.com> 1670678538 -0500 parent 48e5934 author DWesl <22566757+DWesl@users.noreply.github.com> 1651882104 -0400 committer DWesl <22566757+DWesl@users.noreply.github.com> 1670678484 -0500 CI: Add a Cygwin run to GHA CI. Let's see if I translated the workflow even close to correctly. CI: Pin Cygwin setuptools<60 and NumPy !=1.21 setuptools>60 breaks Pillow install, and may cause problems for NumPy. NumPy 1.21 has intermittent segfaults on GHA runners, so let's just skip that. CI: Pin Cygwin setuptools<60 and avoid problem upgrades setuptools>=60 runs into python-pillow/Pillow#6216 and pypa/setuptools#3304. NumPy==1.21.* has intermittent segfaults due to SIMD attempts on unaligned memory. kiwisolver, numpy, and pillow have problems installing, so just verify that they are present, don't try to upgrade them. CI: Upgrade Cygwin pip CI: Install g++ and print info on Python.h Hopefully one of these will convince NumPy to compile. CI: Install libjpeg-devel for Pillow install Let's see if it keeps setuptools old enough for this to work. CI: BLD: Use system freetype and qhull Given that I'm checking a build problem at the moment, this should allow that to show up faster. This should probably be reverted before merging. CI: Specify shell for Cygwin pytest step. CI: Rebase Matplotlib DLLs before running pytest Try to avoid "BlockingIOError: Resource Unavailable" with a fork() error on stderr. CI: Compile against bundled freetype. See if this fixes the various test errors. It will take a few tries to get freetype to actually compile, I think. CI: Install make for libfreetype build Let's see if this lets the build run to completion. BLD: Run autoconf before installing bundled freetype Let's see if this fixes the install problems. CI: Try to get Cygwin build of FreeType working. It doesn't fail the same way locally. CI: Revert to actually-configured python alternatives. BLD: Fix autoconf call on Cygwin. BLD: Fix directory for Cygwin autoreconf. CI: Fix directory for Cygwin autoconf in FreeType. BLD: Fix autoreconf command line. BLD: Run only autoconf on Cygwin, not autoreconf. libtoolize still fails; let's see if this at least gets through configure now. BLD: Run verbose libtoolize on bundled FreeType Let's see if this produces useful information. BLD: Copy the libtool files and ensure write permissions Copy instead of symlinking, and make sure the current user has write permissions for the target files. BLD: Add print statements to update on Autotools progress Let's see if I this narrows down where the error happens. BLD: Add permissions for everybody to fix libtoolize It's the only thing I can think of that would make the copy fail. CI: Try to clean up error reporting. Errors importing matplotlib are distinct from test failures; pip's logs of compile failures have a lot of extra stuff not related to the actual compile failure CI: Skip broken FreeType build on Cygwin I'll have to update many tests, but that should be doable. CI: Add FreeType and QHull to Cygwin CI I'm going the hard way through the testing process. BLD: Use auto-typed result from dlsym Try to avoid the Cygwin-vs-Linux dlsym signature incompatibility. (Cygwin returns FARPROC aka `long long int (*)()`, Linux returns void *) CI: Add Cygwin dependencies and try to rebase more Let's hope this sidesteps the BlockingIOErrors in subprocess.run CI: Split Cygwin rebase step One step for finding the files. a separate step for rebasing, using only rebase. CI: Only rebase files in /usr/ and /usr/local/ GHA gives permissions to write to /usr/local, so pip puts the new files there. CI: Add premissions to the image comparisons See if this lets the artifact upload step work. CI: Use the bundled FreeType on Cygwin again CI: Try to change rebase to avoid fork failures BLD: Fix definition of FT_CONFIG_CONFIG_H on Cygwin The default definition doesn't actually define everything assumed to be defined by that header. It was changed a few versions before, but they seem to have only checked whether it worked a few places. BLD: Add autoconf and libtoolize back in I'd forgotten configure crashed. BLD: Continue on libtoolize failure CI: Try to get more information from make CI: Uses Cygwin FreeType package again. CI: Try doubled backslashes for result directory. Let's see if this allows upload-artifact to succeed. BUG: Rebase DLLs in ediable install I skipped those earlier, which doesn't do anything useful. CI, BUG: Install git on Cygwin runner. Some of the tests use setuptools_scm, which needs git. CI, BUG: Run actions/checkout before Cygwin install Otherwise git panics about someone else owning the repository midway through cloning. CI, BUG: Mark the repo safe again Git is apparently picky about how I specify Windows paths. CI, DBG: Print Windows permissions for image dir Inspired by https://social.technet.microsoft.com/wiki/contents/articles/31139.powershell-how-to-get-folder-permissions.aspx Apparently Windows "dir" doesn't have an option to output permissions, that's a separate utility. CI, DBG: Print more permissions for result_images Try printing permissions of the images that fail to upload. The directory seems to deny only write and delete permissions, which shouldn't cause problems. CI, DBG: Give everyone all permissions to result img dir Maybe this will let upload-artifact succeed. CI: Use dash for /bin/sh on Cygwin This seems to help libtoolize finish without crashing. CI: Try to compile bundled freetype on Cygwin again This should get farther. CI: FIX: Specify proper shell when setting shell. CI, FIX: Spell shell options correctly to set /bin/sh CI, FIX: Use dash to set dash as /bin/sh on Cygwin BLD: Specify /bin/dash explicitly for Cygwin FreeType build TODO: Remember to revert the configure change. CI: Keep bash around when making dash /bin/sh CI: Make /bin/sh a symlink to /bin/dash CI: Set Cygwin tempdir to /tmp CI: Explicitly install m4 on Cygwin. CI, DBG: Clarify what /bin/sh is around def change. I want to see how they change before and after I change it from /bin/bash to /bin/dash CI, FIX: Use bash to change /bin/sh to /bin/dash CI, FIX: Clean up syntax to set sh to dash. Also avoid duplicate work for a failure. BLD: Reduce verbosity of FreeType setup on Cygwin Use autoreconf instead of autoconf --verbose Should help me figure out where the new failure is. BLD: Stop running autoreconf on Cygwin It doesn't seem to help much CI, DBG: Only run upload-artifact debuggers if tests ran Don't try to examine images if there hasn't been anything to create them. CI, FIX: Use single quotes for condition. Double quotes cause problems CI: Allow Cygwin CI to run on patch-2 BLD: Revert most changes to setupext.py Still unconditionally rebuild ./configure on Cygwin, to try to update build system. BLD: Revert the rest of the changes to setupext.py FreeType seems to build on unmodified matplotlib locally, so this shouldn't be needed. CI: Install matplotlib in Cygwin CI Hopefully this pulls in the last dependencies and keeps the FreeType build from failing. CI: Specify MAKEFLAGS, not MAKE https://www.gnu.org/software/make/manual/make.html#Options_002fRecursion suggests specifying MAKEFLAGS to ensue options are passed to make, not MAKE. This might help with the weird "directory not found" errors. CI, BUG: Specify --coverage in CPPFLAGS and LDFLAGS Specifying in CPPFLAGS only causes link to fail. LDFLAGS=-lgcov should also work, but this should be more general. STY: Remove trainling whitespace Get pre-commit to stop sending me emails about trailing whitespace. CI,BUG: Make sure coverage library gets linked There may be a problem in the configtest procedure. CI: Cut separate build step It's failing in the pip install step, and I want to be sure that's not because there's already a build tree. CI: Cygwin: Don't add coverage tracing to matplotlib It causes pip install to fail. CI: Install importlib_resources Apparently not listed in the requirements*.txt BUG: Fix spelling CI: Cygwin: Drop LaTeX packages The PGF tests mostly fail. CI: Set ffmpeg path so Cygwin tests don't use Windows executable CI: Append to mplrc, don't overwrite. Let's see if this fixes the key lookup error on import CI: Cygwin: Create new mplrc file instead of overwriting old Appending creates load errors in forked tests; replacing the line should fix that, and creating a new version should fix the remaining problems. CI,FIX: Cygwin: Create mpl config directories CI: Cygwin: Don't install WX, do install ipython WX imports keep failing. One test uses IPython and fails, let's make sure that's not because IPython isn't installed. CI: Cygwin: Uninstall wxPython It shouldn't be installed, so this should be fast. CI: Cygwin: Try to get remaining tests working. Rebase executables in /bin as well as those in /usr/bin Remove default ACLs on /tmp CI,FIX: Fix syntax and style of previous commit //bin is not POSIX-compliant and doesn't work on Cygwin. Delete trailing whitespace. CI: Cygwin: Set username to root and uid to 0 This should let checks for "can I access all files regardless of permissions" work the way Linux expects. CI,FIX: Cygwin: Try to fix sed expression for euid=0 Sed expression #2, character 0: error: no previous regex. Not really useful, sed. CI: Cygwin: Stop changing username, just change uid CI: Stop rebasing binaries in /bin They are already covered in /usr/bin, and probably the important information is in the file, not a database somewhere (I mean, it's also in a database, but you can turn that off). CI: Cygwin: Upload code coverage data CI: Cygwin: Update run condition for upstream repo I changed it so I'd get quicker feedback on tests some time ago and forgot to change it back. Update .github/workflows/tests.yml Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com> Update .github/workflows/tests.yml Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com> CI: Cygwin: Incorporate suggestions from review. CI: Incorporate suggestions from review. CI: Cygwin: Use dash for /bin/sh instead of bash. Faster, and doesn't fail the FreeType build. CI: Test python 3.8 and 3.9 on Cygwin Co-authored-by: Thomas A Caswell <tcaswell@gmail.com> CI: Cygwin: Correct matplotlibrc location Apparently it doesn't check ~/.config/matplotlib DOC,CI: Explain why CI runs tests with gid set to 0. The other option is to look into cygutils-extra's cygdrop. I have no idea how to use that, so I'm sticking with this.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
Attempting to install the current version of Pillow (9.1.0) fails with Python-3.9.10 under Cygwin64. I attempted installing both via pip3 and via python3 setup.py build. Both fail the same way.
Installation succeeds until the configuration stops with a message that the required headers and/or library files for jpeg were not found.
Jpeg packages (both jpeg and openjpeg) are installed including the developer packages. Both header files and library files are in fact present in the right directories.
I had a hunch that this might be a library file naming issue. In /usr/lib the following jpeg-related files are present:
/usr/lib/libjpeg.dll.a
/usr/lib/libopenjpeg.dll.a
To test, I built and installed a static libjpeg.a and put libjpeg.a file in /usr/local/lib. That done, the build succeeded to the point that the extension was built. However, most other supporting libraries were not detected despite that most of them were in fact installed (including developer packages); the corresponding packages all exclusively install shared libraries (dll's), not static:
The output from the build produced:
There is in fact a zlib static library in /usr/lib and, thus, presence of zlib was detected.
So either:
Under Cygwin the dll files are located in /usr/bin with names such as cygtiff-6.dll, cygjpeg-8.dll, cygopenjpeg-1.dll, cygimagequant-0.dll etc. The link files have names such as libjpeg.dll.a etc.
The text was updated successfully, but these errors were encountered: