Cygwin: Makefile does not install DLL import library

[embray]

BPO	27374
Nosy	@embray, @ma8ma, @iritkatriel
Superseder	bpo-45537: Cygwin is unsupported - close all open issues and list them here.
Files	cygwin-ldlibrary.patch 3.5-issue27374-distutils-build_ext.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-20.12:05:42.701>
created_at = <Date 2016-06-23.15:20:38.185>
labels = ['extension-modules', 'type-bug', 'expert-installation']
title = 'Cygwin: Makefile does not install DLL import library'
updated_at = <Date 2021-10-21.09:57:42.017>
user = 'https://github.com/embray'

bugs.python.org fields:

activity = <Date 2021-10-21.09:57:42.017>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.12:05:42.701>
closer = 'iritkatriel'
components = ['Extension Modules', 'Installation']
creation = <Date 2016-06-23.15:20:38.185>
creator = 'erik.bray'
dependencies = []
files = ['43519', '44685']
hgrepos = []
issue_num = 27374
keywords = ['patch']
message_count = 7.0
messages = ['269116', '276398', '276402', '276669', '277225', '277254', '404426']
nosy_count = 3.0
nosy_names = ['erik.bray', 'masamoto', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '45537'
type = 'behavior'
url = 'https://bugs.python.org/issue27374'
versions = ['Python 3.5', 'Python 3.6']

[embray]

When building Python on Cygwin, both a libpython-X.Y.dll and a
libpython-X.Y.dll.a are created (see https://cygwin.com/cygwin-ug-net/dll.html).

The latter is an "import library"
consisting of stubs for functions in the DLL so that it can be linked
to statically when building, for example, extension modules.

The odd bit is that in the altbininstall target if the
$(DLLLIBRARY) variable (which references the .dll itself) is defined then only it is installed, while $(LDLIBRARY) (which in this cases references the .dll.a import library) is *not* installed, except in $(prefix)/lib/pythonX.Y/config, which is not normally on the linker search path, or even included by
python-config --ldflags.

Therefore static linking to libpython fails, unless the search path is explicitly modified, or a symlink is created from $(prefix)/lib/pythonX.Y/config/libpython.dll.a to $(prefix)/lib. In fact the Makefile already does the latter, again not if $(DLLLIBRARY) is defined.

In fact Cygwin's own package for Python manually creates the latter symlink in its install script. But I would consider it a bug in Python's Makefile that it doesn't handle this.

I first asked on Python-dev about this back in April but got no reply (understandably) probably due to the lack of core devs who can comment on Cygwin :)

[ma8ma]

Hello, Eric.
I tried to build the '_speedups' C extension module of simplejson (third party library) using the python built on cygwin. As you pointed out, the build of '_speedups' C extension module at the stage of the link failed.

build log:
Python3.5.2+ has be used. It has applyed patches to known issues.

$ ./configure --prefix=/opt/py35 --with-threads --with-dbmliborder=gdbm --with-system-ffi --with-system-expat
$ make && make altinstall
$ LANG=C /opt/py35/bin/pip3.5 install simplejson
(snip)
running build_ext
    building 'simplejson._speedups' extension
    creating build/temp.cygwin-2.6.0-i686-3.5
    creating build/temp.cygwin-2.6.0-i686-3.5/simplejson
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/py35/include/python3.5m -c simplejson/_speedups.c -o build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o
    gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o -L/opt/py35/lib/python3.5/config -L/opt/py35/lib -lpython3.5m -o build/lib.cygwin-2.6.0-i686-3.5/simplejson/_speedups.cpython-35m.dll
    /usr/lib/gcc/i686-pc-cygwin/5.4.0/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython3.5m
    collect2: error: ld returned 1 exit status
    ***************************************************************************
    WARNING: The C extension could not be compiled, speedups are not enabled.
    Failure information, if any, is above.
    I'm retrying the build without the C extension now.
    ***************************************************************************
(snip)

In the log, -L option specified to "/opt/py35/lib/python3.5/config". But, the current python has "/opt/py35/lib/python3.5/config-3.5m" instead of "/opt/py35/lib/python3.5/config". And libpython3.5m.dll.a library exists in the config-3.5m. I think that to succeed building a C extension module if -L options is changed to "/opt/py35/lib/python3.5/config-3.5m".
I'm going to try to find the place that add -L option at build time.

[ma8ma]

I'm so sorry that I misspelled your name, Erik.

[ma8ma]

I found -L option addition at Lib/distutils/command/build_ext.py:222. This place is a process that finalize options for the build_ext command.
Comment in there: for extensions under Cygwin and AtheOS Python's library directory must be appended to library_dirs

After that I searched Makefile rule for installing $(LDLIBRARY) into "$(prefix)/lib/python3.5/config-3.5m" to look for way of simplifying code. And I came across the $(LIBPL) configuration variable that is used by installation of the configuration for building library.
Hence I made a patch that adds to library_dirs the $(LIBPL) configuration variable instead of the "$(prefix)/lib/python$(VERSION)/config". I succeeded to build '_speedups' C extension module of simplejson using new python that is applied this patch.
Erik, Would you be able to verify this patch?
Thanks

Build log:
$ patch -p1 <3.5-issue27374-distutils-build_ext.patch
$ ./configure --prefix=/opt/py35 --with-threads --with-dbmliborder=gdbm --with-system-ffi --with-system-expat
$ make touch && make -j2 && make altinstall
$ /opt/py35/bin/python3.5 -V
Python 3.5.2+
$ LANG=C /opt/py35/bin/pip3.5 install -v simplejson
(snip)
running build_ext
building 'simplejson._speedups' extension
creating build/temp.cygwin-2.6.0-i686-3.5
creating build/temp.cygwin-2.6.0-i686-3.5/simplejson
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/py35/include/python3.5m -c simplejson/_speedups.c -o build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o
gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o -L/opt/py35/lib/python3.5/config-3.5m -L/opt/py35/lib -lpython3.5m -o build/lib.cygwin-2.6.0-i686-3.5/simplejson/_speedups.cpython-35m.dll
running install_lib
(snip)

[embray]

Masayuki--concerning your above comments, I think this is similar to, if not the same as bpo-13756

[ma8ma]

I don't know why Cygwin packaging script (cygport) doesn't copy direct libpythonX.Ym.dll.a to /usr/lib. Hence I wrote the patch that doesn't conflict with cygport script.

In current makefile rule for Cygwin, Install $(LDLIBRARY) into $(LIBPL). Therefore I prefer symlink from $(LIBPL) than copying direct library file to $(LIBDIR).
What do you think of makefile rule addition that is same effect (symlink from LIBPL) as cygport script?

[iritkatriel]

See also bpo-19241.