curl-config --libs returns incorrect value on cygwin

[ramsay-jones]

I did this

Last night I updated my cygwin installation and (among others) my curl packages were updated.
This lead to my git build failing while linking git-imap-send.exe as the linker could not find the
libraries: -lnghttp2, -lidn2, -lssh, -lpsl, -lldap, -llber, -lbrotlidec, -lbrotlidec (yes it was noted twice).

$ cygcheck -cd | grep curl
curl 7.71.1-1
libcurl-devel 7.71.1-1
libcurl-doc 7.71.1-1
libcurl4 7.71.1-1
$

The git makefile uses 'curl-config --libs', which produces:

$ curl-config --libs
-lcurl -lnghttp2 -lidn2 -lssh -lpsl -lssl -lcrypto -lldap -llber -lbrotlidec -lbrotlidec -lz
$

Looking at the '/usr/lib/pkgconfig/libcurl.pc' file, I find:

$ cat /usr/lib/pkgconfig/libcurl.pc
...
Libs: -L${libdir} -lcurl
Libs.private: -lnghttp2 -lidn2 -lssh -lpsl -lssl -lcrypto -lldap -llber -lbrotlidec -lbrotlidec -lz
...
$

I had a quick look at the curl git repo, and this seems to be due to a recent commit 98e5904
(libcurl.pc: Merge Libs.private into Libs for static-only builds, 2020-05-11).

The git Makefile allows me to override the value returned by curl-config, by setting 'CURL_LDFLAGS=-lcurl'
in my config.mak, so I have simple solution for now - just thought you should know. ;-)

I expected the following

git to build correctly!

curl/libcurl version

$ curl -V
curl 7.71.1 (x86_64-pc-cygwin) libcurl/7.71.1 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0
Release-Date: 2020-07-01
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli Debug HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB PSL SSL TLS-SRP TrackMemory UnixSockets
$

operating system

$ uname -a
CYGWIN_NT-10.0 satellite 3.1.6(0.340/5/3) 2020-07-09 08:20 x86_64 Cygwin
$

[bagder]

Did you build libcurl static only?

[ramsay-jones]

On 07/08/2020 09:08, Daniel Stenberg wrote: Did you build libcurl static only?

Heh, I didn't build libcurl at all. I am simply a user of the libcurl and libcurl-devel packages on cygwin. The '/var/log/setup.log' file shows that the cygwin update removed the 'old' 7.66.0-1 packages and installed the 'new' 7.71.1-1 packages. (and _didn't_ install any dependent libraries). $ cygcheck -cd | grep curl curl 7.71.1-1 curl-debuginfo 7.71.1-1 libcurl-devel 7.71.1-1 libcurl-doc 7.71.1-1 libcurl4 7.71.1-1 $ [note: I added the curl-debuginfo package today] The libcurl4 package provides (just) the libcurl dll file, whereas the libcurl-devel package provides, among other things, the 'import library' used by the linker (ie. '/usr/lib/libcurl.dll.a'): $ cygcheck -l libcurl4 /usr/bin/cygcurl-4.dll $ cygcheck -l libcurl-devel /usr/bin/curl-config /usr/include/curl/curl.h /usr/include/curl/curlver.h /usr/include/curl/easy.h /usr/include/curl/mprintf.h /usr/include/curl/multi.h /usr/include/curl/stdcheaders.h /usr/include/curl/system.h /usr/include/curl/typecheck-gcc.h /usr/include/curl/urlapi.h /usr/lib/libcurl.dll.a /usr/lib/pkgconfig/libcurl.pc /usr/share/aclocal/libcurl.m4 /usr/share/man/man1/curl-config.1.gz $ [note: I am assuming that the 7.66.0-1 version of curl-config just returned '-lcurl' when invoking 'curl-config --libs'] As I said, I can build git without issue by setting 'CURL_LDFLAGS=-lcurl' in the config.mak file to override the 'curl-config --libs' output, so we can see: $ ldd git-imap-send.exe ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffa7db70000) KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffa7d18000 0) KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffa7b4 70000) cygwin1.dll => /usr/bin/cygwin1.dll (0x180040000) cygcrypto-1.1.dll => /usr/bin/cygcrypto-1.1.dll (0x41c650000) cygcurl-4.dll => /usr/bin/cygcurl-4.dll (0x5d28e0000) cygiconv-2.dll => /usr/bin/cygiconv-2.dll (0x3f6900000) cygintl-8.dll => /usr/bin/cygintl-8.dll (0x5ee2d0000) cygssl-1.1.dll => /usr/bin/cygssl-1.1.dll (0x5b4450000) cygz.dll => /usr/bin/cygz.dll (0x597fd0000) cygbrotlidec-1.dll => /usr/bin/cygbrotlidec-1.dll (0x5deea0000) cygidn2-0.dll => /usr/bin/cygidn2-0.dll (0x5b8850000) cyglber-2-4-2.dll => /usr/bin/cyglber-2-4-2.dll (0x3d9860000) cygnghttp2-14.dll => /usr/bin/cygnghttp2-14.dll (0x5b76c0000) cygpsl-5.dll => /usr/bin/cygpsl-5.dll (0x5b51c0000) cygssh-4.dll => /usr/bin/cygssh-4.dll (0x5b4580000) cygldap-2-4-2.dll => /usr/bin/cygldap-2-4-2.dll (0x5b7fa0000) cygbrotlicommon-1.dll => /usr/bin/cygbrotlicommon-1.dll (0x5deec0000) cygunistring-2.dll => /usr/bin/cygunistring-2.dll (0xcd0000) cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x50caa0000) cygunistring-2.dll => /usr/bin/cygunistring-2.dll (0x5b3b20000) cyggssapi_krb5-2.dll => /usr/bin/cyggssapi_krb5-2.dll (0x5f6cf0000) cygsasl2-3.dll => /usr/bin/cygsasl2-3.dll (0x5b4c40000) cygk5crypto-3.dll => /usr/bin/cygk5crypto-3.dll (0x41bf60000) cygkrb5-3.dll => /usr/bin/cygkrb5-3.dll (0x4cc8c0000) cygkrb5support-0.dll => /usr/bin/cygkrb5support-0.dll (0x433a90000) cygcom_err-2.dll => /usr/bin/cygcom_err-2.dll (0x40f9e0000) $ ... that the dependent dll's are linked, via the cygcurl-4.dll. Indeed, if we look at the 'import tables' from the '.idata' section of the dll, we find: $ objdump -x /usr/bin/cygcurl-4.dll | grep 'DLL Name' DLL Name: cygbrotlidec-1.dll DLL Name: cygcrypto-1.1.dll DLL Name: cygwin1.dll DLL Name: cygidn2-0.dll DLL Name: cyglber-2-4-2.dll DLL Name: cygldap-2-4-2.dll DLL Name: cygnghttp2-14.dll DLL Name: cygpsl-5.dll DLL Name: cygssh-4.dll DLL Name: cygssl-1.1.dll DLL Name: cygz.dll DLL Name: KERNEL32.dll $ Indeed, all of the dependent dll's are available in /usr/bin. If we take the libnghttp2 as an example, we see that the library package is installed, but not the associated '-devel' package. (so the .dll is present, but the 'import library' needed by the linker isn't). $ cygcheck -cd | grep nghttp2 libnghttp2_14 1.37.0-1 $ cygcheck -l libnghttp2_14 /usr/bin/cygnghttp2-14.dll $ cygcheck -p libnghttp2-devel Found 5 matches for libnghttp2-devel libnghttp2-devel-1.23.1-1 - libnghttp2-devel: HTTP/2 C library libnghttp2-devel-1.31.0-1 - libnghttp2-devel: HTTP/2 C library libnghttp2-devel-1.37.0-1 - libnghttp2-devel: HTTP/2 C library nghttp2-1.23.1-1 - nghttp2: HTTP/2 utilities nghttp2-1.31.0-1 - nghttp2: HTTP/2 utilities $ cygcheck -p usr/lib/libnghttp2.dll.a Found 3 matches for usr/lib/libnghttp2.dll.a libnghttp2-devel-1.23.1-1 - libnghttp2-devel: HTTP/2 C library libnghttp2-devel-1.31.0-1 - libnghttp2-devel: HTTP/2 C library libnghttp2-devel-1.37.0-1 - libnghttp2-devel: HTTP/2 C library $ [Note: the 'cygcheck -p' invocations search the cygwin.com site, rather than the local disk.] So, you could say that installing the 'libcurl-devel' package should cause all of the dependent library '-devel' packages to be installed as well. That would (probably) work. But why? Doing so would only cause the linker to search several import libraries which will _never_ supply any symbols! ;-D Hmm, I suppose it could be argued that this is a cygwin-specific packaging issue (no, I am not convinced) and should be addressed by the cygwin package maintainer. (maybe the cygwin list should be my next port of call). What do you think? Thanks! ATB, Ramsay Jones

[bagder]

I asked about static vs shared because the configure generated curl-config --libs should only return the full list like that if not built shared, and AFAIK shared builds are by default. For a shared build/lib (which is what you seem to be using) a single -lcurl is probably the correct thing as the rest will then be dynamically linked.

So yeah, it could be a cygwin-specific packaging issue or perhaps even a cygwin-specific configure issue in curl. I don't know.

[me-and]

@ramsay-jones pointed me at this issue; I'm the Cygwin maintainer for Git. This is indeed a Cygwin-specific packaging issue; it looks like the Curl maintainer mistakenly dropped a patch to the compile-time configuration that affects the linking behaviour.

I'm expecting this to be fixed in Cygwin shortly; a new patch was proposed at https://cygwin.com/pipermail/cygwin/2020-August/245820.html earlier today.

[bagder]

Thanks that input @me-and. I presume we can then consider this case closed @ramsay-jones ?

[ramsay-jones]

On 10/08/2020 22:25, Daniel Stenberg wrote: Thanks that input @me-and <https://github.com/me-and>. I presume we can then consider this case closed @ramsay-jones <https://github.com/ramsay-jones> ?

Hmm, I think I would be more interested about the need for a cygwin specific patch to the curl build system to make it work! ;-) [Also, I think the cygwin maintainer of the libcurl package is not convinced by the current patch]. However, if you think this issue is no longer serving a purpose (I guess the cygwin maintainer can open a new issue if he has a patch he wants to send upstream), please do close this issue. Thanks! ATB, Ramsay Jones

[bagder]

I think I would be more interested about the need for a cygwin specific patch to the curl build system to make it work! ;-)

I would agree, but I don't see any strong desire to fix that. It seems the curl maintainers in cygwin has been doing their own patches on curl for a very long time without having tried to fix them directly upstream - in curl.

[BrianInglis]

I'm the new Cygwin curl maintainer, and I would prefer to drop local patches if we can get appropriate changes made upstream wherever possible.
The local specific patch against an earlier curl release did not apply, and I did not notice any issues, so I released the package upgrade to the latest curl.
If we need to keep a local patch against curl/-config/.in I don't have a problem, but would prefer to discuss an approach and offer a patch/PR if desirable.
Often product maintainers have their own preferred approaches based on their deep knowledge and experience of the packages they maintain, and will come up with a more general approach than my narrow look at one issue would suggest.

It looks to me as if the generated curl-config --libs statements:

        if test "Xyes" = "Xno" -o "Xyes" = "Xyes"; then
          echo ${CURLLIBDIR}-lcurl -lnghttp2 -lidn2 -lssh -lpsl -lssl -lcrypto
-lldap -llber -lbrotlidec -lbrotlidec -lz

based on curl-config.in:

        if test "X@ENABLE_SHARED@" = "Xno" -o "X@REQUIRE_LIB_DEPS@" = "Xyes"; then
          echo ${CURLLIBDIR}-lcurl @LIBCURL_LIBS@

REQUIRE_LIB_DEPS should be no, derived from configure.ac:

if test "X$enable_shared" = "Xyes" -a "X$link_all_deplibs" = "Xno"
then
    REQUIRE_LIB_DEPS=no
else
    REQUIRE_LIB_DEPS=yes
fi
AC_SUBST(REQUIRE_LIB_DEPS)
AM_CONDITIONAL(USE_EXPLICIT_LIB_DEPS, test x$REQUIRE_LIB_DEPS = xyes)

but for Cygwin and a number of other platforms link_all_deplibs remains defaulted to unknown, so either that variable should be set in Cygwin configure to no, or that condition should perhaps be changed to:

if test "X$enable_shared" = "Xyes" -a "X$link_all_deplibs" != "Xyes"

[bagder]

Thanks! It feels a little risky as this might change a little too much for other platforms, but hey let's try this. See #5804 for what I take is your suggested fix?

[kbrow1i]

This strikes me as a very fragile fix, likely to confuse anyone reading the code in the future, for two reasons.

1. The libtool manual says the following about link_all_deplibs: "Whether libtool must link a program against all its dependency libraries. Set to ‘yes’ or ‘no’. Default is ‘unknown’, which is a synonym for ‘yes’." But this patch treats 'unknown' as different from 'yes'.

2. Setting REQUIRE_LIB_DEPS to 'no' on Cygwin gives the impression that we don't need to specify all dependent libraries when building curl on Cygwin. [In fact we do need to do this, and libtool is smart enough to do it regardless of how REQUIRE_LIB_DEPS is defined.]

How about the following much simpler fix, which is essentially what's been done in the Cygwin build of curl for years, until the most recent build:

--- origsrc/curl-7.71.1/curl-config.in 2020-06-27 18:03:53.000000000 -0400
+++ src/curl-7.71.1/curl-config.in 2020-08-12 12:34:53.341934900 -0400
@@ -160,7 +160,7 @@ while test $# -gt 0; do
else
CURLLIBDIR=""
fi

•    if test "X@ENABLE_SHARED@" = "Xno" -o "X@REQUIRE_LIB_DEPS@" = "Xyes"; then
  

•    if test "X@ENABLE_SHARED@" = "Xno"; then
       echo ${CURLLIBDIR}-lcurl @LIBCURL_LIBS@
     else
       echo ${CURLLIBDIR}-lcurl
  

[kbrow1i]

Whoops! I see the patch got mangled. Let me try attaching it instead.
curl-7.71.1-1.src.txt

[BrianInglis]

This strikes me as a very fragile fix, likely to confuse anyone reading the code in the future, for two reasons.

1. The libtool manual says the following about link_all_deplibs: "Whether libtool must link a program against all its dependency libraries. Set to ‘yes’ or ‘no’. Default is ‘unknown’, which is a synonym for ‘yes’." But this patch treats 'unknown' as different from 'yes'.

I noticed that in the autoconf/libtool docs and thought that perhaps a better option might be to make changes or get changes made in m4/libtool.m4 to set link_all_deplibs=no like Linux in the LT_SHLIBS section.

2. Setting REQUIRE_LIB_DEPS to 'no' on Cygwin gives the impression that we don't need to specify all dependent libraries when building curl on Cygwin. [In fact we do need to do this, and libtool is smart enough to do it regardless of how REQUIRE_LIB_DEPS is defined.]

I understood the need for the patch was that Cygwin needs dynamically linked against libcurl only link_all_deplibs=no, but statically also needs linked against all the libcurl dependencies link_all_deplibs=yes, effectively the difference between libcurl.pc Libs list and Libs.private list, and that is what I understood to be the meaning of link_all_deplibs.

I did not need a patch because I was able to build both curl and git Cygwin packages, including the git programs where others had issues, with no changes to curl and git cygport build control scripts in the Cygwin package sources, so was unable to reproduce the issues reported by others.

With a large number of Cygwin package-devel packages and dependencies installed, I understand my development configuration may be atypical, and as curl-config did not generate the expected output, thought consideration should be given to getting that issue fixed.

How about the following much simpler fix, which is essentially what's been done in the Cygwin build of curl for years, until the most recent build:

To include a patch or any code block, use Markdown verbatim code block quotes around content: three graves ``` on separate lines before and after the block. If you also name the language or source type immediately following the graves on the first line - in this case ```patch additional source highlighting may occur. e.g.

--- origsrc/curl-7.71.1/curl-config.in	2020-06-27 18:03:53.000000000 -0400
+++ src/curl-7.71.1/curl-config.in	2020-08-12 12:34:53.341934900 -0400
@@ -160,7 +160,7 @@ while test $# -gt 0; do
         else
            CURLLIBDIR=""
         fi
-        if test "X@ENABLE_SHARED@" = "Xno" -o "X@REQUIRE_LIB_DEPS@" = "Xyes"; then
+        if test "X@ENABLE_SHARED@" = "Xno"; then
           echo ${CURLLIBDIR}-lcurl @LIBCURL_LIBS@
         else
           echo ${CURLLIBDIR}-lcurl