Skip to content

Conversation

@vszakats
Copy link
Member

@vszakats vszakats commented Apr 5, 2025

Rework the way curl's custom Find modules advertise their properties.

Before this patch, Find modules returned detected dependency properties
(header dirs, libs, libdirs, C flags, etc.) via global variables. curl's
main CMakeLists.txt copied their values into global lists, which it
later applied to targets. This solution worked internally, but it was
unsuited for the public, distributed CURLConfig.cmake and publishing
curl's Find modules with it, due to polluting the namespace of consumer
projects. It's also impractical to apply the many individual variables
to every targets depending on libcurl.

To allow using Find modules in consumer projects, this patch makes them
define as imported interface targets, named CURL::<dependency>. Then
store dependency information as target properties. It avoids namespace
pollution and makes the dependency information apply automatically
to all targets using CURL::libcurl_static.

Find modules continue to return *_FOUND and *_VERSION variables.

For dependencies detected via pkg-config, CMake 3.16+ is recommended.
Older CMake versions have a varying degree of support for
propagating/handling library directories. This may cause issues in envs
where dependencies reside in non-system locations and detected via
pkg-config (e.g. macOS + Homebrew). Use CURL_USE_PKGCONFIG=OFF
to fix these issues. Or upgrade to newer CMake, or link libcurl
dynamically.

Also:

  • re-enable pkg-config for old cmake find_library() integration
    tests.
  • make curlinfo build after these changes.
  • distribute local Find modules.
  • export the raw list of lib dependencies via CURL_LIBRARIES_PRIVATE.
  • CURLconfig.cmake: use curl's Find modules to detect dependencies in
    the consumer env.
  • add custom property to target property debug function.
  • the curl build process no longer modifies CMAKE_C_FLAGS.
    Follow-up to e865420 cmake: prefer COMPILE_OPTIONS over CMAKE_C_FLAGS for custom C options #17047

Ref: #14930
Ref: libssh2/libssh2#1535
Ref: libssh2/libssh2#1571
Ref: libssh2/libssh2#1581
Ref: libssh2/libssh2#1623


w/o sp https://github.com/curl/curl/pull/16973/files?w=1

@vszakats vszakats added the cmake label Apr 5, 2025
@vszakats vszakats marked this pull request as draft April 5, 2025 01:49
@github-actions github-actions bot added the build label Apr 5, 2025
@vszakats vszakats changed the title cmake: define dependencies as interface targets cmake: define dependencies as imported interface targets Apr 5, 2025
@vszakats
Copy link
Member Author

vszakats commented Apr 6, 2025

There is more legwork to do here (esp. with GSS, a special case), but the bigger worry, Old Linux compatibility seems to be fixed now.

@vszakats
Copy link
Member Author

vszakats commented Apr 6, 2025

After tests with 15 different, old cmake versions, it turns out that anything older
than 3.19 is what feels like overly strict when it comes to interface
target properties. They don't allow to add custom ones and refuse to
read valid, existing ones too. None of this is documented, and no obvious
mention it in the release notes, when it was finally relaxed/fixed:
https://cmake.org/cmake/help/v4.0/release/3.19.html

The refusal to read the LOCATION property is still an issue to solve
for these versions.

Also, it isn't just Old Linux, but multiple jobs tested in AppVeyor too.
They fail to a varying degree depending on which dependencies they
touch (most likely just OpenSSL.)

@vszakats
Copy link
Member Author

vszakats commented Apr 6, 2025

Remaining issues:

  • GSS (edit: turned out to be post-feature detection issues, ref to deleted global detection vars)
  • lib order changed (not sure why), breaking gcc with its picky linker

@vszakats
Copy link
Member Author

vszakats commented Apr 7, 2025

It seems that CMake messes up lib order when using imported interface targets. A rather terrible surprise.

Since 2018:
https://stackoverflow.com/questions/53942683/cmake-link-order-of-imported-targets-incorrect

After recently fixing lib order using non-modern cmake, it surprises me that "modern" cmake makes this actively impossible. The suggested solution on Stack Overflow could just as well be suggesting to format C: and switch professions.

Honorable mention to binutils ld which lets everyone indulge in the multi-decade old tradition of manually topo-sorting the input lib list, because this isn't a task a computer program (let alone a LINKER) could possibly solve automatically.

@vszakats
Copy link
Member Author

vszakats commented Apr 7, 2025

So, what CMake is doing is moving OpenSSL::SSL, OpenSSL::Crypto and ZLIB::ZLIB
to the beginning of the lib list on the linker command-line, while these should come
after libssh* for example. This is a CMake "feature", or possibly a side-effect of the
complexities of "imported targets", which come in different flavors, settings, constraints
and colorful error messages when anything is amiss (extra fun to make it work with older
versions). And then, CMake automagically decides their order.

There is also a core assumption in CMake, that a library can always be addressed
with an absolute path on disk. This may be true with MSVC, but in general, it
isn't true. Most linkers don't work like this. pkg-config also doesn't work like
this, and there is no way to translate its detection results into absolute paths.
(well, there may be by trying to guess and match what the linker would do.)

CMake seems to have invented find_library() around the idea, which, by passing
pkg-config -L directories to it as "hints", it will magically convert it to an absolute
path to feed its core idea, and make things work as designed. But, passing hints and
some hard-coded libnames will not produce the same results as just simply using
pkg-config would. It also needs to maintain a list of hard-coded names, and generally
also assumes that one dependency == one library on disk.

Most likely I'm missing something and this assessment is just as skewed as any other,
but the end result is that we're now painted in a corner, where we'd want to be compatible
with pkg-config, do the right thing and support libnames + libpaths, use modern CMake
syntax, and still be able to pass libs in the correct order to picky linkers like ld.

I don't know what is the magic trick to resolve this Gordian Knot.

edit: all this mostly (exclusively?) affects static linking.

@vszakats vszakats force-pushed the cm-interface branch 6 times, most recently from b003226 to 18c3b77 Compare April 14, 2025 23:55
@vszakats vszakats marked this pull request as ready for review April 16, 2025 17:56
@vszakats
Copy link
Member Author

vszakats commented Apr 17, 2025

I guess one way to deal with the broken lib order emitted by CMake
is to convert CURL_LIBS into raw options and pass those to targets,
instead of relying on CMake to do this (and messing up the lib order
in the process.)

We already do this as part of generating libcurl.pc.

A better approach would be telling CMake about lib interdependencies,
to make it create a correct lib order, but so far it resisted my attempts
to do this.

vszakats added a commit to curl/curl-for-win that referenced this pull request Apr 17, 2025
An earlier upstream patch made it possible to drop this hack.

There is an open PR in curl, that's aiming to modernize dependency
detection by using imported interface targets and target properties.
That PR revealed an issue, where this solution trigger CMake automatisms
that makes it override the lib order as passed to CMake. This in turn
reintroduces the issue of linker errors with gcc's picky binuntils `ld`
linker, which is sensitive to this.

I've found no solution to get back control over lib order when using
imported interface targets. Yet the PR seems like a good direction
to take, and improves static linking by automatically pulling in all lib
dependencies when consuming libcurl. It overall seems like a useful
improvement despite the regression in lib order.

To work around the lib order regression, let's reintroduce the previous
hack to make `ld` work with the lib order CMake emits.

Ref: curl/curl#16973

Reverts 2ed700e
@github-actions github-actions bot added CI Continuous Integration tests labels Apr 20, 2025
vszakats added a commit to vszakats/curl that referenced this pull request Apr 22, 2025
Fixes:
```
CMake Error at bld-curl/_pkg/lib/cmake/CURL/CURLConfig.cmake:62 (add_library):
  add_library cannot create ALIAS target "CURL::libcurl" because target
  "CURL::libcurl_shared" is imported but not globally visible.
Call Stack (most recent call first):
  CMakeLists.txt:39 (find_package)
```

Seen with cmake 3.12 when running tests/cmake with:
```shell
export CMAKE_CONSUMER=/path/to/CMake-3.12.0/bin/cmake
./test.sh find_package -DBUILD_STATIC_CURL=ON
```

I don't claim to understand what this error says, why it happens in
certain CMake versions, and why such workaround is necessary for what
seems like rather standard export/consume configuration.

Cherry-picked from curl#16973
vszakats added a commit that referenced this pull request Apr 22, 2025
…ming libcurl with old cmake

Fixes:
```
CMake Error at bld-curl/_pkg/lib/cmake/CURL/CURLConfig.cmake:62 (add_library):
  add_library cannot create ALIAS target "CURL::libcurl" because target
  "CURL::libcurl_shared" is imported but not globally visible.
Call Stack (most recent call first):
  CMakeLists.txt:39 (find_package)
```

tests/cmake reproducer (requires #16973):
```shell
export CMAKE_CONSUMER=/path/to/CMake-3.12.0/bin/cmake
./test.sh find_package
```

I don't understand what this error says, why it happens in certain CMake
versions, and why a workaround is necessary for what seems like
a standard export/consume configuration. This patch is based on internet
suggestions and other projects ending up with this workaround.

Cherry-picked from #16973
Closes #17140
@vszakats
Copy link
Member Author

augment review

@augmentcode
Copy link

augmentcode bot commented Nov 28, 2025

Sorry, Augment does not have access to the org this pull request's branch is from.

@vszakats vszakats closed this in 16f073e Nov 29, 2025
@vszakats vszakats removed on-hold feature-window A merge of this requires an open feature window labels Nov 29, 2025
@vszakats vszakats deleted the cm-interface branch November 29, 2025 00:42
@dg0yt
Copy link
Contributor

dg0yt commented Nov 29, 2025

At least not after rc2.

@dg0yt
Copy link
Contributor

dg0yt commented Nov 29, 2025

libcurl.pc for x64-linux (vcpkg, static library linkage, pristine source):

Requires: libssh2,libidn2,ldap,lber,libcares,openssl,mbedtls,mbedx509,mbedcrypto,wolfssl,gnutls,nettle,zlib,libbrotlidec,libbrotlicommon,libzstd,libnghttp2,libpsl,libgsasl,mit-krb5-gssapi,librtmp
Requires.private: libssh2,libidn2,ldap,lber,libcares,openssl,mbedtls,mbedx509,mbedcrypto,wolfssl,gnutls,nettle,zlib,libbrotlidec,libbrotlicommon,libzstd,libnghttp2,libpsl,libgsasl,mit-krb5-gssapi,librtmp
Libs: -L${libdir} -lcurl -lssh2 -lcrypto -ldl -lz -lidn2 -lunistring -lldap -lssl -lcrypto -ldl -llber -lcares -lssl -lcrypto -lmbedtls -lmbedx509 -lmbedcrypto -lwolfssl -lm -lm -lgnutls -lgmp -lunistring -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 -lunistring -lz -lnettle -lz -lbrotlidec -lbrotlicommon -lm -lzstd -lnghttp2 -lpsl -lunistring -lidn2 -lunistring -lgsasl -lidn -lntlm -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lkrb5support -lresolv -lrtmp -lz -lssl -lcrypto -ldl -lcrypto -lz
Libs.private: -L/home/kpa/Projekte/vcpkg/vcpkg/installed/x64-linux/lib -lssh2 -lcrypto -ldl -lz -lidn2 -lunistring -lldap -lssl -lcrypto -ldl -llber -lcares -lssl -lcrypto -lmbedtls -lmbedx509 -lmbedcrypto -lwolfssl -lm -lm -lgnutls -lgmp -lunistring -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 -lunistring -lz -lnettle -lz -lbrotlidec -lbrotlicommon -lm -lzstd -lnghttp2 -lpsl -lunistring -lidn2 -lunistring -lgsasl -lidn -lntlm -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lkrb5support -lresolv -lrtmp -lz -lssl -lcrypto -ldl -lcrypto -lz

Pretty much redundancy. Even more with pkgconfig -static.

@vszakats
Copy link
Member Author

vszakats commented Nov 29, 2025

Requires:

Requires: libssh2,libidn2,ldap,lber,libcares,openssl,mbedtls,mbedx509,mbedcrypto,wolfssl,gnutls,nettle,zlib,libbrotlidec,libbrotlicommon,libzstd,libnghttp2,libpsl,libgsasl,mit-krb5-gssapi,librtmp
Requires.private: libssh2,libidn2,ldap,lber,libcares,openssl,mbedtls,mbedx509,mbedcrypto,wolfssl,gnutls,nettle,zlib,libbrotlidec,libbrotlicommon,libzstd,libnghttp2,libpsl,libgsasl,mit-krb5-gssapi,librtmp
Libs: -L${libdir} -lcurl -lssh2 -lcrypto -ldl -lz -lidn2 -lunistring -lldap -lssl -lcrypto -ldl -llber -lcares -lssl -lcrypto -lmbedtls -lmbedx509 -lmbedcrypto -lwolfssl -lm -lm -lgnutls -lgmp -lunistring -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 -lunistring -lz -lnettle -lz -lbrotlidec -lbrotlicommon -lm -lzstd -lnghttp2 -lpsl -lunistring -lidn2 -lunistring -lgsasl -lidn -lntlm -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lkrb5support -lresolv -lrtmp -lz -lssl -lcrypto -ldl -lcrypto -lz
Libs.private: -L/home/kpa/Projekte/vcpkg/vcpkg/installed/x64-linux/lib -lssh2 -lcrypto -ldl -lz -lidn2 -lunistring -lldap -lssl -lcrypto -ldl -llber -lcares -lssl -lcrypto -lmbedtls -lmbedx509 -lmbedcrypto -lwolfssl -lm -lm -lgnutls -lgmp -lunistring -latomic -lhogweed -lgmp -lnettle -ltasn1 -lidn2 -lunistring -lz -lnettle -lz -lbrotlidec -lbrotlicommon -lm -lzstd -lnghttp2 -lpsl -lunistring -lidn2 -lunistring -lgsasl -lidn -lntlm -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lkrb5support -lresolv -lrtmp -lz -lssl -lcrypto -ldl -lcrypto -lz

This has been working like this for a long time, and unrelated to this PR, or am I missing something?
Also what would be the expected behavior in this case? (static-only builds)

vszakats added a commit to libssh2/libssh2 that referenced this pull request Nov 29, 2025
vszakats added a commit that referenced this pull request Nov 29, 2025
@dg0yt
Copy link
Contributor

dg0yt commented Nov 29, 2025

It is not related to this PR.
I just never look at the pristine installation results because in vcpkg these parts always had to be patched heavily. (And it will probably remain like that.

The expected behavior is at least that information from Requires* isn't duplicated in Libs*, and that [Libs/Requires].private isn't duplicated in [Libs/Requires]. (.private being empty is a reasonable approach when there is no shared lib.)

@vszakats
Copy link
Member Author

It is not related to this PR. I just never look at the pristine installation results because in vcpkg these parts always had to be patched heavily. (And it will probably remain like that.

The expected behavior is at least that information from Requires* isn't duplicated in Libs*, and that [Libs/Requires].private isn't duplicated in [Libs/Requires]. (.private being empty is a reasonable approach when there is no shared lib.)

Sounds reasonable, but touching this seems like asking for a lot of trouble,
when it breaks someone's use-case. And a lot of work, also with autotools
to keep them in sync. If someone wants to do this, it would be useful stuff.
Till then, being redundant is not beautiful, but does the job.

(I wonder how lib order (combined from modules and raw libs) is turning out
without redundancy, thinking of the gcc ld 1-pass linker.)

@dg0yt
Copy link
Contributor

dg0yt commented Nov 29, 2025

(I wonder how lib order (combined from modules and raw libs) is turning out
without redundancy, thinking of the gcc ld 1-pass linker.)

Modules encode transitive usage requirements. Nothing is lost. It just works.
(Also: Broken modules in, broken link order out.)

touching this seems like asking for a lot of trouble,
when it breaks someone's use-case.

Studying downstream issues is a lot of trouble with complex link library lists.

vszakats added a commit that referenced this pull request Nov 30, 2025
```
CMakeConfigurableFile.in
cmake_uninstall.cmake.in
curl-config.cmake.in
```

Follow-up to 16f073e #16973
Closes #19773
vszakats added a commit to vszakats/curl that referenced this pull request Nov 30, 2025
vszakats added a commit that referenced this pull request Nov 30, 2025
vszakats added a commit that referenced this pull request Dec 1, 2025
Set it only while using local Find modules, leave it as-is while using
system ones.

Follow-up to 16f073e #16973
Cherry-picked from #19776
vszakats added a commit that referenced this pull request Dec 1, 2025
Show a message if the CMake version is lower than that when consuming
libcurl via the CMake config.

The minimum CMake version on consumption is for now the same as
the minimum required (v3.7) to build curl itself.

Ref: https://cmake.org/cmake/help/v3.7/variable/CMAKE_MINIMUM_REQUIRED_VERSION.html
Ref: #18704 (discussion)
Follow-up to 16f073e #16973
Closes #19776
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Development

Successfully merging this pull request may close these issues.

3 participants