cmake: add native `pkg-config` detection for mbedTLS, MSH3, Quiche, Rustls, wolfSSL by vszakats · Pull Request #15193 · curl/curl

vszakats · 2024-10-08T10:53:52Z

Also:

detect and add required system libraries for Rustls on macOS and
non-Windows.
add Linux CMake jobs for the touched dependencies.
Caveats:
- MSH3 generates a broken libmsh3.pc, so needs manual config.
  Upstream PR: fix generated libmsh3.pc file nibanks/msh3#225
- Rustls .pc file missing, so needs manual config.

An internal change worthy of mention is that we are using the lib path
and name information returned by pkg-config as-is. Meaning the libname
doesn't include the full path, like it's usual with native cmake
detection. The path comes separately and needs to be rolled separately.
For this we add it to targets via link_directories(). We also keep tab
of them in CURL_LIBDIRS and use that in libcurl.pc. Feature checks
also need to receive these paths. CMake doesn't offer
a CMAKE_REQUIRED_* variable for this purpose, only
a CMAKE_REQUIRED_LINK_OPTIONS accepting raw linker flags. Add a macro
to convert a list of paths to linker options to solve it. wolfSSL
requires this for now.

w/o whitespace: https://github.com/curl/curl/pull/15193/files?w=1

dfandrich · 2024-10-08T19:44:02Z

Analysis of PR #15193 at 15287516:

Test 2055 failed, which has NOT been flaky recently, so there could be a real issue in the PR.

Generated by Testclutch

- update mbedTLS repo URL. - switch local mbedTLS build to use CMake, and Ninja. CMake build is required to create and install mbedTLS `pkg-config` files. (as of v3.6.1) `-DCMAKE_POSITION_INDEPENDENT_CODE=ON` required to avoid this error when linking mbedtls to `libcurl.so`: ``` /usr/bin/ld: /home/runner/mbedtls/lib/libmbedcrypto.a(cipher.c.o): warning: relocation against `mbedtls_cipher_base_lookup_table' in read-only section `.text' /usr/bin/ld: /home/runner/mbedtls/lib/libmbedtls.a(ssl_tls.c.o): relocation R_X86_64_PC32 against symbol `mbedtls_x509_crt_profile_suiteb' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: bad value ``` Ref: https://github.com/curl/curl/actions/runs/11245069259/job/31264386723#step:40:43 - make local mbedTLS build 10x smaller by omitting programs and tests. - GHA/linux: fix cmake warning by adding `-B .` option. - GHA/linux: add build-only cmake job for packaged mbedTLS (2.x). - fix compiler warning when building with mbedTLS 2.x: ``` /home/runner/work/curl/curl/lib/vtls/mbedtls.c:344:1: error: ‘mbed_cipher_suite_get_str’ defined but not used [-Werror=unused-function] 344 | mbed_cipher_suite_get_str(uint16_t id, char *buf, size_t buf_size, | ^~~~~~~~~~~~~~~~~~~~~~~~~ ``` Ref: https://github.com/curl/curl/actions/runs/11244999065/job/31264168295#step:40:50 Also in preparation for #15193 Closes #15208

vszakats · 2024-10-09T13:23:26Z

Field notes:

msh3 generates a broken libmsh3.pc (misses @ONLY from the CMake function call).
It also fails to write the actual install prefix into it. → fix generated libmsh3.pc file nibanks/msh3#225
mbedTLS cmake build consistently fails two valgrind tests.
rustls in theory supports a .pc file, but it's not present in the build.
https://github.com/curl/curl/actions/runs/11256157378/job/31297407708?pr=15193#step:30:149
This was based on cmake: add rustls #14534 (comment) and m4/curl-rustls.m4. But this is apparently false?
On ArchLinux there is a .pc file: https://archlinux.org/packages/extra/x86_64/librustls/files/
In the source repo: https://github.com/rustls/rustls-ffi/blob/58bb1f0a23b105e5ad71fc3b1d8b5c1377ecf30f/Makefile.pkg-config
Something like this should build the .pc file (...and also shared lib?) for Rustls: make PREFIX=$HOME/rustls install -f Makefile.pkg-config. But then it fails with the command cargo cinstall. Failed to find more about this command. Package cargo-c seemed like a package that offers it based on the ArchLinux package source, but such package doesn't existing on Ubuntu. Looks like it requires 24.04. This feels like extreme complexity just to obtain a simple .pc file...

use `CMAKE_LIBRARY_PATH_FLAG` https://cmake.org/cmake/help/v3.6/variable/CMAKE_LIBRARY_PATH_FLAG.html

mbedtls.pc only has references to its dependencies when building statically. mbedtls.pc ``` prefix=/usr/local/Cellar/mbedtls/3.6.1 includedir=${prefix}/include libdir=${prefix}/lib Name: Mbed TLS Description: Mbed TLS is a C library that implements cryptographic primitives, X.509 certificate manipulation and the SSL/TLS and DTLS protocols. Its small code footprint makes it suitable for embedded systems. URL: https://www.trustedfirmware.org/projects/mbed-tls/ Version: 3.6.1 Requires.private: mbedcrypto mbedx509 Cflags: -I"${includedir}" Libs: -L"${libdir}" -lmbedtls ``` mbedx509.pc ``` prefix=/usr/local/Cellar/mbedtls/3.6.1 includedir=${prefix}/include libdir=${prefix}/lib Name: Mbed TLS Description: Mbed TLS is a C library that implements cryptographic primitives, X.509 certificate manipulation and the SSL/TLS and DTLS protocols. Its small code footprint makes it suitable for embedded systems. URL: https://www.trustedfirmware.org/projects/mbed-tls/ Version: 3.6.1 Requires.private: mbedcrypto Cflags: -I"${includedir}" Libs: -L"${libdir}" -lmbedx509 ``` mbedcrypto.pc ``` prefix=/usr/local/Cellar/mbedtls/3.6.1 includedir=${prefix}/include libdir=${prefix}/lib Name: Mbed TLS Description: Mbed TLS is a C library that implements cryptographic primitives, X.509 certificate manipulation and the SSL/TLS and DTLS protocols. Its small code footprint makes it suitable for embedded systems. URL: https://www.trustedfirmware.org/projects/mbed-tls/ Version: 3.6.1 Cflags: -I"${includedir}" Libs: -L"${libdir}" -lmbedcrypto ```

- limit `SIZEOF_SA_FAMILY_T` detection to non-Windows. - make sure `sys/socket.h` exists before detecting `SIZEOF_SA_FAMILY_T`. - limit `mach_absolute_time()` detection to `APPLE`. Drop from Windows pre-cache. - skip `HAVE_LIBSOCKET` detection for Windows, drop pre-cached value. - drop redundant pre-cached `HAVE_LIBZ` for Windows. - `curl_required_libpaths()`: stop accepting multiple arguments. To prepare for `CMAKE_REQUIRED_LINK_DIRECTORIES` support. Follow-up to 7bab201 #15193 - GSS: fix recent rebase mistakes: - fix variable name. - do not add a header twice. Follow-up to 91d451b #15157 - GSS: quote a variable. Closes #15271

Add a build-only cmake job with system mbedTLS package and `pkg-config` enabled. Ubuntu 24.04 comes with mbedTLS 2.28.8 which supports `pkg-config`. Follow-up to 7bab201 #15193 Closes #15286

To sync with other similar variables and hide them from cmake UIs. Follow-up to 7bab201 #15193 Closes #15407

`pkg_check_modules()` seems to leave `<PACKAGE>_VERSION` defined with an empty value, if the package is not found. When the package is also not found in the fallback branch, `find_package_handle_standard_args()` logs and error message. In this message it includes the bogus empty value as: `(found version "")`: ``` Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) (found version "") ``` https://github.com/curl/curl/actions/runs/11509727553/job/32040378958?pr=15408#step:31:99 Clear the version number to avoid the confusion: ``` Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) ``` https://github.com/curl/curl/actions/runs/11510022503/job/32041149129?pr=15408#step:31:99 Seen with CMake v3.30.5 Follow-up to 7bab201 curl#15193

`pkg_check_modules()` seems to leave `<PACKAGE>_VERSION` defined with an empty value, if the package is not found. When the package is also not found in the fallback branch, `find_package_handle_standard_args()` logs and error message. In this message it includes the bogus empty value as: `(found version "")`: ``` Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) (found version "") ``` https://github.com/curl/curl/actions/runs/11509727553/job/32040378958?pr=15408#step:31:99 Clear the version number to avoid the confusion: ``` Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) ``` https://github.com/curl/curl/actions/runs/11510022503/job/32041149129?pr=15408#step:31:99 Seen with CMake v3.30.5. Follow-up to 7bab201 #15193 Closes #15409

Use `CMAKE_REQUIRED_LINK_DIRECTORIES` with CMake 3.31.0 and upper, in local macro `curl_required_libpaths()`. Kitware/CMake@9e95bd4 https://gitlab.kitware.com/cmake/cmake/-/commit/9e95bd49f278cd2a05caf21fd624a41e4bfaba60 https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9795 https://cmake.org/cmake/help/v3.31/module/CheckSymbolExists.html Tested OK with cmake 3.31.0-rc1. Follow-up to 01a8157 #15271 Follow-up to 7bab201 #15193 Closes #15280

brotli, c-ares, libpsl, libssh2, nghttp2, nghttp3, ntgcp2, zstd. Also: Add workaround for CMake reporting successful libssh2 detection, but leaving the header directory empty, and causing `libssh2.h` not found while compiling. It happens when `pkgconf` is not detecting libssh2 dependency libcrypto in Homebrew after `brew unlink openssl` (as in GHA/macos). The workaround is to require a non-empty header directory to consider the detection successful. This workaround may need to be tweaked and/or applied to other Find modules. Follow-up to 7bab201 #15193 Closes #15408

- add `iphlpapi` library for c-ares. Ref: curl/curl-for-win@082d98b - fix to not add system libs if the dependency was not found. librtmp, Rustls, wolfSSL Follow-up to 421e592 #15832 Follow-up to 7bab201 #15193 Closes #16089

- update mbedTLS repo URL. - switch local mbedTLS build to use CMake, and Ninja. CMake build is required to create and install mbedTLS `pkg-config` files. (as of v3.6.1) `-DCMAKE_POSITION_INDEPENDENT_CODE=ON` required to avoid this error when linking mbedtls to `libcurl.so`: ``` /usr/bin/ld: /home/runner/mbedtls/lib/libmbedcrypto.a(cipher.c.o): warning: relocation against `mbedtls_cipher_base_lookup_table' in read-only section `.text' /usr/bin/ld: /home/runner/mbedtls/lib/libmbedtls.a(ssl_tls.c.o): relocation R_X86_64_PC32 against symbol `mbedtls_x509_crt_profile_suiteb' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: bad value ``` Ref: https://github.com/curl/curl/actions/runs/11245069259/job/31264386723#step:40:43 - make local mbedTLS build 10x smaller by omitting programs and tests. - GHA/linux: fix cmake warning by adding `-B .` option. - GHA/linux: add build-only cmake job for packaged mbedTLS (2.x). - fix compiler warning when building with mbedTLS 2.x: ``` /home/runner/work/curl/curl/lib/vtls/mbedtls.c:344:1: error: ‘mbed_cipher_suite_get_str’ defined but not used [-Werror=unused-function] 344 | mbed_cipher_suite_get_str(uint16_t id, char *buf, size_t buf_size, | ^~~~~~~~~~~~~~~~~~~~~~~~~ ``` Ref: https://github.com/curl/curl/actions/runs/11244999065/job/31264168295#step:40:50 Also in preparation for curl#15193 Closes curl#15208

…ustls, wolfSSL Also: - detect and add required system libraries for Rustls on macOS and non-Windows. - add Linux CMake jobs for the touched dependencies. Caveats: - MSH3 generates a broken `libmsh3.pc`, so needs manual config. Upstream PR: nibanks/msh3#225 - Rustls `.pc` file missing, so needs manual config. An internal change worthy of mention is that we are using the lib path and name information returned by `pkg-config` as-is. Meaning the libname doesn't include the full path, like it's usual with native cmake detection. The path comes separately and needs to be rolled separately. For this we add it to targets via `link_directories()`. We also keep tab of them in `CURL_LIBDIRS` and use that in `libcurl.pc`. Feature checks also need to receive these paths. CMake doesn't offer a `CMAKE_REQUIRED_*` variable for this purpose, only a `CMAKE_REQUIRED_LINK_OPTIONS` accepting raw linker flags. Add a macro to convert a list of paths to linker options to solve it. wolfSSL requires this for now. Closes curl#15193

- limit `SIZEOF_SA_FAMILY_T` detection to non-Windows. - make sure `sys/socket.h` exists before detecting `SIZEOF_SA_FAMILY_T`. - limit `mach_absolute_time()` detection to `APPLE`. Drop from Windows pre-cache. - skip `HAVE_LIBSOCKET` detection for Windows, drop pre-cached value. - drop redundant pre-cached `HAVE_LIBZ` for Windows. - `curl_required_libpaths()`: stop accepting multiple arguments. To prepare for `CMAKE_REQUIRED_LINK_DIRECTORIES` support. Follow-up to 7bab201 curl#15193 - GSS: fix recent rebase mistakes: - fix variable name. - do not add a header twice. Follow-up to 91d451b curl#15157 - GSS: quote a variable. Closes curl#15271

Add a build-only cmake job with system mbedTLS package and `pkg-config` enabled. Ubuntu 24.04 comes with mbedTLS 2.28.8 which supports `pkg-config`. Follow-up to 7bab201 curl#15193 Closes curl#15286

To sync with other similar variables and hide them from cmake UIs. Follow-up to 7bab201 curl#15193 Closes curl#15407

`pkg_check_modules()` seems to leave `<PACKAGE>_VERSION` defined with an empty value, if the package is not found. When the package is also not found in the fallback branch, `find_package_handle_standard_args()` logs and error message. In this message it includes the bogus empty value as: `(found version "")`: ``` Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) (found version "") ``` https://github.com/curl/curl/actions/runs/11509727553/job/32040378958?pr=15408#step:31:99 Clear the version number to avoid the confusion: ``` Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) ``` https://github.com/curl/curl/actions/runs/11510022503/job/32041149129?pr=15408#step:31:99 Seen with CMake v3.30.5. Follow-up to 7bab201 curl#15193 Closes curl#15409

Use `CMAKE_REQUIRED_LINK_DIRECTORIES` with CMake 3.31.0 and upper, in local macro `curl_required_libpaths()`. Kitware/CMake@9e95bd4 https://gitlab.kitware.com/cmake/cmake/-/commit/9e95bd49f278cd2a05caf21fd624a41e4bfaba60 https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9795 https://cmake.org/cmake/help/v3.31/module/CheckSymbolExists.html Tested OK with cmake 3.31.0-rc1. Follow-up to 01a8157 curl#15271 Follow-up to 7bab201 curl#15193 Closes curl#15280

brotli, c-ares, libpsl, libssh2, nghttp2, nghttp3, ntgcp2, zstd. Also: Add workaround for CMake reporting successful libssh2 detection, but leaving the header directory empty, and causing `libssh2.h` not found while compiling. It happens when `pkgconf` is not detecting libssh2 dependency libcrypto in Homebrew after `brew unlink openssl` (as in GHA/macos). The workaround is to require a non-empty header directory to consider the detection successful. This workaround may need to be tweaked and/or applied to other Find modules. Follow-up to 7bab201 curl#15193 Closes curl#15408

- add `iphlpapi` library for c-ares. Ref: curl/curl-for-win@082d98b - fix to not add system libs if the dependency was not found. librtmp, Rustls, wolfSSL Follow-up to 421e592 curl#15832 Follow-up to 7bab201 curl#15193 Closes curl#16089

vszakats added build cmake labels Oct 8, 2024

vszakats force-pushed the cm-find-pkgconf-2 branch 3 times, most recently from 65c7bf5 to 0151919 Compare October 8, 2024 11:02

vszakats changed the title ~~cmake: add native pkg-config detection for mbedTLS, MSH3, Quiche, Rustls~~ cmake: add native pkg-config detection for mbedTLS, MSH3, Quiche, Rustls, wolfSSL Oct 8, 2024

vszakats added the TLS label Oct 8, 2024

github-actions Bot added the CI Continuous Integration label Oct 8, 2024

vszakats force-pushed the cm-find-pkgconf-2 branch from bae93ef to 56e9795 Compare October 8, 2024 21:49

vszakats mentioned this pull request Oct 8, 2024

GHA: add Linux and macOS mbedTLS jobs, fix issue #15208

Closed

vszakats added a commit to vszakats/curl that referenced this pull request Oct 8, 2024

delete new job that requires curl#15193

1a04cfd

vszakats force-pushed the cm-find-pkgconf-2 branch 2 times, most recently from c25512c to ca9b053 Compare October 9, 2024 01:21

vszakats force-pushed the cm-find-pkgconf-2 branch from 08309f1 to c431d80 Compare October 9, 2024 18:09

vszakats added 13 commits October 10, 2024 13:21

FindMSH3: native pkg-config support

597ef2f

FindRustls: native pkg-config support

d68fdd9

FindQuiche: native pkg-config support

488099c

FindMbedTLS: native pkg-config support

7ddcf62

FindWolfSSL: native pkg-config support

a56477c

Rustls: add system lib dependencies for macOS and non-Windows

9f11c01

cmake: use new Find result vars (MSH3)

4bec667

cmake: use new Find result vars (Quiche)

da1e7fa

cmake: use new Find result vars (Rustls)

4a30fb2

cmake: use new Find result vars (MbedTLS)

d1b5955

cmake: use new Find result vars (WolfSSL)

3bd48e7

pass bare libpaths to detection where necessary

57c542f

use `CMAKE_LIBRARY_PATH_FLAG` https://cmake.org/cmake/help/v3.6/variable/CMAKE_LIBRARY_PATH_FLAG.html

vszakats closed this in 7bab201 Oct 10, 2024

vszakats deleted the cm-find-pkgconf-2 branch October 10, 2024 12:46

github-actions Bot added the hacktoberfest-accepted label Oct 10, 2024

vszakats removed the hacktoberfest-accepted label Oct 10, 2024

vszakats mentioned this pull request Oct 11, 2024

cmake: tidy-ups and rebase fixups #15271

Closed

This was referenced Oct 12, 2024

cmake: use CMAKE_REQUIRED_LINK_DIRECTORIES #15280

Closed

GHA/linux: add cmake job for system mbedTLS with pkg-config #15286

Closed

vszakats mentioned this pull request Oct 24, 2024

cmake: mark as advanced some internal Find* variables #15407

Closed

vszakats added a commit that referenced this pull request Oct 24, 2024

cmake: mark as advanced some internal Find* variables

ffc4e6a

To sync with other similar variables and hide them from cmake UIs. Follow-up to 7bab201 #15193 Closes #15407

vszakats mentioned this pull request Oct 25, 2024

cmake: add native pkg-config detection for remaining Find modules #15408

Closed

2 tasks

vszakats mentioned this pull request Oct 25, 2024

cmake: clear package version after pkg-config detection #15409

Closed

cpu mentioned this pull request Nov 26, 2024

Standardize on cargo-c for building rustls-ffi, CMake for building test programs rustls/rustls-ffi#493

Merged

vszakats mentioned this pull request Jan 26, 2025

cmake/Find: add iphlpapi for c-ares, omit syslibs if dep not found #16089

Closed

pps83 pushed a commit to pps83/curl that referenced this pull request Apr 26, 2025

cmake: mark as advanced some internal Find* variables

3a64707

To sync with other similar variables and hide them from cmake UIs. Follow-up to 7bab201 curl#15193 Closes curl#15407

Navigation Menu

Uh oh!

Uh oh!

cmake: add native `pkg-config` detection for mbedTLS, MSH3, Quiche, Rustls, wolfSSL - #15193

cmake: add native `pkg-config` detection for mbedTLS, MSH3, Quiche, Rustls, wolfSSL#15193
vszakats wants to merge 21 commits into
curl:masterfrom
vszakats:cm-find-pkgconf-2

vszakats commented Oct 8, 2024 •

edited

Loading

Uh oh!

dfandrich commented Oct 8, 2024

Uh oh!

vszakats commented Oct 9, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

vszakats commented Oct 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dfandrich commented Oct 8, 2024

Generated by Testclutch

Uh oh!

vszakats commented Oct 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

vszakats commented Oct 8, 2024 •

edited

Loading

vszakats commented Oct 9, 2024 •

edited

Loading