Windows CMake build fails with OpenSSL unresolved external symbols #14405
Comments
|
It may be a case of OpenSSL header vs lib confusion. Can you test if either of these settings would fix your build?:
|
|
@vszakats Apologies for the delay. I tested with both settings and the build works. I haven't tried just one or the other. Would you like me to test that? Or test your PR? |
|
@micahsnyder: Thanks! Yes, it would help if you could do a test with only the first setting ( It'd also be useful to see your build output, after you add this code to curl's main foreach(_env IN ITEMS PKG_CONFIG_PATH PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR)
if(DEFINED ENV{${_env}})
message(STATUS "pkg-config ${_env}: '$ENV{${_env}}'")
endif()
endforeach() |
|
@vszakats For powershell it seems the correct way to set the environment variable is After switching to use |
|
@vszakats I tried your suggested change: I also made sure to unset that environment variable. This didn't work: |
|
Thanks for your tests @micahsnyder! May I ask for one more: without changing/clearing I'm interested in the top of the cmake output that may show us if any of these |
|
@vszakats here ya go |
|
Thank you @micahsnyder! What seems to be happening is that your env has the More details could be seen via a verbose build log. Unless there is a good reason to keep the As for curl's side, we're not yet sure what's the best way to allowlist and handle toggling pkg-config lookups. |
|
One more: Just a shot in the dark for configuring OpenSSL: It might work better via |
|
Sounds good. Thanks @vszakats. I don't do as much dev on Windows as I do on Linux these days. To be honest, didn't know that I have I modified the curl CMakeLists to print where it found libidn2 and discovered that it pkg-config somehow found it in In the past we've used ActivePerl to build OpenSSL. But ActiveState has been trying to force users to register accounts and use their development tools in order to obtain ActivePerl. Hence switching to Strawberry Perl. I'll probably work around it by setting @vszakats thank you for your support on this issue. |
|
Ah okay, so Peeking into Strawberry: It comes with a Perl implemenation of Windows can present interesting situations indeed when mixing its many "islands" of quasi package envs. OpenSSL specifically is prone for confusion, in other envs too due when using forks or versions side by side. We use tricks in CI to avoid issues. |
|
Another thought -- it would be nice if we could disable pkg-config. I tried IMO, the worst thing about the C language (beyond memory safety) is that the build systems comb your harddrive to find libraries. |
|
Oh, this works: |
|
@vszakats If you're interested, I built Mussels as a cross-platform solution to automate building C library dependency chains in a way that is as deterministic as possible, while still using each library's build system. https://github.com/Cisco-Talos/Mussels Our recipe for building libcurl is here: https://github.com/Cisco-Talos/clamav-mussels-cookbook/blob/master/recipes/libcurl-8.yaml Time for me to update that recipe with what I've learned today. |
|
@micahsnyder Thanks for the Yeah, Windows having no standards for nothing at all, makes it an anything-goes kind of environment. I wish there was a standard lib format, a universal Windows SDK and some naming conventions for starters. One can dream! You can find my attempt for building curl (and dependencies) reproducibly at: https://github.com/curl/curl-for-win |
|
@vszakats Your curl-for-win project is impressive! Akin to my homegrown Mussels project, Microsoft has "vcpkg": They have a ton of recipes for Windows, and some for other platforms. A couple years ago they made it so you an define your own recipes, which I guess makes Mussels obsolete. 😅 But I'll probably still keep using Mussels. |
|
@micahsnyder Thank you! Yes, vcpkg, we just recently started using it in curl CI. It also happens to be the MSVC env where If reproducibility is a goal, I think for now there is no other option then rolling our own. |
That's ... disappointing. |
|
|
|
@micahsnyder: Can confirm that I like it better than the env |
|
@micahsnyder Regarding patching this issue within curl: would you mind making a test with this patch? (without using any of the other workarounds discussed in this Issue): --- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -959,8 +959,10 @@ if(USE_LIBIDN2)
check_include_file_concat("idn2.h" HAVE_IDN2_H)
endif()
if(NOT HAVE_LIBIDN2 OR NOT HAVE_IDN2_H)
- find_package(PkgConfig QUIET)
- pkg_search_module(LIBIDN2 "libidn2")
+ if(NOT MSVC OR VCPKG_TOOLCHAIN)
+ find_package(PkgConfig QUIET)
+ pkg_search_module(LIBIDN2 "libidn2")
+ endif()
if(LIBIDN2_FOUND)
include_directories(${LIBIDN2_INCLUDE_DIRS})
set(HAVE_LIBIDN2 ON) |
libidn2 is detected by default, which triggers a `pkg-config` detectio attempt by default. This in turn may pick up libidn2 inadvertently from the disk, and append the libidn2 header directory to the include path. This header directory might contain incompatible system and/or component headers, causing confusion and failed builds. Some of these side-effects may be the result of an unknowningly configured (or misconfigured) `pkg-config`. In another reported case, it was hit by the `pkg-config` from Strawberry Perl. Until we investigate the reasons and come up with a technique to avoid these issues, limit `pkg-config` detection to UNIX platforms, like we already do in `Find*` modules. Notice that `-DCURL_USE_LIBSSH=ON`, `-DCURL_USE_GSASL=ON`, and `-DCURL_USE_LIBUV=ON` options continue to have the above side-effects, though these options are disabled by default. Follow-up to f43adc2 #14137 Reported-by: Micah Snyder Fixes #14405 Closes #14408
|
@vszakats sure I'll give it a try tomorrow. |
|
The build with this patch also completed successfully. Idle curiosity - I wonder if vcpkg has some method to prevent pkg-config from picking up undesired packages, like those from Strawberry. |
|
Nice, thanks @micahsnyder! Good question indeed. |
|
Maybe... vcpkg overrides the pkg-config tool with its own, which then comes with its own default config, that doesn't include Strawberry? |
Before this patch, `pkg-config` was used for `UNIX` builds only (with a few exceptions like wolfSSL, libssh, gsasl, libuv). This patch extends `pkg-config` use to all envs except: `MSVC` without vcpkg. Meaning MSVC with vcpkg will now use it. Also mingw on Windows. Also apply the new condition to options where `pkg-config` was used unconditionally (= for all targets). These are: `-DCURL_USE_WOLFSSL=ON`, `-DCURL_USE_LIBSSH=ON`, `-DCURL_USE_GSASL=ON` and `-DCURL_USE_LIBUV=ON` This patch may still cause regressions for cross-builds (e.g. mingw cross-build from Unix) and potentially other cases. If that happens, we recommend using some of these methods to explicitly disable `pkg-config` when using CMake: - CMake option: `-DPKG_CONFIG_EXECUTABLE=` (or `-DPKG_CONFIG_EXECUTABLE=nonexistent` or similar) This is similar to the (curl-specific) `PKG_CONFIG` env for autotools. - export env: `PKG_CONFIG_LIBDIR=` (or `PKG_CONFIG_PATH`, `PKG_CONFIG_SYSROOT_DIR`, or the CMake-specific `PKG_CONFIG`) We may improve control over this in a future patch, also allowing opting in MSVC (without vcpkg). Ref: #14405 Ref: #14408 Ref: #14140 Closes #14483
|
I doubt we can ever be in control of this. Whether pkg-config or not, if two or more lib directories are specified/detected during library search and it turns out that more than one has some library we're going to use (eg OpenSSL) then we can't guarantee which one will be used. It seems arbitrary. Even though we can change or otherwise alter the detection to accommodate one user then some other user may be affected. edit: to clarify, I'm saying like if library foo is detected in -L/x/foo but library bar was specified -L/x/bar but is also in -L/x/foo then bar may be pulled from /x/foo or /x/bar depending on the order passed to the linker. |
I did this
I compiled libcurl 8.9.1 on Windows 11 using cmake with the OpenSSL backend (version 3.3.1, though I also observed the issue with 3.2.0).
Build command:
cmake.exe .. -G "Visual Studio 17 2022" -A x64 -D CMAKE_INSTALL_PREFIX="C:/Users/micasnyd/dev/clamav-deps/x64" -D BUILD_LIBCURL_DOCS=OFF -D BUILD_SHARED_LIBS=ON -D CURL_USE_OPENSSL=ON -D OPENSSL_INCLUDE_DIR="C:/Users/micasnyd/dev/clamav-deps/x64/include" -D LIB_EAY_RELEASE="C:/Users/micasnyd/dev/clamav-deps/x64/lib/libcrypto.lib" -D SSL_EAY_RELEASE="C:/Users/micasnyd/dev/clamav-deps/x64/lib/libssl.lib" -D ZLIB_INCLUDE_DIR="C:/Users/micasnyd/dev/clamav-deps/x64/include" -D ZLIB_LIBRARY_RELEASE="C:/Users/micasnyd/dev/clamav-deps/x64/lib/zlibstatic.lib" -D LIBSSH2_INCLUDE_DIR="C:/Users/micasnyd/dev/clamav-deps/x64/include" -D LIBSSH2_LIBRARY="C:/Users/micasnyd/dev/clamav-deps/x64/lib/libssh2.lib" -D USE_NGHTTP2=ON -D NGHTTP2_INCLUDE_DIR="C:/Users/micasnyd/dev/clamav-deps/x64/include" -D NGHTTP2_LIBRARY="C:/Users/micasnyd/dev/clamav-deps/x64/lib/nghttp2.lib" -D BUILD_CURL_EXE=OFFI encountered a build failure due to unresolved external symbols.
I think the unresolved symbols in question were renamed in OpenSSL 3.x and the original names were converted to macros for backwards compatibility. So I suspect there's a header issue of some sort, and the macros aren't being included.
I checked with previous versions:
I expected the following
build should complete successfully
curl/libcurl version
curl 8.9.0
operating system
Windows 11 x64
The text was updated successfully, but these errors were encountered: