[libarchive] Build failure on x64-linux (static only)

[chrismile]

Operating system

Ubuntu 20.04

Compiler

GCC

Steps to reproduce the behavior

I have set up a test repo for reproducing this issue with GitHub actions: https://github.com/chrismile/ActionsTestRepo2

Compilation of libarchive only fails for the static build due to undefined references to, e.g., `dlopen'. This is a regression, as this issue did not appear in commit 2a6371b.

Disclaimer: Due to the ongoing issues with liblzma (#37839), I needed to set a different mirror.

Failure logs

out.log

Additional context

None.

[SchaichAlonso]

You can use vcpkg-settings.json to add an overlay path with your customized triplet definitions. What a triplet does is more obvious if it can be reviewed in the repository rather then having to guess what some shell magic did to the vcpkg shipped triplet files.

Does your problem reproduce with the vcpkg-bundled x64-linux triplet, or does it only reproduce with your custom one?

Also, a consumer CMakeList.txt doesn't need to include the triplet files explicitly, because the vcpkg toolchain processes them before the consumer CMakeLists.txt is processed.

[chrismile]

@SchaichAlonso I apoologize for the hard to understand shell syntax.
Luckily, the default triplet also uses static linking on Linux, so it seems it is reproducible without any custom triplets (I mainly used them for testing with which configurations it is triggered). Here's a severely simplified test case in a separate repo: https://github.com/chrismile/ActionsTestRepo4/blob/main/.github/workflows/build-libarchive.yml

This uses only the default triplet. The fork of vcpkg is used due to the current issues with liblzma and includes a mirror for it.

[JonLiu1993]

Could you send me your triplet file? I want to confirm the specific content of the x64-linux-static-release-onlytriplet you installed.

[chrismile]

Could you send me your triplet file? I want to confirm the specific content of the x64-linux-static-release-onlytriplet you installed.

@JonLiu1993 Thanks for your message. I'm no longer using a custom triplet in the latest test, and the build also fails there (https://github.com/chrismile/ActionsTestRepo4/blob/main/.github/workflows/build-libarchive.yml).

[dg0yt]

/usr/bin/ld: /home/runner/work/ActionsTestRepo/ActionsTestRepo/sgl-repo/build/vcpkg_installed/x64-linux-static-release-only/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o): in function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x17): undefined reference to `dlopen'
/usr/bin/ld: dso_dlfcn.c:(.text+0x2a): undefined reference to `dlsym'
/usr/bin/ld: dso_dlfcn.c:(.text+0x35): undefined reference to `dlclose'

Make sure that you have a proper installation of port openssl which is the provider of libcrypto. That port comes with a wrapper which also takes care of adding dl.
A libarchive port build with trace would show whether the wrapper is invoke correctly.

[dg0yt]

(FTR I can reproduce the error iff I remove port openssl's wrapper.)

[chrismile]

@dg0yt As you are able to reproduce this, is the build with trace still necessary? If yes, should I simply be using --x-cmake-args=--trace-expand?

I'm not able to reproduce the linking issues right now with ./vcpkg install libarchive[bzip2,lz4,lzma,zstd]. But maybe I'm using it wrong, as the output suggests it uses more features than those provided on the command line.

Building libarchive[bzip2,core,crypto,libxml2,lz4,lzma,zstd]:x64-linux@3.7.2...

[cenit]

same issue here

/usr/bin/ld: /path/to/vcpkg/x64-linux-release/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o): in function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x17): undefined reference to `dlopen'
/usr/bin/ld: dso_dlfcn.c:(.text+0x2a): undefined reference to `dlsym'
/usr/bin/ld: dso_dlfcn.c:(.text+0x35): undefined reference to `dlclose'
/usr/bin/ld: /path/to/vcpkg/x64-linux-release/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o): in function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x1cf): undefined reference to `dlsym'
/usr/bin/ld: dso_dlfcn.c:(.text+0x2ce): undefined reference to `dlerror'
/usr/bin/ld: /path/to/vcpkg/x64-linux-release/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o): in function `dlfcn_load':
dso_dlfcn.c:(.text+0x345): undefined reference to `dlopen'
/usr/bin/ld: dso_dlfcn.c:(.text+0x3d0): undefined reference to `dlclose'
/usr/bin/ld: dso_dlfcn.c:(.text+0x401): undefined reference to `dlerror'
/usr/bin/ld: /path/to/vcpkg/x64-linux-release/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o): in function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x4d6): undefined reference to `dladdr'
/usr/bin/ld: dso_dlfcn.c:(.text+0x547): undefined reference to `dlerror'
/usr/bin/ld: /path/to/vcpkg/x64-linux-release/lib/libcrypto.a(libcrypto-lib-dso_dlfcn.o): in function `dlfcn_unload':
dso_dlfcn.c:(.text+0x6d8): undefined reference to `dlclose'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

used to work without any problem in the past (darknet ci pipeline running on github runners)

[dg0yt]

Please test with --trace-expand add to libarchive OPTIONS or via --cmake-args=-DVCPKG_CMAKE_CONFIGURE_OPTIONS=--trace-expand. (We need a trace of project mode, not of script mode.)

[chrismile]

Please test with --trace-expand add to libarchive OPTIONS or via --cmake-args=-DVCPKG_CMAKE_CONFIGURE_OPTIONS=--trace-expand. (We need a trace of project mode, not of script mode.)

Sorry to double-check on this. Where do I need to add this? I can't seem to find any documentation online. What do you mean with "OPTIONS"? And where does --cmake-args=-DVCPKG_CMAKE_CONFIGURE_OPTIONS=--trace-expand need to be passed to? I tried passing it to CMake, but that results in CMake Error: Unknown argument --cmake-args=-DVCPKG_CMAKE_CONFIGURE_OPTIONS=--trace-expand.

[cenit]

i added a

set(VCPKG_CMAKE_CONFIGURE_OPTIONS "--trace-expand")

line to my triplet file and it is doing its job. It's now rebuilding everything as expected, I will post here the expanded trace as soon as i have it

(i know i might have wrapped the line inside a "if port matches libarchive" sentence, but i already have the triplet ready and it's quicker for me to keep this pipeline and just rebuild some artifacts in the meantime)

[cenit]

config-x64-linux-release-trace-out.zip
@dg0yt here it is the config in trace mode. The install log is identical to the original one, with the same error message i copy/pasted above

What shall I look in it to help debug the problem? I didn't find anything "obvious", to be honest for me it's perfectly reproducible always and constantly, so I am wondering how is it possible that CI here does not catch it

[dg0yt]

Well, the problem shouldn't exist because find_package(OpenSSL) would invoke the toolchain's find_package macro which delegates to share/openssl/vcpkg_cmake_wrapper.cmake to do the actual call (_find_package(OpenSSL)). The wrapper would add the required link libraries.

That's why I can simulate the problem when I remove the wrapper.

Now the trace has:

vcpkg/buildtrees/libarchive/src/v3.7.2-2a3da38173.clean/CMakeLists.txt(844):  FIND_PACKAGE(OpenSSL )
vcpkg/scripts/buildsystems/vcpkg.cmake(774):  if(VCPKG_TRACE_FIND_PACKAGE )
vcpkg/scripts/buildsystems/vcpkg.cmake(782):  math(EXPR z_vcpkg_find_package_backup_id 0 + 1 )
vcpkg/scripts/buildsystems/vcpkg.cmake(783):  set(z_vcpkg_find_package_package_name OpenSSL )
vcpkg/scripts/buildsystems/vcpkg.cmake(784):  set(z_vcpkg_find_package_1_ARGN  )
vcpkg/scripts/buildsystems/vcpkg.cmake(785):  set(z_vcpkg_find_package_1_backup_vars  )
vcpkg/scripts/buildsystems/vcpkg.cmake(791):  if(CMAKE_SYSTEM_NAME STREQUAL iOS )
vcpkg/scripts/buildsystems/vcpkg.cmake(799):  string(TOLOWER OpenSSL z_vcpkg_find_package_lowercase_package_name )
vcpkg/scripts/buildsystems/vcpkg.cmake(800):  set(z_vcpkg_find_package_vcpkg_cmake_wrapper_path project/build_release/vcpkg_installed/x64-linux-release-trace/share/openssl/vcpkg-cmake-wrapper.cmake )
vcpkg/scripts/buildsystems/vcpkg.cmake(802):  if(EXISTS project/build_release/vcpkg_installed/x64-linux-release-trace/share/openssl/vcpkg-cmake-wrapper.cmake )
vcpkg/scripts/buildsystems/vcpkg.cmake(814):  elseif(z_vcpkg_find_package_package_name STREQUAL Boost AND EXISTS project/build_release/vcpkg_installed/x64-linux-release-trace/include/boost )

i.e. the toolchain comes to the conclusion that share/openssl/vcpkg_cmake_wrapper.cmake doesn't exist.

Is this true?
If yes, how did we get there? Is it a faulty cached artifact of openssl or a fresh build?

[dg0yt]

BTW is it normal that VCPKG_INSTALLED_DIR is a relative path?

[dg0yt]

All paths are passed as relative ones into CMake in this trace.

[cenit]

i cleaned up paths with a wrong regex and so you don't see them absolute, but they were initially

anyway, there is NO share/openssl/vcpkg_cmake_wrapper.cmake in its folder, just

copyright
usage
vcpkg.spdx.json
vcpkg_abi_info.txt

any idea what's going wrong with the missing file?

[cenit]

i am sorry i cannot really look deeply into it
the fix looks not so obvious!
anyway, as @chrismile clearly described in the opening post, reproducing it is quite easy with a github actions runner that builds libarchive:x64-linux

[dg0yt]

reproducing it is quite easy with a github actions runner

... assuming some familarity with GH actions.

[dg0yt]

any idea what's going wrong with the missing file?

Given the structure of the openssl port and the specific problem, I now suspect that the actual openssl build may remove share/openssl. That is where the wrapper is placed before the build, and everything else is added after the build.

(I simulated this now. Don't ask me for the trigger, maybe a specific version of make, or bad timestamps.)

[cenit]

the problem is still present
i'll try my best to dedicate some time to it, since it's breaking a lot of our ci pipelines.
i am surprised this is not impacting many more people!

[dg0yt]

I didn't find anything strange in the openssl buildsystem.
However, you might try moving the wrapper installation to after the openssl installation.

[yashrajsapra]

I am facing the same issue

[cenit]

with #38398 works for me. Can you confirm?

[chrismile]

with #38398 works for me. Can you confirm?

I can confirm that this fixes the issue.

[yashrajsapra]

Yeah it worked for me also

…

On Fri, Apr 26, 2024, 9:08 AM Christoph Neuhauser ***@***.***> wrote: with #38398 <#38398> works for me. Can you confirm? I can confirm that this fixes the issue. — Reply to this email directly, view it on GitHub <#37987 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVAQDXIXBWVB3E77Q74XHZLY7HD5BAVCNFSM6AAAAABFYUOFKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYGU3TKMRRGI> . You are receiving this because you commented.Message ID: ***@***.***>