[BUG] CMake's find_library returns incorrect libdl

[neobrain]

Description

In CMake projects, find_library picks up the wrong libdl.a: It returns ndk/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libdl.a instead of ndk/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/24/libdl.so. The former is just a stub implementation (dlopen() always returns NULL), so it seems preferable to use the latter.

Notably, this breaks SDL2 for me, since it relies on dlopen to do anything interesting on Android.

Example CMakeLists.txt:

cmake_minimum_required(VERSION 3.13)
project(dlopen_test)
find_library(DLPATH dl)

# cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake -DANDROID_PLATFORM_LEVEL=24 -DANDROID_PLATFORM=android-24 && cat CMakeCache.txt  | grep DLPATH

I found that reverting the change to CMAKE_SYSTEM_LIBRARY_PATH in 12d79d64f29f5f392183a1d1f763dbdfcebe7919 (i.e. prefixing that variable with the sysroot path again) resolves the issue for me, but I'm not sure this is a proper fix.

Environment Details

• NDK Version: 19.1.5304403 (also tried copying android.toolchain.cmake from git over my NDK)
• Build system: cmake
• Host OS: Linux, openSUSE Tumbleweed
• ABI: Any (e.g. arm64-v8a)
• NDK API level: Any (e.g. 24)

[alexcohn]

AFAIK, there are no stub static libraries in NDK. For me, android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/24/libdl.a and android-ndk-r19b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libdl.a are binary the same.

Normally, CMake will use the shared libdl.so to link your project, and does not rely on find_library(… dl). And yes, NDK ships with a stub for libdl.so, as well as other system libs. This should not cause trouble for you, because this stub should not be part of your deliverables. The real libdl is waiting for your app in /system/lib/.

[neobrain]

@alexcohn Hey, thanks for the quick reply (and sorry I couldn't reply any earlier!)

As you're saying, CMake usually picks up libdl automatically without using find_library. SDL2 itself, however, uses find_library to link against libdl (see https://hg.libsdl.org/SDL/file/3477a301a5a8/CMakeLists.txt#l884). It's this find_library call that returns …/arm-linux-androideabi/libdl.a rather than the shared library …/arm-linux-androideabi/24/libdl.so. (Note that indeed you're right and the two libdl.a files are identical - this was a typo in my top post. The second path should've referred to the shared lib.)

Does this clarify the issue?

[neobrain]

I ran CMake through a debugger and set a breakpoint in the find_library handler. Apparently the list of paths it searches in is:

(gdb) p SearchPaths
$8 = std::vector of length 14, capacity 16 = {
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/local/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/",
  "/opt/android/sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/24/",
  "/usr/local/lib64/", "/usr/local/lib/", "/usr/local/", "/usr/lib64/", "/usr/lib/", "/usr/", "/lib64/", "/lib/", "/opt/"
}

Notice the path that contains the desired libdl.so (.../aarch64-linux-android/24) is at the back at the list, preceded by .../aarch64-linux-android/ (which contains libdl.a).

I haven't been able to figure out where specifically those paths come from, but maybe that gives us some clues as to what's wrong?

[alexcohn]

If you must use find_library() for libdl, one solution would be to use

find_library(DLPATH NAMES libdl.so dl)

This way, CMake will first try to find the shared library, and fall back to whatever matches the target platform if libdl.so cannot be found.

[neobrain]

Cool, I'll make sure to see if that adjusted find_library call will do too.

As for the static library, are you sure libdl.a is not a stub of sorts, though? Running objdump -D libdl.a, this is what I get:

0000000000000000 <dlopen>:
   0:   aa1f03e0        mov     x0, xzr
   4:   d65f03c0        ret

Am I missing some hidden purpose of this placeholder implementation?

[enh]

no, that's expected: Android has never supported the dl* functions from static binaries. they just fail at runtime. you never want libdl.a unless you're trying to build something where you know you don't actually need the dl* functionality but aren't able to actually cut it out of the source.

[alexcohn]

Android has never supported the dl* functions from static binaries. they just fail at runtime.
🤦‍♂️

[neobrain]

@enh Hah, interesting. Thanks for shedding some light on that :)

I can now confirm that adding NAMES libdl.so to the find_library call works. I'll check with the SDL developers to see if this change can be upstreamed.

This works on my end - feel free to close this issue at your discretion. From what you're saying it seems this is a necessary evil that we can't really do anything about in the NDK.

Thanks for your help!

[DanAlbert]

We'll leave it open. That workaround shouldn't be necessary.

[Zingam]

If the workaround should not be necessary what would be the proper solution?

[DanAlbert]

I'm not sure yet because I haven't had a chance to look deeper. I suspect this is a bug that we'll need to fix in the NDK or CMake. The workaround is what you'll need to rely on for now, but I want to leave the bug open so we can fix it properly later.

[DanAlbert]

It seems the issue is that CMake searches CMAKE_SYSROOT first and CMAKE_SYSTEM_LIBRARY_PATH after. CMake doesn't natively understand the NDK sysroot layout (or perhaps it does now, but has the inverted preference for API-generic as opposed to API-specific libraries).

I tried a handful of tweaks and the only way I could get it to prefer the shared library was to stop setting CMAKE_SYSROOT. That might have other search order effects, and if we remove that we'll break compatibility with some versions of Studio (they identify Android targets based on --sysroot pointing to an NDK, rather than based on the Clang target name like they do now). I'll have to dig up when we made that change in AGP to see if it's reasonable to break compatibility. Triaging to r22 to figure that out.

If that doesn't work out, this will need to be fixed in CMake proper. We should teach it about the NDK sysroot layout anyway, but that'll happen naturally as part of #463.

[alexcohn]

Maybe you can simply move libdl.a away from the eyes of find_library? If I understand correctly, this stub is only relevant in very special situations, and those who really need it, can go one extra mile (e.g. find_library(DLPATH NAMES libdl.a dl))

[alexcohn]

Actually, the easiest way I know to link libdl.so is to remove find_library altogether, and write instead something like target_link_libraries(${project} dl). This adds to linker command line flag -ldl and clang happily picks up the correct shared library.

[DanAlbert]

Yeah, I never understood the CMake idiom of using find_library when there's no fallback action. That's definitely the best advice, but I'd still like to get this fixed when we can.

[DanAlbert]

(#1179 is another case of this)

[akoeplinger]

The same happens for libz, using find_package(ZLIB REQUIRED) resolves libz.a instead of the .so

[DanAlbert]

Yep, already tracked. See the previous comment for more info.

[DanAlbert]

Looks like the change that made --sysroot unnecessary was in Studio 3.3, so we'd be breaking compatibility with 3.2 and older. 3.2 made it to stable in September 2018, so I think that's sufficiently old that it's worth dropping support for if it lets us fix a bug. I'll try doing that and see if anything else breaks.

[DanAlbert]

https://android-review.googlesource.com/c/platform/ndk/+/1318770 seems to work, and fortunately we alkready have a number of tests for other find_path, find_library, etc behaviors :)

[DanAlbert]

AGP grew new dependencies on the explicit sysroot argument in 4.0, so the change needs to be reverted. Not sure when we'll be able to fix AGP, but very doubtful that it can be before 4.3.

[DanAlbert]

#1179 (comment)

@huangqinjin mind sending your patch to AOSP? @hhb to confirm, but it looks like a good fix we can use until we finish the rest of the CMake cleanup.

[huangqinjin]

@DanAlbert https://android-review.googlesource.com/c/platform/ndk/+/1486800

[DanAlbert]

Merged for r23. Thanks for the fix, @huangqinjin!

[ViliusSutkus89]

@huangqinjin , thank you for the fix.

I stumbled upon this issue because like you, @akoeplinger and possibly others I've tried to find_package(ZLIB REQUIRED) and it went pretty bad after it returned libz.a (same as issue #1179).

It is my understanding that the fix is merged for NDK r23.
We are still on r22, so here's my workaround for today:

SET(ZLIB_LIBRARY         "${CMAKE_SYSROOT}/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/${ANDROID_NATIVE_API_LEVEL}/libz.so")
SET(ZLIB_LIBRARY_RELEASE "${CMAKE_SYSROOT}/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/${ANDROID_NATIVE_API_LEVEL}/libz.so")
# ...
find_package(ZLIB REQUIRED)

It doesn't work for libdl, because we can't find_package(dl).
We can't find_package(dl), because we have no CMake package helper for libdl, like we do for ZLIB

Quick googling led me to FindDL.cmake by wengxt. No idea if it works, but it's worth to try if you're stuck on this.

SET(DL_LIBRARY         "${CMAKE_SYSROOT}/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/${ANDROID_NATIVE_API_LEVEL}/libdl.so")
SET(DL_LIBRARY_RELEASE "${CMAKE_SYSROOT}/usr/lib/${CMAKE_LIBRARY_ARCHITECTURE}/${ANDROID_NATIVE_API_LEVEL}/libdl.so")

I've noticed some suggestions to target_link_library(whateverProgram dl), but that isn't the CMake way.
CMake has several ways for your things to link against other libraries:

1. target_link_library(whateverProgram z) - -lz is passed to linker and nothing else. Failure to find Zlib will be reported during link stage.
   If that library has a subfolder that you need to put in your include path (for example -I/usr/include/glib-2.0 for GLib) you're on your own to figure it out.

find_library(Z z)
target_link_libraries(whateverProgram ${Z})

You are still on your own to figure out what paths you need to include to get the right headers, but at least you get a failure during CMake configure stage if Zlib is not found.

find_package(ZLIB)
target_link_libraries(whateverProgram ZLIB::ZLIB)

Linking against (properly) imported target solves directory includes, so you don't need to worry about them. Failure happens during CMake configure stage.