cmake fails on fedora 38 (missing: LUV_INCLUDE_DIR)

[ChrisJStone]

Problem

When attempting to build neovim on Fedora 38 using -DUSE_BUNDLED=OFF as per steps outlined in https://github.com/neovim/neovim/wiki/Building-Neovim I receive the following output

-- The C compiler identification is GNU 13.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found GNU Make at /usr/bin/gmake
-- CMAKE_INSTALL_PREFIX=/usr/local
-- CMAKE_BUILD_TYPE not specified, default is 'Debug'
-- Using Lua interpreter: /usr/bin/luajit
-- Using Lua interpreter for code generation: /usr/bin/luajit
-- Using Lua compiler: /usr/bin/luajit -b -s %s -
-- Could NOT find libuv (missing: libuv_DIR)
-- Looking for dlopen in dl
-- Looking for dlopen in dl - found
-- Looking for kstat_lookup in kstat
-- Looking for kstat_lookup in kstat - not found
-- Looking for kvm_open in kvm
-- Looking for kvm_open in kvm - not found
-- Looking for gethostbyname in nsl
-- Looking for gethostbyname in nsl - not found
-- Looking for perfstat_cpu in perfstat
-- Looking for perfstat_cpu in perfstat - not found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for sendfile in sendfile
-- Looking for sendfile in sendfile - not found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Libuv: /usr/lib64/libuv.so (Required is at least version "1.28.0")
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Luv (missing: LUV_INCLUDE_DIR) (Required is at least version
"1.43.0")
Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
cmake/FindLuv.cmake:12 (find_package_handle_standard_args)
src/nvim/CMakeLists.txt:24 (find_package)

Steps to reproduce

mkdir .deps
cd .deps
cmake ../
cmake ../cmake.deps/ -DUSE_BUNDLED=OFF -DCMAKE_BUILD_TYPE=Release -G Ninja
ninja
cd ..
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -G Ninja

Expected behavior

cmake should complete so I can run ninja to build nvim

Neovim version (nvim -v)

master branch

Vim (not Nvim) behaves the same?

not applicable

Operating system/version

Fedora 38

Terminal name/version

Kitty 0.28.1

$TERM environment variable

xterm-kitty

Installation

build from repo master branch

[ChrisJStone]

I am currently working on a patch. over at https://github.com/CJ-Systems/neovim/tree/NoDepsFedora I have been able to resolve the main issue as cmake is now able to complete. However ninja fails with the following: /usr/bin/ld: cannot find -lluv: No such file or directory
manually editing build.ninja and changing -lluv to /usr/lib64/lua/5.1/luv.so for LINK_LIBRARIES allows the build to complete. I found this "hack" in the spec file for fedoras build of nvim. I'm assuming it's related to the following output I get when I run cmake -- Could NOT find libuv (missing: libuv_DIR)

[dundargoc]

Same problem as #23395 (comment). We can do the same workaround as that issue, but I think it'd be more correct to contact the luv packager for fedora and ask them to build it with SONAME.

[dundargoc]

@cryptomilk are you the luv packager for fedora? Would you mind taking a look at this and see if luv can be built with SONAME?

[cryptomilk]

a) I don't see what the INCLUDE_DIR has to do with the SONAME
b) Fedora packages several versions of luv for several lua implementations
c) What's described by @ChrisJStone in his last comment is a bug in CMake. If you specify a full path to the .so file on the command line like cmake -DLUV_LIBRARY=/usr/lib64/lua/5.1/luv.so it should use the absolute path for linking and not reduce it to -lluv! It worked before so it is a regression in cmake. Please report a bug to cmake!

Also on x86_64 Fedora you want to use luajit2.1-luv-devel.x86_64 ...

[dundargoc]

a) I was referring to OP's latest comment, the warning /usr/bin/ld: cannot find -lluv: No such file or directory. This triggers because the shared library doesn't have a SONAME.
b) Good to know
c) No, I don't think this is true at all. This has been the behavior for a long time. See for example #10407, which is the same problem as this. Pretty sure this is the intended behavior by cmake maintainers as I understand it, and that their perspective is that shared libraries should have SONAME, period. @bradking feel free to correct me or add your perspective.

Also on x86_64 Fedora you want to use luajit2.1-luv-devel.x86_64 ...

I don't understand why this is relevant tbh, did a quick skim of the thread and luajit wasn't mentioned.

[ChrisJStone]

@cryptomilk for me using -DLUV_LIBRARY="path to luv.so" has no effect on the output of cmake. I still have to manually edit the generated build ninja and apply the "hack" I've tried both libjt/2.1/luv.so and lua/5.1/luv.so and still get -lluv in my build.ninja.

While looking through the cmake docs I found cmake policies CMP0060 Link libraries by full path even in implicit directories. and CMP0003 Libraries linked via full path no longer produce linker search paths.. I'm wondering if one of these might apply in this case?

Based on #10407 using -DLUV_LIBRARY:STRING=-Wl,/usr/lib64/lua/5.1/luv.so allows ninja and ld to complete.

[ChrisJStone]

I've submitted #23909 for review. I've considered changing the base Makefile to allow for a fedora option. Not sure it this would be appropriate considering Ubuntu had a similar issue with finding luv.so.

[cryptomilk]

@cryptomilk for me using -DLUV_LIBRARY="path to luv.so" has no effect on the output of cmake. I still have to manually edit the generated build ninja and apply the "hack" I've tried both libjt/2.1/luv.so and lua/5.1/luv.so and still get -lluv in my build.ninja.

CMP0060 states:

For these reasons, CMake 3.3 and above prefer to drop the special case and link libraries by full path even when they are in implicit link directories.

It doesn't look like it is doing that, it converts it to -lluv. Did they miss a case? Looks for me like a CMake bug :-)

[ChrisJStone]

I submitted an issue to cmake. Was able to find link_directories not working correctly in CMake 3.18.0 from two years ago. In that resolution the ordering was backward for add_executable and link_directories so I'm not sure if that would apply here. looking in my pkgconfig there is no file for luv. Could this be the root cause of the issue since I am able to build with -DUSE_BUNDLED=OFF -DUSE_BUNDLED_LUV=ON when I run cmake in .deps?

[dundargoc]

Could this be the root cause of the issue since I am able to build with -DUSE_BUNDLED=OFF -DUSE_BUNDLED_LUV=ON when I run cmake in .deps?

No. I've already specified the root cause, which is that fedora shared libraries doesn't have a SONAME and that cmake can't consistently find shared libraries without a SONAME. See https://gitlab.kitware.com/cmake/cmake/-/issues/22703.

[jamessan]

  mkdir .deps
  cd .deps
  cmake ../
  cmake ../cmake.deps/ -DUSE_BUNDLED=OFF -DCMAKE_BUILD_TYPE=Release -G Ninja
  ninja
  cd ..

Why are you running these steps at all? If you're not going to use the bundled deps, then just build nvim.

  mkdir build
  cd build
  cmake .. -DCMAKE_BUILD_TYPE=Release -G Ninja

FWIW, what I do in Debian is:

  dh_auto_configure -- -G Ninja -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) -DBUSTED_OUTPUT_TYPE=nvim -DPREFER_LUA=$(PREFER_LUA) -DLIBLUV_LIBRARY:FILEPATH=/usr/lib/$(DEB_HOST_MULTIARCH)/lua/5.1/luv.so -DLIBLUV_INCLUDE_DIR:PATH=/usr/include/lua5.1 -DCOMPILE_LUA=$(COMPILE_LUA)

dh_auto_configure just calls cmake with a few standard options for Debian packaging. The relevant part here is specifying the path to luv.so and the header file directory. This works just fine whether I'm building against LuaJIT or PUC Lua.

Neovim isn't dynamically linking against libluv. It's using the luv Lua module, whose loading is handled by Lua's require (or C equivalent), so there's no need for the dynamic linker to find the shared object in the standard library load paths.

No changes are required to Neovim's build system to fix this build issue. #23909 shouldn't be necessary.

[bradking]

[cmake maintainers]' perspective is that shared libraries should have SONAME, period.

If they are meant to be linked using ld and found/loaded at runtime by a dynamic loader, then yes. However, from #23898 (comment):

Neovim isn't dynamically linking against libluv. It's using the luv Lua module

luv.so is not meant to be linked, and thus does not need a SONAME.

I submitted an issue to cmake.

For reference, that is CMake Issue 24973. However, this is not a CMake bug AFAICT.

it converts [/usr/lib64/lua/5.1/luv.so] to -lluv. Did they miss a case? Looks for me like a CMake bug

It is not a bug in CMake. If CMake detects that the library file does not have any SONAME, then it does still convert to -l. Otherwise the linked binary's NEEDED field references the absolute path instead of just the .so file name. See the code here.

However, as discussed above, CMake should not be told to link anything to luv.so in the first place.

[ChrisJStone]

Why are you running these steps at all? If you're not going to use the bundled deps, then just build nvim.

At the time I thought it was needed to perform those steps. I've never claimed to be a developer or expert in programming. Just an old-school true hacker that enjoys figuring things out.

However, as discussed above, CMake should not be told to link anything to luv.so in the first place.

When building ninja from a fresh clone on master using cmake ../cmake.deps/ -DUSE_BUNDLED=OFF -DUSE_BUNDLED_LUV=ON as per the example for DEB 10 then running cmake .. -G Ninija in build the output from a grep -l luv of build.nija is this
LINK_LIBRARIES = -Wl,-rpath,/usr/lib64/lua/5.1: -Wl,--no-undefined /home/cstone/repos/neo/.deps/usr/lib/libluv.a

running a cmake .. -G Ninja from build for no depends with only my fixes in FindLuv.cmake applied as I have to use those other wise cmake fails to find luv at all. The output from grep -i luv is this
LINK_LIBRARIES = -Wl,-rpath,/usr/lib64/lua/5.1: -Wl,--no-undefined -lluv /usr/lib64/libtermkey.so

Clearly in both cases ld is linking to luv either the static in the case of using the bundled depends or shared when using no bundled depends. My initial fix to get this to work as I stated in my update post was to manually edit the generated build.ninja and change the -lluv to path to lib.so as this is the same thing that the package maintainer for neovim on fedora was having to do.

[dundargoc]

@bradking well, as it so happens luv.so does need to be linked for the build to complete. If we remove LUV_LIBRARY related code the build will fail with lines such as

/usr/bin/ld: /home/dundar/programs/neovim/src/nvim/lua/executor.c:576: undefined reference to `luv_set_loop'
/usr/bin/ld: /home/dundar/programs/neovim/src/nvim/lua/executor.c:577: undefined reference to `luv_set_callback'
/usr/bin/ld: /home/dundar/programs/neovim/src/nvim/lua/executor.c:579: undefined reference to `luaopen_luv'

This brings us to the heart of the matter. Many package maintainers seem convinced that SONAME should not be needed. Cmake seem convinced that SONAME is needed. The result is that neovim (and other software) end up needing to mediate this discrepancy. The best workaround so far has been to define the library as an unknown imported library to force cmake to use the full path. I was hoping we (as in fedora, cmake and neovim) could come up with a better long-term solution.

[ChrisJStone]

I've submitted 2212583 for lua-luv to see if we can at least get the fedora maintainer to build with soname.

[jamessan]

Clearly in both cases ld is linking to luv either the static in the case of using the bundled depends or shared when using no bundled depends.

Yes, I was mistaken. I had forgotten that Neovim uses luv directly in the C code, too.

Debian has a common infrastructure for building Lua modules, which completely ignores any upstream buildsystem. :) We set our own SONAME in the resulting objects so multiple builds for different Lua versions can be co-installed. For example, we set SONAME to liblua5.1-luv.so.0 for luv.

I don't think this process is well defined in the general Lua ecosystem, which is why Debian took the approach it did.

[ChrisJStone]

Yes, I was mistaken. I had forgotten that Neovim uses luv directly in the C code, too.

I do apologize if my reply seem overly harsh. I meant no offense.

[jamessan]

I do apologize if my reply seem overly harsh. I meant no offense.

None taken. :) I was confidently wrong and should have double checked what I was stating, especially since I've been a bit out of touch with the project lately.

[ChrisJStone]

New builds of libluv and libluv-devel are now available for fedora rawhide. I can confirm installing libluv and libluv-devel 1.44.2.1-3.fc39 resolves this issue with out the need for any workarounds or hacks.

[dundargoc]

This specific issue may have been solved, but the root problem still exists. It's only a matter of time until a clone of this issue pops up (since it has popped up time and time again before). As I see it there is roughly three different routes we can take:

A) Accept the status quo as is. This will imply that building lua shared libraries as UNKNOWN IMPORTED libraries won't be a workaround, but the canonical way to build all lua shared libraries for the forseeable future. This would without a doubt be the simplest way for neovim, because we can end the discussion there and be done with. Of course, any other project that uses cmake and lua will have to rediscover this fun easter egg on their own.

B) Fix the problem on package maintainers side. This would mean convincing them that they need to add SONAME on all lua shared libraries. Another approach could maybe be that they create duplicates/symlinks, one of them names libluv.so and the other luv.so.

C) Fix the problem on the cmake side. This would mean that we need to convince them that the current situation is a problem and of an appropriate solution/workaround. I have a few ideas in mind, but I am not sure which would be the best. I am thinking one approach would be to expand the fallback mechanism of cmake to search both lib[package].so as well as [package].so. Another could be to add a "use the full path as fallback" flag. There already is one called CMAKE_PLATFORM_USES_PATH_WHEN_NO_SONAME, but it's an undocumented, internal variable that isn't meant to be used, so we'd need to convince them to to make this public.

I think for now option A will suffice, but I'm writing my thoughts when they're still fresh for the next time this problem show up.

[cryptomilk]

You should actually build a libluv.so library which then is used by the luv module. They way you can link against it correctly. That's what I've done in the Fedora package now.

ldd /usr/bin/nvim | grep luv
        libluv.so.1 => /lib64/libluv.so.1 (0x00007efd3a84e000)

[bradking]

If a .so is meant to be linked then it needs a SONAME so that the linker knows what to put in the NEEDED field of the dependent binary for the dynamic loader to find the .so at runtime. If a .so is meant only to be loaded by dlopen then it does not need a SONAME, but may optionally have one. This is a general rule due to the way shared linking and dynamic loading works, and is not a cmake-specific thing.

Otherwise, an imported target with IMPORTED_LOCATION set to an absolute path is the way to tell CMake to link by that absolute path without checking for any SONAME.

@cryptomilk's approach looks like the correct solution to me. It provides both a libluv.so that can be linked, and a luv.so module that can be loaded.

[ChrisJStone]

Ok at this point in time I'm a little confused on if this has been resolved or not at least when building nvim on fedora with no bundled dependencies. Before I submitted a bug report to fedora asking for libluv to be built with soname in my /lib64 there was no reference to libluv and also in /lib64/pkgconfig there was no libluv.pc. Which to my my understanding was what was causing the build to fail. Simply put.

After I had submitted the report and the package maintainer updated libluv and libluv-devel I installed the packages for fc39 / rawhide as I had not received this update yet.
After installation in my /lib64 I now have libluv.so libluv.so.1 libluv.so.1.44.2 present and in /lib64/pkgconfig libluv.pc is present.

I performed a fresh clone of nvim and attemted a cmake .. -DCMAKE_BUILD_TYPE=Release -G Ninja. When I performed a grep -l luv.so On LINK_LIBRARIES /usr/lib64/libluv.so was present. running ninja was able to bulid nvim. The only "error" that I was still receiving before updating my libluv packages and after was Could NOT find libuv (missing: libuv_DIR)

In reference to #23938 my understanding is the OS-specific code for Ubuntu was added so Github Actions could be used based on this comment on my pr #23909

[dundargoc]

@ChrisJStone if neovim builds fine for you then you can consider your issue solved, I just hijacked this issue to be a discussion about a larger problem.