RdKafka_LIBRARY_DIR-NOTFOUND error

[AndreMoukarzel]

I've been trying to compile cppkafka as instructed in the README.md, by running the cmake .. from inside a build folder.

When I ran the command, I get the following message:

-- Build will generate a shared library. Use CPPKAFKA_BUILD_SHARED=0 to perform a static build
-- RdKafkaConfig.cmake not found. Attempting to find module instead...
-- Could NOT find RDKAFKA (missing: RdKafka_LIBRARY_DIR) 
-- Found valid rdkafka version
-- RdKafka module found.
-- Disabling tests because submodule Catch2 isn't checked out
-- Configuring done
CMake Error in src/CMakeLists.txt:
  Target "RdKafka::rdkafka" contains relative path in its
  INTERFACE_LINK_DIRECTORIES:

    "RdKafka_LIBRARY_DIR-NOTFOUND"

I have downloaded librdkafka (version 1.1.0) from it's repository and built it both the "regular" (with . /configure, make and sudo make install) and using cmake, as instructed in librdkafka/packaging/cmake/README.md

A RdKafkaConfig.cmake file was created in my /librdkafka/_cmake_build/generated folder.

I've tried to set RDKAFKA_ROOT or RDKAFKA_DIR to this directory and more top-level directories of the librdkafka folder, but the error persists. I image I just need to use one of the CMake options to make it work, but don't know which or what to set it to...

Thank you in advance for any help.

[accelerated]

@DetriL, you should not point the RDKAFKA_ROOT/DIR at the build folder. You should point them to the install location. If they have been installed in the default locations such as /usr/lib and /usr/include I don't think you need to specify anything as find_package will also search those locations by default.
Also note that RDKAFKA_ROOT should point to the top level install location (under which you have lib/lib64/include) directories and RDKAFKA_DIR should point to the folder where the RdKfkaConfig.cmake is installed which is lib64/cmake/rdkafka/. Either one will work but the RDKAFKA_ROOT is the preferred way.

If you have also built and installed CppKafka correctly, all you actually need to do in your application's CMakeLists.txt:

find_package(CppKafka REQUIRED)
target_link_libraries(<your_target> CppKafka::cppkafka <your other dependencies>)

This should find and pull in RdKafka and Boost packages.

[AndreMoukarzel]

@accelerated thanks for the input!

I've taken to exploring the directories you mentioned to make sure that my RdKafka installation is alright (take into account that I have a standard Ubuntu 16.04 build, so I don't have a lib64 folder).

I found a librdkafka subdirectory under /usr/include/, which I supose is a good sign.

What I DIDN'T find was the RdKafkaConfig.cmake in my /usr/local/lib/cmake (which was were I found a cmake subfolder). Actually, there was no rdkafka subdirectory AT ALL in my cmake subfolder!

I should mention that trying to run CppKafka's cmake with RDKAFKA_ROOT=/usr/include/librdkafka/ or RDKAFKA_DIR=/home/user/rdkafka/librdkafka-1.1.0/_cmake_build/generated/ didn't fix the problem.

Does that mean I have executed rdkafka's cmake build incorrectly?

I'm new to this, so I just followed the instructions in the rdkafka's /librdkafka/packaging/cmake/README.md, which created the RdKafkaConfig.cmake file in the /librdkafka/_cmake_build/generated/ subdirectory. Should I move this files to /usr/local/lib/cmake?

[accelerated]

Well you are using the autotools rdkafka build to install which probably isn't configured to copy any cmake artifacts. You should compile RdKafka with cmake which will also install the cmake config and version files. Alternatively you can install rdkafka dpkg or rpm which will contain the binaries but no cmake config files and use RDKAFKA_ROOT (this will avoid you having to compile rdkafka manually). Btw if your location is /usr/local/lib and /usr/include then I don't think you need to specify RDKAFKA_ROOT at all since these are default locations and cmake will search there by default. In any case your RDKAFKA_ROOT=/usr not /usr/include/librdkafka.

[kerenor23]

Hi, I'm also encountering this error while using all defaults - I did however only installed librdkafka-dev, not the sources. I don't have any librdkafka cmake artifacts, cppKafka finds librdkafka as a module.

Funny thing is, it does find correctly RdKafka_LIBRARY_PATH which is the full path of librdkafka.so, and RdKafka_LIBRARY_DIR is basically just the directory that contains librdkafka.so... kind of weird that it's not found.
Anyways, it causes an error loading the CMake, but no error compiling & linking (probably since I installed librdkafka in the default location, /usr/lib/)

(totally unrelated question - when is the next version expected?)

[AndreMoukarzel]

Setting RDKAFKA_ROOT=/usr seems to have fixed the RdKafka_LIBRARY_DIR-NOTFOUND error in while compiling CppKafka, so the compiling SEEMS correct.

When I try to use CppKafka in a project, though, running the cmake command in my project results in a similar erro than before:

cmake ../
-- Could NOT find RDKAFKA (missing: RdKafka_LIBRARY_DIR) 
-- Found valid rdkafka version
-- Configuring done
CMake Error in CMakeLists.txt:
  Target "CppKafka::cppkafka" contains relative path in its
  INTERFACE_LINK_DIRECTORIES:

    "RdKafka_LIBRARY_DIR-NOTFOUND"


CMake Warning at CMakeLists.txt:10 (add_executable):
  Cannot generate a safe runtime search path for target main because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [librdkafka.so] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /usr/local/lib

  Some of these libraries may not be found correctly.


-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

[kerenor23]

@DetriL
Yes, Looks exactly like what I encountered.
For the record, my installation of librdkafka was purely default, and the library ended up in /usr/lib/x86_64-linux-gnu/librdkafka.so.
FindRdKafka.cmake of cppkafka managed to find the following

-- RdKafka_INCLUDE_DIR = /usr/include
-- RdKafka_LIBNAME = librdkafka.so
-- RdKafka_LIBRARY_PATH = /usr/lib/x86_64-linux-gnu/librdkafka.so

but not RdKafka_LIBRARY_DIR ... Not even when passing -DRDKAFKA_ROOT=/usr

Due to lack of time I made an ugly path in FindRdKafka.cmake to set RdKafka_LIBRARY_DIR if it was not found but RdKafka_LIBRARY_PATH was found. Ugly but working. In case you haven't found a better solution here's what I added-

if(RdKafka_LIBRARY_DIR MATCHES "NOTFOUND")
    if(RdKafka_LIBRARY_PATH MATCHES "${RdKafka_LIBNAME}")
        message(STATUS "RdKafka_LIBRARY_DIR was not found & RdKafka_LIBRARY_PATH was found - resetting RdKafka_LIBRARY_DIR")
        get_filename_component(RdKafka_LIBRARY_DIR ${RdKafka_LIBRARY_PATH} DIRECTORY)
    endif()
endif()

[accelerated]

Will take a look. Thanks @kerenor23 for pointing that out.

[accelerated]

Could @kerenor23 or @DetriL verify if this works? Thanks!

[kerenor23]

@accelerated works for me, thanks

[mfontanini]

@kerenor23 can you try removing this line and this line and see if that fixes it too? While #219 may fix the problem, that's probably not the ideal way to do that. (also, sorry I haven't replied to this issue at all).