ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

[fillest]

Ubuntu 14 here, pip==7.1.2, setuptools==18.3.2, virtualenv==13.1.2.
First, I want to build latest stable (seems it's 0.9.2) librdkafka into /opt/librdkafka.

curl https://codeload.github.com/edenhill/librdkafka/tar.gz/v0.9.2 | tar xzf - -C /tmp/
cd /tmp/librdkafka-0.9.2/
./configure --prefix=/opt/librdkafka
make
sudo make install
cd -
rm -r /tmp/librdkafka-0.9.2

I have a virtualenv and want to install latest stable confluent-kafka.

source venv/bin/activate
$ C_INCLUDE_PATH=/opt/librdkafka/include LIBRARY_PATH=/opt/librdkafka/lib pip install confluent-kafka
Collecting confluent-kafka
  Using cached confluent-kafka-0.9.2.tar.gz
Building wheels for collected packages: confluent-kafka
  Running setup.py bdist_wheel for confluent-kafka
  Stored in directory: /home/f/.cache/pip/wheels/blabla
Successfully built confluent-kafka
Installing collected packages: confluent-kafka
Successfully installed confluent-kafka-0.9.2

But then a test script fails:

$ python ktest.py 
Traceback (most recent call last):
  File "ktest.py", line 1, in <module>
    from confluent_kafka import Producer
  File "/home/bla/proj/bla/venv/local/lib/python2.7/site-packages/confluent_kafka/__init__.py", line 2, in <module>
    from .cimpl import *
ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

Am I doing something wrong?

[fillest]

--prefix=/usr/local gives the same result.
--prefix=/usr works but it is problematic to interfere with system-wide folders (that's why I wanted to use /opt).

This issue may have something in common with #45. Also if setup.py will provide a way to automatically install librdkafka locally, this issue will become a non-issue (at least for me) as of course I would prefer such way.

[edenhill]

If you install librdkafka to a location outside the standard ld.so paths, such as /usr/local, you will need to provide a hint to the runtime linker where to find it and you typically do this with the environment variable LD_LIBRARY_PATH=/usr/local/lib ./app.py

[edenhill]

Or preferably: you could install a librdkafka RPM, Deb or Homebrew package (depending on your platform).

[fillest]

LD_LIBRARY_PATH is inconvenient cause it has to be set each time on program invocation.
Maintained system packages are old. Building a package yourself is extra hassle and it will conflict with maintained package anyway if you need to install it too (e.g. for syslog).

After an hour of fighting with pip I've came up with a solution: this can be just put in requirement file confluent-kafka==0.9.2 --global-option=build_ext --global-option="--rpath=/opt/librdkafka/lib/". It prevents building a wheel but I don't see a problem with this.
I think it's worth documenting.

[fillest]

Also according to https://pip.pypa.io/en/stable/reference/pip_install/#per-requirement-overrides it requires pip>=7.0

[fillest]

It prevents building a wheel but I don't see a problem with this.

Arrrgh, it turned out to be a problem. After fighting with pip for another couple of hours here's a working receipt to sum it up:

to be able to put it into requirements.txt:

• pip>=7.0 (https://pip.pypa.io/en/stable/reference/pip_install/#per-requirement-overrides)
• confluent-kafka==0.9.2 --global-option=build_ext --global-option="--rpath=/opt/librdkafka/lib" --global-option="--library-dirs=/opt/librdkafka/lib" --global-option="--include-dirs=/opt/librdkafka/include"
• this requirement should go to a separate file and pip install -r should be invoked separately for this file, otherwise the using of wheels would be prevented for all requirements in the file (see --global-option for a specific requirement in requirements file disables use of wheels globally pypa/pip#4118)

[edenhill]

That looks like a good solution, the problem is that it hardcodes the librdkafka library path, which might be /opt on some systems, /usr/local or ~/hi/there on others.

We might want to consider linking librdkafka statically by default, so the path setup is only needed on build, and providing an option to do dynamic linking (for people that want to be able to just upgrade the C library - which is the preferred method from an ops perspective)

[mentholi]

This is oldish issue but I just happened to run in to this so I thought that I share my solution to help others.

I did build and install librdkafka from git checkout and installed confluent-kafka after that with pip. First everything seemed to be okay but when I tried to import something I also got that ImportError. To fix that I needed to run ldconfig and after that everything seems to be working.

Here are the steps:

1. git clone https://github.com/edenhill/librdkafka && cd librdkafka && git checkout <commit_sha>

2. cd librdkafka && ./configure && make && make install && ldconfig

After that I'm able to import stuff without specifying LD_LIBRARY_PATH.

[lucilecoutouly]

thanks a lot for

ldconfig

[edenhill]

We'll keep this issue open so that new users can find it easily.

[rnpridgeon]

@edenhill Does it make sense to add a note in the README with a link to this issue for additional details?

[edenhill]

@rnpridgeon I don't think we need to since the binary wheels probably solves this for most users.

[rnpridgeon]

👍 added 'FAQ' tag to mark informative issues

[rnpridgeon]

Closing since binary wheels where added with 0.11.4 which should solve this in most cases. It will continue to be archived with the FAQ tag under closed issues.

[suvendurinku]

This how to build in DOCKER to resolve this

WORKDIR /opt/
RUN curl https://codeload.github.com/edenhill/librdkafka/tar.gz/v0.11.5-PRE1 | tar xzf -
WORKDIR /opt/librdkafka-0.11.5-PRE1/
RUN ./configure --prefix=/usr
RUN make -j
RUN make install

WORKDIR /
RUN pip3 install --no-binary :all: confluent-kafka

RUN apt-get install -y git-core
WORKDIR /opt/
RUN git clone https://github.com/confluentinc/confluent-kafka-python.git
WORKDIR /opt/confluent-kafka-python/
RUN python setup.py build

[tpschmidt]

I'm also stuck at this in some way, getting the librdkafka.so.1: cannot open shared object file.
I'm trying to package my consumer & producer into a zip file with all needed dependencies so I can deploy it to s3 and run it on AWS lambda.

If I do it via an EC2 instance (with pipenv), this is working fine and it get the librdkafka inside site-packages/confluent_kafka/.libs & it's starting on lambda without issues.

If I package this inside inside my running jenkins container (alpine based), which also got the librdkafka installed (like suvendurinku described), the site-packages/confluent_kafka/.libs is missing completely, so it's obvious it can't find librdkafka anywhere at runtime.

Is there something I'm missing?

[edenhill]

@tpschmidt Make sure to use the binary wheels of confluent-kafka as they come with a prebaked librdkafka.

[dhgokul]

thanks a lot for

ldconfig

Thanks lot, it's save my day

[Chinmaya910]

2. && ldconfig
How to use the binary wheels of confluent-kafka