Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory #65

Closed
fillest opened this issue Nov 9, 2016 · 17 comments

Comments

Projects
None yet
7 participants
@fillest
Copy link

commented Nov 9, 2016

Ubuntu 14 here, pip==7.1.2, setuptools==18.3.2, virtualenv==13.1.2.
First, I want to build latest stable (seems it's 0.9.2) librdkafka into /opt/librdkafka.

curl https://codeload.github.com/edenhill/librdkafka/tar.gz/v0.9.2 | tar xzf - -C /tmp/
cd /tmp/librdkafka-0.9.2/
./configure --prefix=/opt/librdkafka
make
sudo make install
cd -
rm -r /tmp/librdkafka-0.9.2

I have a virtualenv and want to install latest stable confluent-kafka.

source venv/bin/activate
$ C_INCLUDE_PATH=/opt/librdkafka/include LIBRARY_PATH=/opt/librdkafka/lib pip install confluent-kafka
Collecting confluent-kafka
  Using cached confluent-kafka-0.9.2.tar.gz
Building wheels for collected packages: confluent-kafka
  Running setup.py bdist_wheel for confluent-kafka
  Stored in directory: /home/f/.cache/pip/wheels/blabla
Successfully built confluent-kafka
Installing collected packages: confluent-kafka
Successfully installed confluent-kafka-0.9.2

But then a test script fails:

$ python ktest.py 
Traceback (most recent call last):
  File "ktest.py", line 1, in <module>
    from confluent_kafka import Producer
  File "/home/bla/proj/bla/venv/local/lib/python2.7/site-packages/confluent_kafka/__init__.py", line 2, in <module>
    from .cimpl import *
ImportError: librdkafka.so.1: cannot open shared object file: No such file or directory

Am I doing something wrong?

@fillest

This comment has been minimized.

Copy link
Author

commented Nov 10, 2016

--prefix=/usr/local gives the same result.
--prefix=/usr works but it is problematic to interfere with system-wide folders (that's why I wanted to use /opt).

This issue may have something in common with #45. Also if setup.py will provide a way to automatically install librdkafka locally, this issue will become a non-issue (at least for me) as of course I would prefer such way.

@edenhill

This comment has been minimized.

Copy link
Member

commented Nov 10, 2016

If you install librdkafka to a location outside the standard ld.so paths, such as /usr/local, you will need to provide a hint to the runtime linker where to find it and you typically do this with the environment variable LD_LIBRARY_PATH=/usr/local/lib ./app.py

@edenhill

This comment has been minimized.

Copy link
Member

commented Nov 10, 2016

Or preferably: you could install a librdkafka RPM, Deb or Homebrew package (depending on your platform).

@fillest

This comment has been minimized.

Copy link
Author

commented Nov 16, 2016

LD_LIBRARY_PATH is inconvenient cause it has to be set each time on program invocation.
Maintained system packages are old. Building a package yourself is extra hassle and it will conflict with maintained package anyway if you need to install it too (e.g. for syslog).

After an hour of fighting with pip I've came up with a solution: this can be just put in requirement file confluent-kafka==0.9.2 --global-option=build_ext --global-option="--rpath=/opt/librdkafka/lib/". It prevents building a wheel but I don't see a problem with this.
I think it's worth documenting.

@fillest

This comment has been minimized.

Copy link
Author

commented Nov 16, 2016

@fillest

This comment has been minimized.

Copy link
Author

commented Nov 17, 2016

It prevents building a wheel but I don't see a problem with this.

Arrrgh, it turned out to be a problem. After fighting with pip for another couple of hours here's a working receipt to sum it up:

to be able to put it into requirements.txt:

  • pip>=7.0 (https://pip.pypa.io/en/stable/reference/pip_install/#per-requirement-overrides)
  • confluent-kafka==0.9.2 --global-option=build_ext --global-option="--rpath=/opt/librdkafka/lib" --global-option="--library-dirs=/opt/librdkafka/lib" --global-option="--include-dirs=/opt/librdkafka/include"
  • this requirement should go to a separate file and pip install -r should be invoked separately for this file, otherwise the using of wheels would be prevented for all requirements in the file (see pypa/pip/issues/4118)
@edenhill

This comment has been minimized.

Copy link
Member

commented Nov 25, 2016

That looks like a good solution, the problem is that it hardcodes the librdkafka library path, which might be /opt on some systems, /usr/local or ~/hi/there on others.

We might want to consider linking librdkafka statically by default, so the path setup is only needed on build, and providing an option to do dynamic linking (for people that want to be able to just upgrade the C library - which is the preferred method from an ops perspective)

@mentholi

This comment has been minimized.

Copy link

commented Jan 2, 2017

This is oldish issue but I just happened to run in to this so I thought that I share my solution to help others.

I did build and install librdkafka from git checkout and installed confluent-kafka after that with pip. First everything seemed to be okay but when I tried to import something I also got that ImportError. To fix that I needed to run ldconfig and after that everything seems to be working.

Here are the steps:

  1. git clone https://github.com/edenhill/librdkafka && cd librdkafka && git checkout <commit_sha>

  2. cd librdkafka && ./configure && make && make install && ldconfig

After that I'm able to import stuff without specifying LD_LIBRARY_PATH.

@lucilecoutouly

This comment has been minimized.

Copy link

commented Jan 4, 2017

thanks a lot for

ldconfig
@edenhill

This comment has been minimized.

Copy link
Member

commented Jan 20, 2017

We'll keep this issue open so that new users can find it easily.

@rnpridgeon

This comment has been minimized.

Copy link
Contributor

commented May 17, 2018

@edenhill Does it make sense to add a note in the README with a link to this issue for additional details?

@edenhill

This comment has been minimized.

Copy link
Member

commented May 17, 2018

@rnpridgeon I don't think we need to since the binary wheels probably solves this for most users.

@rnpridgeon

This comment has been minimized.

Copy link
Contributor

commented May 17, 2018

👍 added 'FAQ' tag to mark informative issues

@rnpridgeon rnpridgeon added the FAQ label May 17, 2018

@rnpridgeon

This comment has been minimized.

Copy link
Contributor

commented Jun 29, 2018

Closing since binary wheels where added with 0.11.4 which should solve this in most cases. It will continue to be archived with the FAQ tag under closed issues.

@rnpridgeon rnpridgeon closed this Jun 29, 2018

@suvendurinku

This comment has been minimized.

Copy link

commented Jul 3, 2018

This how to build in DOCKER to resolve this

WORKDIR /opt/
RUN curl https://codeload.github.com/edenhill/librdkafka/tar.gz/v0.11.5-PRE1 | tar xzf -
WORKDIR /opt/librdkafka-0.11.5-PRE1/
RUN ./configure --prefix=/usr
RUN make -j
RUN make install

WORKDIR /
RUN pip3 install --no-binary :all: confluent-kafka

RUN apt-get install -y git-core
WORKDIR /opt/
RUN git clone https://github.com/confluentinc/confluent-kafka-python.git
WORKDIR /opt/confluent-kafka-python/
RUN python setup.py build

@tpschmidt

This comment has been minimized.

Copy link

commented Nov 27, 2018

I'm also stuck at this in some way, getting the librdkafka.so.1: cannot open shared object file.
I'm trying to package my consumer & producer into a zip file with all needed dependencies so I can deploy it to s3 and run it on AWS lambda.

If I do it via an EC2 instance (with pipenv), this is working fine and it get the librdkafka inside site-packages/confluent_kafka/.libs & it's starting on lambda without issues.

If I package this inside inside my running jenkins container (alpine based), which also got the librdkafka installed (like suvendurinku described), the site-packages/confluent_kafka/.libs is missing completely, so it's obvious it can't find librdkafka anywhere at runtime.

Is there something I'm missing?

@edenhill

This comment has been minimized.

Copy link
Member

commented Nov 27, 2018

@tpschmidt Make sure to use the binary wheels of confluent-kafka as they come with a prebaked librdkafka.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.