Cannot pipenv install confluent-kafka 1.4.0 from source

[vvaradan-isp]

Description

pipenv install confluent-kafka works fine, but PIP_NO_BINARY=confluent-kafka pipenv install confluent-kafka==1.4.0 fails indicating that version 1.4.0 is not available. Max version available is shown to be 1.3.0. Also, PIP_NO_BINARY=confluent-kafka pipenv install confluent-kafka==1.3.0 works fine.

How to reproduce

$ PIP_NO_BINARY=confluent-kafka pipenv install confluent-kafka==1.4.0
Installing confluent-kafka==1.4.0…
Adding confluent-kafka to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock (feeb6a) out of date, updating to (d72df4)…
Locking [dev-packages] dependencies…
✔ Success!
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (feeb6a)!
Installing dependencies from Pipfile.lock (feeb6a)…
An error occurred while installing confluent-kafka==1.4.0 --hash=sha256:11b9d863b5f8e4da24e0497dc527d0bc170f78e5e15b85f4e170946dd6e86247 --hash=sha256:138eb1eb80a26eabbc00acf9d23d049132d2dbe262601ecd5d17c9078ce399e7 --hash=sha256:240be96c88f09ea1b0767875db1b0b4a1f8dd7b68bb7e89fcf6dbab53fd7a620 --hash=sha256:39a354e727d198fb0f644530d33867199d53e7e25384cf6518147897c46404ff --hash=sha256:39b6d05ec01ce98be583b890b2a80c23e3d752773f108f60b39fa6d8d54cfbca --hash=sha256:3f067b85ebc11f2f72367bd129e746ad32737f24cdd25cedf5780a643d36295b --hash=sha256:403cdca6bda6af5d32f66c4b8920b098c8912b3b703d4500fc15c375368aae22 --hash=sha256:4bb577bb8ccb921857fda601b4282936d696f29ea825fafea237bb76b2f5707e --hash=sha256:4ec0fc40eea26d21586e09eea9046efc3f24df78911b14decf36f613568bfa7e --hash=sha256:54b101bfe4b073712cd2d4523ecda56138eb3737f4825dcf3575bb9183334774 --hash=sha256:57f7a2570587fef85ef0a97c11f05ef2e9e9929fd752351974b75e2c8c65e585 --hash=sha256:5bc72cafd9a857618b3ef1f851bc3c9e023182cd74ec3a711d0f335f31ae1803 --hash=sha256:6e99142b37ea58274f42ddd40ecea46520d14cd40689e03ed2dfedb2863c127e --hash=sha256:74c68a1c6c63eaf4056e55b5162e9f56f26fbd872571660ab629195c63b92198 --hash=sha256:79ec9c47ff2f76d5fcac9345b61d2e2cb6a659c96ac74cf0f78e424b06755822 --hash=sha256:80ee541bf779a8bf38dee7b2014350459df527ed98cf68d1777f30e18d307d1e --hash=sha256:9b2493f898e044dcd222ef64ea8594352c2e229f6d150e6640d9aea15d206aef --hash=sha256:9fcf8f2952c265edfa11556add670d7f24f58a835a8d245d4b5bf9747adc1913 --hash=sha256:a35e997cbaba4d28816b1d83255a65796faffbc39b00f414f8b9eb1c9d967a0f --hash=sha256:c58cf16770fb7fa2defab727c57a4cd4f03e083f6e7d7a4494c7bd42ed15ab37 --hash=sha256:cc28538c6e8af7a75adaa68e1f5905ba60a7b24f2bcc239e0d4dc07dcfd72889 --hash=sha256:d454a17697dafe422934fc917e5e905ebd4fb5c682d5bcffc261167748da6ffe --hash=sha256:f5b3e323714ecd824690c85cf8b06fcca47f7f2cdfe636184d8df23ae0e1f31a --hash=sha256:f8a505d22186f416cc8e4d6c7bbca862e961352d0fef2de870bfc841e071c072 --hash=sha256:fe3617035bad6360bf50b52d140c904a772512fae95f12f769b4a8923a2f2e68! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 4/4 — 00:00:01
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1983, in do_install
[pipenv.exceptions.InstallError]:       do_init(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1246, in do_init
[pipenv.exceptions.InstallError]:       do_install_dependencies(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 858, in do_install_dependencies
[pipenv.exceptions.InstallError]:       batch_install(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Looking in indexes: https://pypi.python.org/simple']
[pipenv.exceptions.InstallError]: ['ERROR: Could not find a version that satisfies the requirement confluent-kafka==1.4.0 (from -r /var/folders/h5/ds18pks91694txcsjt16k9lm0000gn/T/pipenv-vjkrm31h-requirements/pipenv-ixir4ei4-requirement.txt (line 1)) (from versions: 0.9.1.1, 0.9.1.2, 0.9.2, 0.9.4, 0.11.0, 0.11.4, 0.11.5, 0.11.6, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.3.0)', 'ERROR: No matching distribution found for confluent-kafka==1.4.0 (from -r /var/folders/h5/ds18pks91694txcsjt16k9lm0000gn/T/pipenv-vjkrm31h-requirements/pipenv-ixir4ei4-requirement.txt (line 1))']

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[witekest]

The source package confluent-kafka-1.4.0.tar.gz hasn't been published at PyPI for the last version.

We're also affected. We use Alpine Linux which doesn't support wheel packages and the library must be built from source package.

[rnpridgeon]

Sorry about that, you should be able to install the sdist now.

https://pypi.org/project/confluent-kafka/#files

[marzetas]

Could be by any chance that the requirement files were not included in the tar file with the source code?

I've tried to install 1.4.0 in an alpine distro, and I'm seeing this error:

Processing ./confluent-kafka-1.4.0.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-14u6lptl/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-14u6lptl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-14u6lptl/pip-egg-info
         cwd: /tmp/pip-req-build-14u6lptl/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-14u6lptl/setup.py", line 12, in <module>
        with open(os.path.join(mod_dir, 'requirements.txt')) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-14u6lptl/confluent_kafka/requirements.txt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

When I downloaded the tar.gz from the url in the previous comment, and unpacked it, I couldn't see any requirements.txt files in it, not in the root dir, not in the tests, avro or docs.

[witekest]

Thanks for the quick fix. Unfortunately I see the same behavior as @marzetas now.

[rnpridgeon]

I see, apparently requirements.txt files are left behind. Standby I'll see what I can do

[rnpridgeon]

Alright created a micro version and put it on test.pypi to ensure it worked.

/librdkafka #  pip install --no-binary :all: -i https://test.pypi.org/simple/ confluent-kafka==v1.4.0.1
Looking in indexes: https://test.pypi.org/simple/
Collecting confluent-kafka==v1.4.0.1
  Downloading https://test-files.pythonhosted.org/packages/5c/64/ead8eb3b01d307d8f5970e578b7c81c5117a82f2650cf2fba974a720f580/confluent-kafka-1.4.0.1.tar.gz (171 kB)
     |████████████████████████████████| 171 kB 2.9 MB/s 
Skipping wheel build for confluent-kafka, due to binaries being disabled for it.
Installing collected packages: confluent-kafka
    Running setup.py install for confluent-kafka ... done
Successfully installed confluent-kafka-1.4.0.1
/librdkafka # 

Prior to opening a PR to get this on the mainline pypi do you mind trying it out as well. I'll see about adding some automated testing for sdist install as we have for wheels as well.

[nicocti]

Working on my side 👍

[nicocti]

However, librdkafka 1.4 is not available in on packages.confluent.io/deb/5.4 yet...

[KeithMnemonic]

Working on my side 👍

not working with how monasca is using it.

There error is still

| ubuntu-bionic | �[0m�[91m ERROR: Traceback (most recent call last):
| ubuntu-bionic | File "", line 1, in
| ubuntu-bionic | File "/tmp/pip-install-vsji3vxo/confluent-kafka/setup.py", line 12, in
| ubuntu-bionic | with open(os.path.join(mod_dir, 'requirements.txt')) as f:
| ubuntu-bionic | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-vsji3vxo/confluent-kafka/confluent_kafka/requirements.txt'
| ubuntu-bionic | ----------------------------------------
| ubuntu-bionic | �[0m�[91mERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-vsji3vxo/confluent-kafka/

[vvaradan-isp]

However, librdkafka 1.4 is not available in on packages.confluent.io/deb/5.4 yet...

@nicocti - I build librdkafka from source by git cloneing the tag 1.4.0 - that works.

ENV LIBRDKAFKA_VERSION v1.4.0
RUN git clone https://github.com/edenhill/librdkafka.git /tmp/.librdkafka && \
    cd /tmp/.librdkafka && \
    git checkout $LIBRDKAFKA_VERSION && \
    ./configure --install-deps && \
    make && \
    make install

[rnpridgeon]

However, librdkafka 1.4 is not available in on packages.confluent.io/deb/5.4 yet...

Librdkafka 1.4 ships with cp 5.5 which should be out any day now.

I too built librdkafka, should have mentioned that.

not working with how monasca is using it.

Just to clarify you are installing 1.4.0.1 from test.pypi right? The error you are seeing seems to imply that distutils is still looking for requirements.txt however I removed all references to the requirements files from setup.py

pip install --no-binary :all: -i https://test.pypi.org/simple/ confluent-kafka==v1.4.0.1

I'll try to get it approved, merged and pushed to pypi propper tomorrow now that others have had success as well.

[marzetas]

Btw, many thanks for the quick turnaround, @rnpridgeon

[KeithMnemonic]

However, librdkafka 1.4 is not available in on packages.confluent.io/deb/5.4 yet...

Librdkafka 1.4 ships with cp 5.5 which should be out any day now.

I too built librdkafka, should have mentioned that.

not working with how monasca is using it.

Just to clarify you are installing 1.4.0.1 from test.pypi right? The error you are seeing seems to imply that distutils is still looking for requirements.txt however I removed all references to the requirements files from setup.py

pip install --no-binary :all: -i https://test.pypi.org/simple/ confluent-kafka==v1.4.0.1

I'll try to get it approved, merged and pushed to pypi propper tomorrow now that others have had success as well.

the openstack CI still uses 1.4.0. once you get 1.4.0.1 released the "constraints" will get updated to 1.4.0.1 and i can retest

[rnpridgeon]

Yes, I'll do it tonght/first thing tomorrow. Going through to see if there are any other small things that can be included.

[KeithMnemonic]

any updates?

[rnpridgeon]

Sorry for the hold up, the packages are up and available. Do note we decided to just go ahead and bump it to v1.4.1 as opposed to v1.4.0.1