Missing libdouble-conversion.so.3 when using pyarrow

[JulianWgs]

Hello folks,

I want to install pyarrow as an additional conda dependency. The installation works, but when I try to import it the following error appears: ImportError: libdouble-conversion.so.3: cannot open shared object file: No such file or directory

This does not happen with daskdev/dask.

Below are the commands to reproduce the error and the full console output of the error (I'm sorry for installing s3fs also, but it shouldn't make a difference):

Commands:

sudo docker run -it -e EXTRA_CONDA_PACKAGES="pyarrow s3fs -c conda-forge" daskdev/dask-notebook:2.16.0 bash
python
import pyarrow

Console output:

sudo docker run -it -e EXTRA_CONDA_PACKAGES="pyarrow s3fs -c conda-forge" daskdev/dask-notebook:2.16.0 bash
+ '[' '' ']'
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
no environment.yml
+ '[' 'pyarrow s3fs -c conda-forge' ']'
+ echo 'EXTRA_CONDA_PACKAGES environment variable found.  Installing.'
EXTRA_CONDA_PACKAGES environment variable found.  Installing.
+ /opt/conda/bin/conda install -y pyarrow s3fs -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.7.12
  latest version: 4.8.3

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /opt/conda

  added / updated specs:
    - pyarrow
    - s3fs


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    arrow-cpp-0.13.0           |   py37hdbb9910_4         3.5 MB  conda-forge
    boost-cpp-1.70.0           |       h8e57a91_2        21.1 MB  conda-forge
    boto3-1.13.16              |     pyh9f0ad1d_0          69 KB  conda-forge
    botocore-1.16.16           |     pyh9f0ad1d_0         3.8 MB  conda-forge
    brotli-1.0.7               |    he1b5a44_1001         1.0 MB  conda-forge
    bzip2-1.0.8                |       h516909a_2         396 KB  conda-forge
    docutils-0.15.2            |           py37_0         736 KB  conda-forge
    gflags-2.2.2               |    he1b5a44_1002         175 KB  conda-forge
    glog-0.4.0                 |       h49b9bf7_3         104 KB  conda-forge
    jmespath-0.10.0            |     pyh9f0ad1d_0          21 KB  conda-forge
    libevent-2.1.10            |       h72c5cf5_0         1.3 MB  conda-forge
    libprotobuf-3.7.1          |       h8b12597_0         4.6 MB  conda-forge
    lz4-3.0.2                  |   py37hb076c26_1          43 KB  conda-forge
    lz4-c-1.8.3                |    he1b5a44_1001         187 KB  conda-forge
    parquet-cpp-1.5.1          |                2           3 KB  conda-forge
    pyarrow-0.13.0             |   py37h8b68381_2         2.2 MB  conda-forge
    re2-2019.08.01             |       he6710b0_0         456 KB  defaults
    s3fs-0.4.2                 |             py_0          21 KB  conda-forge
    s3transfer-0.3.3           |   py37hc8dfbb8_1          90 KB  conda-forge
    snappy-1.1.8               |       he1b5a44_1          39 KB  conda-forge
    thrift-cpp-0.12.0          |    hf3afdfd_1004         2.4 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        42.2 MB

The following NEW packages will be INSTALLED:

  arrow-cpp          conda-forge/linux-64::arrow-cpp-0.13.0-py37hdbb9910_4
  boost-cpp          conda-forge/linux-64::boost-cpp-1.70.0-h8e57a91_2
  boto3              conda-forge/noarch::boto3-1.13.16-pyh9f0ad1d_0
  botocore           conda-forge/noarch::botocore-1.16.16-pyh9f0ad1d_0
  brotli             conda-forge/linux-64::brotli-1.0.7-he1b5a44_1001
  bzip2              conda-forge/linux-64::bzip2-1.0.8-h516909a_2
  docutils           conda-forge/linux-64::docutils-0.15.2-py37_0
  gflags             conda-forge/linux-64::gflags-2.2.2-he1b5a44_1002
  glog               conda-forge/linux-64::glog-0.4.0-h49b9bf7_3
  jmespath           conda-forge/noarch::jmespath-0.10.0-pyh9f0ad1d_0
  libevent           conda-forge/linux-64::libevent-2.1.10-h72c5cf5_0
  libprotobuf        conda-forge/linux-64::libprotobuf-3.7.1-h8b12597_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-2
  pyarrow            conda-forge/linux-64::pyarrow-0.13.0-py37h8b68381_2
  re2                pkgs/main/linux-64::re2-2019.08.01-he6710b0_0
  s3fs               conda-forge/noarch::s3fs-0.4.2-py_0
  s3transfer         conda-forge/linux-64::s3transfer-0.3.3-py37hc8dfbb8_1
  snappy             conda-forge/linux-64::snappy-1.1.8-he1b5a44_1
  thrift-cpp         conda-forge/linux-64::thrift-cpp-0.12.0-hf3afdfd_1004

The following packages will be DOWNGRADED:

  lz4                                  3.0.2-py37h5a7ed16_2 --> 3.0.2-py37hb076c26_1
  lz4-c                                    1.9.2-he1b5a44_1 --> 1.8.3-he1b5a44_1001



Downloading and Extracting Packages
arrow-cpp-0.13.0     | 3.5 MB    | ############################################### | 100% 
botocore-1.16.16     | 3.8 MB    | ############################################### | 100% 
s3fs-0.4.2           | 21 KB     | ############################################### | 100% 
lz4-c-1.8.3          | 187 KB    | ############################################### | 100% 
boost-cpp-1.70.0     | 21.1 MB   | ############################################### | 100% 
pyarrow-0.13.0       | 2.2 MB    | ############################################### | 100% 
s3transfer-0.3.3     | 90 KB     | ############################################### | 100% 
libevent-2.1.10      | 1.3 MB    | ############################################### | 100% 
bzip2-1.0.8          | 396 KB    | ############################################### | 100% 
jmespath-0.10.0      | 21 KB     | ############################################### | 100% 
glog-0.4.0           | 104 KB    | ############################################### | 100% 
gflags-2.2.2         | 175 KB    | ############################################### | 100% 
docutils-0.15.2      | 736 KB    | ############################################### | 100% 
boto3-1.13.16        | 69 KB     | ############################################### | 100% 
re2-2019.08.01       | 456 KB    | ############################################### | 100% 
thrift-cpp-0.12.0    | 2.4 MB    | ############################################### | 100% 
lz4-3.0.2            | 43 KB     | ############################################### | 100% 
brotli-1.0.7         | 1.0 MB    | ############################################### | 100% 
snappy-1.1.8         | 39 KB     | ############################################### | 100% 
libprotobuf-3.7.1    | 4.6 MB    | ############################################### | 100% 
parquet-cpp-1.5.1    | 3 KB      | ############################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
+ '[' '' ']'
+ exec bash
jovyan@2df202a75cc8:~$ python
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/pyarrow/__init__.py", line 47, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: libdouble-conversion.so.3: cannot open shared object file: No such file or directory
>>> exit()

[martindurant]

Same reported at apache/arrow#2213 (which itself was already a duplicate of threads on apache-pyarrow and conda-forge). Apparently you need more recent versions, which suggests there is some pinning in the given image which is preventing an upgrade to latest. You could try giving an explicit version for pyarrow in the extra conda packages (0.17.1 current, 0.13.0 in the log above).

[JulianWgs]

This works: sudo docker run -it -e EXTRA_CONDA_PACKAGES="pyarrow=0.17.1 -c conda-forge" daskdev/dask-notebook:2.16.0 bash

Thank you very much for the fast help!