Importing conda-forge pyarrow fails

[asfimport]

Possibly related to: https://issues.apache.org/jira/projects/ARROW/issues/ARROW-2770

Steps to reproduce:

$ conda create -n test python=3 pyarrow -c conda-forge -y
$ conda activate test
$ python -c 'import pyarrow'

This gives:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/phillip/miniconda3/envs/py36/lib/python3.6/site-packages/pyarrow/__init__.py", line 47
, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: libboost_system.so.1.65.1: cannot open shared object file: No such file or directory

Downgrading boost to 1.65.1 gives a symbol lookup error:

$ conda install boost-cpp=1.65.1 -y -c conda-forge
$ python -c 'import pyarrow'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/phillip/miniconda3/envs/py36/lib/python3.6/site-packages/pyarrow/__init__.py", line 47
, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: /home/phillip/miniconda3/envs/py36/lib/python3.6/site-packages/pyarrow/../../../libarrow.so.0: undefined symbol: _ZN5boost13match_resultsIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS_9sub_matchISB_EEEE12maybe_assignERKSF_

Installing pyarrow from defaults and importing it works fine.

cc @kszucs @xhochy

Reporter: Phillip Cloud / @cpcloud
Assignee: Uwe Korn / @xhochy

_{Note: This issue was originally created as ARROW-2783. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
On OSX with both python 2.7 and 3.6:

Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/krisz/.miniconda/envs/pina1/lib/python3.6/site-packages/pyarrow/__init__.py", line 47, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: dlopen(/Users/krisz/.miniconda/envs/pina1/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-darwin.so, 2): Symbol not found: __ZNK5boost16re_detail_10650131cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_
  Referenced from: /Users/krisz/.miniconda/envs/pina1/lib/libparquet.1.dylib
  Expected in: /Users/krisz/.miniconda/envs/pina1/lib/libboost_regex.dylib
 in /Users/krisz/.miniconda/envs/pina1/lib/libparquet.1.dylib

Debian stretch produces similar undefined symbol error like Phillip has.

[asfimport]

Phillip Cloud / @cpcloud:

LD_DEBUG=all python -c 'import pyarrow'

shows that libparquet.so is trying to load libboost_regex.so.1.65.1:

     18209:	file=libboost_regex.so.1.65.1 [0];  needed by /home/phillip/miniconda3/envs/py36/lib/python3.6/site-packages/pyarrow/../../../libparquet.so.1 [0]

so this doesn't appear to be an issue with Arrow per se.

[asfimport]

Krisztian Szucs / @kszucs:
@cpcloud with build number 0 it works

conda install parquet-cpp=1.4.0=0 -c conda-forge

[asfimport]

Wes McKinney / @wesm:
I'm confident this is related to the toolchain migration; @xhochy has been furiously moving things along but it has been rather chaotic. I haven't been able to spend much time maintaining the conda-forge PRs

[asfimport]

Phillip Cloud / @cpcloud:
Looks like parquet-cpp is somehow building against libboost=1.65.1: https://circleci.com/gh/conda-forge/parquet-cpp-feedstock/72?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

I'm not sure 1) how libboost is different from boost-cpp or 2) how it's possible to install different versions of these.

[asfimport]

Krisztian Szucs / @kszucs:
I've a bad feeling about this pin

[asfimport]

Phillip Cloud / @cpcloud:
Interestingly, this actually shows up as a build warning:

patchelf: file: /home/conda/feedstock_root/build_artifacts/parquet-cpp_1530403424640/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libparquet.so.1.4.0
    setting rpath to: $ORIGIN/.
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): lib/libarrow.so.0 not found in any packages
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): Needed DSO lib/libboost_regex.so.1.65.1 found in ['libboost']
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): .. but ['libboost'] not in reqs/run, i.e. it is overlinked (likely) or a missing dependency (less likely)
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): did not find - or even know where to look for: /lib64/libc.so.6
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): lib/libstdc++.so.6 not found in any packages
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): lib/libgcc_s.so.1 not found in any packages
WARNING (parquet-cpp,lib/libparquet.so.1.4.0): did not find - or even know where to look for: /lib64/libm.so.6

I'm going to look into trying to make the build fail on such warnings.

[asfimport]

Krisztian Szucs / @kszucs:
@cpcloud locally building parquet-cpp-feedstock with

conda build recipe

picks up the right arrow-cpp version, however running it with

conda build recipe -m .ci_support/osx_.yaml

reproduces the wrong version seen on the online builds.

[asfimport]

Uwe Korn / @xhochy:
Should be fixed now again, will finish the migration with the pyarrow-feedstock now but the chain hopefully works again. Their automated bot changes sadly introduced some silent breakages due to the new way of pinning package versions and specifying windows support. Sadly as always with conda-forge, builds are stuck in a long queue in Travis.

[asfimport]

Uwe Korn / @xhochy:
@kszucs @cpcloud no need to investigate further between, I'm aware of the problems and also know the solutions. Simply the CI is blocking me.

[asfimport]

Krisztian Szucs / @kszucs:
@xhochy Would You please share the solution? :)

[asfimport]

Phillip Cloud / @cpcloud:
@xhochy Thanks for doing that. I too am curious to know what the solution is.

[asfimport]

Uwe Korn / @xhochy:
The solution is to rebuild the chain. The pinning changed to Boost 1.67 and the current release is build with 1.66 but through some magic I think it picks up an older version where we tried to vendor boost.

[asfimport]

Uwe Korn / @xhochy:
So we still depend on conda-forge/pyarrow-feedstock#49

[asfimport]

Uwe Korn / @xhochy:
Vendoring is the wrong term: We had boost statically linked and the parquet-cpp was depening on some boost symbols that were statically linked into arrow-cpp.