pyimport("pyarrow") causes segfault

[jrevels]

This is likely not a problem with PyCall, but more of a setup problem on my end; wondering if you had any insights on a potential culprit.

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i9-7940X CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

julia> ENV["PYTHON"]="/usr/bin/python3.7"
"/usr/bin/python3.7"

julia> using Pkg

julia> Pkg.build("PyCall")
  Building Conda ─→ `~/.julia/packages/Conda/CpuvI/deps/build.log`
  Building PyCall → `~/.julia/packages/PyCall/a5Jd3/deps/build.log`

julia> using PyCall

julia> pyimport("pyarrow")

signal (11): Segmentation fault
in expression starting at no file:0
_ZN4llvm13StringMapImpl15LookupBucketForENS_9StringRefE at /home/jrevels/repos/julia/usr/bin/../lib/libLLVM-6.0.so (unknown line)
unknown function (ip: 0x18)
Allocations: 14260566 (Pool: 14257853; Big: 2713); GC: 31
[1]    13946 segmentation fault  ./julia

Python 3.7 was installed via sudo apt install python3.7-dev (from the deadsnakes PPA), pyarrow 0.13.0 was installed via python3.7 -m pip install pyarrow, and Julia was built from source (also tried the prebuilt Julia 1.1.0 binary, didn't make a difference). The segfault also occurs using a Python 3.6 installation (installed via sudo apt install python3).

This segfault does not occur if...

• ...using macOS 10.14.3, with the same Julia/python/pyarrow versions
• ...PyCall is built using Python 2.7, with the same OS/Julia/pyarrow versions
• ...using pyarrow 0.12.x, with the same OS/Julia/python versions

Thanks!

[stevengj]

Seems like a conflict between an LLVM version used by Arrow and the LLVM version used by Julia itself. See also #476.

[jrevels]

Hmmm...any thoughts on why it's OS specific? I guess there are just platform-specific differences between one/both of the involved LLVM versions (Julia's LLVM 6 fork and the LLVM 7 release Arrow uses) that causes the conflict?

Any ideas for a workaround? Attempted to build Arrow from source using LLVM 6, but it looks like some its dependencies really do require 7. Not even sure if that's the right kind of thing to try though, I'm pretty out of my depth when it comes to build issues 😛

[stevengj]

You would need to build it against the patched LLVM that Julia uses...

[tkf]

Is LLVM a hard requirement for basic arrow functionality? Maybe you can turn it off with ARROW_GANDIVA=off? https://arrow.apache.org/docs/developers/cpp.html#optional-components

[ihnorton]

Hmmm...any thoughts on why it's OS specific?

macOS defaults to RTLD_GLOBAL nevermind, that shouldn't matter here.

[ihnorton]

Any ideas for a workaround?

You could try Libdl.dlopen("/path/to/gandiva.so", Libdl.RTLD_DEEPBIND) before loading pyarrow. (x-ref
JuliaLang/julia#12644 (comment))

[jrevels]

Awesome, that workaround seems to work perfectly. Thanks for the help!

[tkf]

@jrevels It'd be nice if pyarrow do this by default. Can you report it to https://issues.apache.org/jira/browse/ARROW? I think I'll do this myself once I get the same problem. But it seems you already investigated this in detail and have data with multiple environments.