Segfault with Root and Julia

[lyon-fnal]

Hi - I'm trying to use Root and Julia with your Root.jl package. I followed the instructions for installing and am using the special build of Julia. If I run your example in the README.md, things work until I do @cxx canvas->SaveAs(pointer("myhist.png")) and then Julia segfaults. The important part of the stack trace seems to be...

[vagrant@localhost ~]$ rjulia run_root.jl

signal (11): Segmentation fault
while loading /home/vagrant/run_root.jl, in expression starting on line 11
_ZN4llvm19SmallPtrSetImplBase9erase_impEPKv at /Users/lyon/Development/julia/julia-centos6/julia/usr/lib/libLLVM-3.9.so (unknown line)
_ZN5clang4Sema30UpdateMarkingForLValueToRValueEPNS_4ExprE at /Users/lyon/Development/julia/julia-centos6/pkgs/v0.6/Cxx/src/../deps/usr/lib/libcxxffi.so (unknown line)
_ZN5clang4Sema23DefaultLvalueConversionEPNS_4ExprE at /Users/lyon/Development/julia/julia-centos6/pkgs/v0.6/Cxx/src/../deps/usr/lib/libcxxffi.so (unknown line)
_ZN5clang4Sema36DefaultFunctionArrayLvalueConversionEPNS_4ExprEb at /Users/lyon/Development/julia/julia-centos6/pkgs/v0.6/Cxx/src/../deps/usr/lib/libcxxffi.so (unknown line)
_ZN5clang4Sema31PerformMemberExprBaseConversionEPNS_4ExprEb at /Users/lyon/Development/julia/julia-centos6/pkgs/v0.6/Cxx/src/../deps/usr/lib/libcxxffi.so (unknown line)
unknown function (ip: 0x7feb60fe7c4e)
_ZN5clang4Sema24BuildMemberReferenceExprEPNS_4ExprENS_8QualTypeENS_14SourceLocationEbRNS_12CXXScopeSpecES4_PNS_9NamedDeclERKNS_19DeclarationNameInfoEPKNS_24TemplateArgumentListInfoEPNS0_26ActOnMemberAccessExtraArgsE at /cvmfs/gm2.opensciencegrid.org/prod7/external/root/v6_06_04b/Linux64bit+2.6-2.12-e10-prof/lib/libCling.so (unknown line)
_ZN5clang4Sema21ActOnMemberAccessExprEPNS_5ScopeEPNS_4ExprENS_14SourceLocationENS_3tok9TokenKindERNS_12CXXScopeSpecES5_RNS_13UnqualifiedIdEPNS_4DeclEb at /cvmfs/gm2.opensciencegrid.org/prod7/external/root/v6_06_04b/Linux64bit+2.6-2.12-e10-prof/lib/libCling.so (unknown line)
# ... a lot more  ...

You'll notice that up until _ZN5clang4Sema31PerformMemberExprBaseConversionEPNS_4ExprEb, functions are being pulled from Root's libCling.so. At that function, it switches to libcxxffi.so from Julia. That function is defined in both libCling.so and libcxxffi.so, so my guess is that the wrong one is being chosen and that eventually leads to the segfault. I imagine that the version of Clang/LLVM in libcxxffi.so is quite a bit newer than what libCling.so can handle.

Any ideas here? I realize this is not an easy problem. My setup is also somewhat complicated...

I'm running centos 6.8 (in a VirtualBox virtual machine on my Mac).
The version of Root is 6.06/04 built with gcc 4.9.3 . This Root is in a special place and LD_LIBRARY_PATH is used to get the shared objects correct. I build Julia, Cxx.jl and Root.jl with this same compiler. My julia versioninfo (from the special executable) is,

julia> versioninfo()
Julia Version 0.6.1
Commit 0d7248e (2017-10-24 22:15 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

Thanks for any advice!

[oschulz]

Hi Adam,

is rjulia an alias for the special Julia binary that ROOT.jl builds?

[lyon-fnal]

Yes - it is an alias. If I run the executable directly,

/Users/lyon/Development/julia/julia-centos6/pkgs/v0.6/ROOT/deps/usr/bin/julia run_root.jl

I get the same segfault.

[oschulz]

Yes, the interplay between the ROOT/Cling LLVM and Julia's LLVM is problematic. I think it's mostly ROOT's fault, but I don't have a deep understanding of the matter. The workaround in ROOT.jl (loading ROOT first) seems to work, mostly - but apparently not in all cases. Long term, I hope the ROOT-team and the Julia-team (e.g. @Keno) will figure out a clean solution. But I would assume that for the moment the Julia team has their hand more than full with Julia v0.7 (and then v1.0, afterwards).

I'm using CentOS-7 with the official CERN ROOT binaries for it and a Julia compiled on the same CentOS - but the official Julia binaries worked for me, too. I can provide Docker and Singularity images, if you're interested.

[lyon-fnal]

Thanks Oliver - indeed I think the problem is that Root has a version of clang/LLVM in its libClang.so and julia is using a different (likely much newer) version. I'm also on a fairly old version of Root. Unfortunately, I have to live within a particular ecosystem.

If you have a singularity image (or better, the bootstrap file so I can see how you made it), that would be great.

I am able to work around this problem ... I need Root to access some data, so I have a little python server that does that and then communicates that data to a Julia client over a unix socket. It's a little crazy, but it works! But would be nice to do all of the work in one script.

Thanks again for your help! I know some of the Root people, so I may be able to find more out about what Root is doing. Will let you know.

[oschulz]

If you have a singularity image (or better, the bootstrap file so I can see how you made it)

We build a Docker image first, and then convert to Singularity. Here's our Docker image build repo. It's a bit of a beast, because it includes CUDA, ROOT, Geant4, etc. - you may want to take a few things out of the Dockerfile (it's structured) and maybe use FROM centos:7 instead of FROM nvidia/cuda:8.0-cudnn5-devel-centos7.

Once you have the Docker image, you convert to Singularity via singularity build or docker2singularity.py (if the image is local).

I can also provide our binary Singularity image, but it's 4GB, let me see where best to upload ... ;-)

[lyon-fnal]

Thanks! Don't worry about uploading the image (though putting it on Dockerhub would be cool!)

[oschulz]

Oh, sorry, I meant uploading the Singularity image. The docker image is on Dockerhub - I just don't build it there, because it would exceed DockerHub's build time limit. You should be able to pull and convert via

sudo singularity build /tmp/tmp.sqsh docker://mppmu/gedet-base:julia06-avx2

(resp. -avx, if your systems don't support avx2). The docker image itself may come in handy if you want to run on Cori via Shifter.

[lyon-fnal]

Thanks again! Your root is quite a bit newer than mine. I'll try your version. .. if you want to close this ticket, that's fine with me. I think the problem is some incompatibility issue on my end.

[oschulz]

I did use ROOT-6.06.08 before, that worked fine, too (sometimes even without the special Julia binary), so I think ROOT-0.06.04 should work too. If you like, just change the ROOT version in my Dockerfile and rebuild, for a quick test.

I'll leave this issue open, for now - I'm not sure it's fixable any time soon, but we can collect more information here. :-)

[lyon-fnal]

I got things to work!!! That is, I can run your Root examples and code that accesses Root libraries without a segfault.

I did three things:

• Used a newer root (v6_08_06)
• Used a newer compiler (gcc 6.3)
• Compiled julia (including its dependencies), Cxx and ROOT with the flag -std=c++14

I think the last item above was the thing that fixed the segfault. I'm using the Root that I get from my experiment and it is compiled with the flag -std=c++14. That means that everything that could possibly link against Root must also have std=c++14.

A long time ago, I was mixing code into an executable where some source was built with -std=c++11 and some without the flag (so it was built as C++98). I got segfaults then too. We traced the problem to having two standard template libraries linked in (one for C++11 and the other for C++98 -- they are not compatible and when you mix code the wrong one can get used causing a segfault -- really hard to debug!).

I'm happy this works! Thanks again for all your help.

[oschulz]

Thanks for the analysis - good to know!

[oschulz]

Quick question, @lyon-fnal - how did you set -std=c++11 , resp. -std=c++14 when building Julia?

I just realized that while the ROOT I use (official CERN CentOS-7 build) isn't built with C++14, it's certainly built with C++11, so maybe I should build Julia in C++11-mode, too. We do see crashes when using ROOT.jl in conjunction with MXNet.jl, I wonder if somethings like this may be (at least part of) the reason.

[lyon-fnal]

Sorry I left out the good part!

When I build Julia I do,

export CC=$(which gcc)
export CXX=$(which g++)
export CXXFLAGS="-std=c++14"
CC=$(which gcc) CXX=$(which g++) CXXFLAGS="-std=c++14" make -j 4

I probably don't need the exports and definitions on the make command line, but it can't hurt and this seems to work. I need to define CC and CXX because those don't come from the system in my environment.

You need to build Cxx.jl and ROOT.jl with the flag too (so long as your Root was built with -std=c++14) ...

cd $JULIA_PKGDIR/v0.6/Cxx/deps
# Edit BuildBootstrap.Makefile and add -std=c++14 to the definition of FLAGS
#   or change c++11 to 14 if c++11 is there
# e.g. FLAGS = -std=c++14 $(CPPFLAGS) $(CFLAGS) $(CXXJL_CPPFLAGS)
julia build.jl

cd $JULIA_PKGDIR/v0.6/ROOT/deps
# Edit Makefile and add std=c++14 to FLAGS
# e.g. FLAGS := -I$(INCDIR_JULIA_H) -I$(INCDIR_UV_H) -I$(INCDIR_SUPPORT_H) -std=c++14
julia build.jl

That should do it.

[lyon-fnal]

I updated the previous comment to fix some typos.