Building with LLVM and cling doesn't work

[isuruf]

Cling kernel dies with

: CommandLine Error: Option 'debug-pass' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

cc @SylvainCorlay

[SylvainCorlay]

Hum never seen this.

Note that the only way to get cling with conda is

conda install xeus-cling -c QuantStack -c conda-forge

The conda-forge channel's cling is not functional.

[isuruf]

Yes, that's what I did.

# packages in environment at /home/isuru/miniconda3/envs/cling:
#
# Name                    Version                   Build  Channel
arb                       2.13.0               hb705a9b_1    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.0.2                      py_0    conda-forge
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.10.15           ha4d7672_0    conda-forge
certifi                   2018.10.15            py37_1000    conda-forge
clangdev                  5.0.0                   cling_1  [cling]  QuantStack
cling                     0.5                           7  [cling]  QuantStack
cling-patches             1                             0    conda-forge
cmake                     3.12.3               h011004d_0    conda-forge
cppzmq                    4.3.0                hc4df557_0    conda-forge
cryptopp                  5.6.5                   cling_0  [cling]  QuantStack
curl                      7.61.1               h74213dd_2    conda-forge
decorator                 4.3.0                      py_0    conda-forge
entrypoints               0.2.3                 py37_1002    conda-forge
expat                     2.2.5                hfc679d8_2    conda-forge
gcc-7                     7.2.0                         2    QuantStack
gf2x                      1.1                           0    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
ipykernel                 5.1.0              pyh24bf2e0_0    conda-forge
ipython                   7.0.1           py37h24bf2e0_1000    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
isl                       0.17.1                        1    conda-forge
jedi                      0.13.1                py37_1000    conda-forge
jinja2                    2.10                       py_1    conda-forge
jsonschema                2.6.0                 py37_1002    conda-forge
jupyter_client            5.2.3                      py_1    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
krb5                      1.16.1               hbb41f41_0    conda-forge
libcurl                   7.61.1               hbdb9355_2    conda-forge
libcxx                    5.0.0                         0    conda-forge
libedit                   3.1.20170329         haf1bffa_1    conda-forge
libffi                    3.2.1                hfc679d8_5    conda-forge
libflint                  2.5.2                heb735ee_2    conda-forge
libgcc-7                  7.2.0                         0    QuantStack
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libiconv                  1.15                 h470a237_3    conda-forge
libsodium                 1.0.16               h470a237_1    conda-forge
libssh2                   1.8.0                h5b517e9_2    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
libuv                     1.23.2               h470a237_0    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
llvmdev                   5.0.0                   cling_1  [cling]  QuantStack
markupsafe                1.0              py37h470a237_1    conda-forge
mistune                   0.8.4            py37h470a237_0    conda-forge
mpc                       1.1.0                hb705a9b_6    conda-forge
mpfr                      4.0.1                h16a7912_0    conda-forge
nbconvert                 5.3.1                      py_1    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1                  hfc679d8_1    conda-forge
nlohmann_json             3.1.2                         0    QuantStack
notebook                  5.7.0                 py37_1000    conda-forge
ntl                       10.3.0                        1    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pandoc                    2.3.1                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.3.1                      py_0    conda-forge
pexpect                   4.6.0                 py37_1000    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pip                       18.1                  py37_1000    conda-forge
prometheus_client         0.4.2                      py_0    conda-forge
prompt_toolkit            2.0.6                      py_0    conda-forge
ptyprocess                0.6.0                 py37_1000    conda-forge
pygments                  2.2.0                      py_1    conda-forge
python                    3.7.0                h5001a0f_4    conda-forge
python-dateutil           2.7.3                      py_0    conda-forge
pyzmq                     17.1.2           py37hae99301_1    conda-forge
readline                  7.0                  haf1bffa_1    conda-forge
rhash                     1.3.6                h470a237_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                40.4.3                   py37_0    conda-forge
simplegeneric             0.8.1                      py_1    conda-forge
six                       1.11.0                py37_1001    conda-forge
sqlite                    3.25.2               hb1c47c0_0    conda-forge
system                    5.8                           2  
terminado                 0.8.1                 py37_1001    conda-forge
testpath                  0.4.2                 py37_1000    conda-forge
tk                        8.6.8                ha92aebf_0    conda-forge
tornado                   5.1.1            py37h470a237_0    conda-forge
traitlets                 4.3.2                 py37_1000    conda-forge
util-linux                2.21                          0  
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.32.2                   py37_0    conda-forge
xeus                      0.14.1                  cling_0  [cling]  QuantStack
xeus-cling                0.4.7                         0    QuantStack
xtl                       0.4.16                        0    QuantStack
xz                        5.2.4                h470a237_1    conda-forge
zeromq                    4.2.5                   cling_0  [cling]  QuantStack
zlib                      1.2.11                        0    QuantStack

[isuruf]

According to root-project/cling#201, this error shouldn't happen if symengine links to llvm statically.

[isuruf]

Hmm, looks like libsymengine.so still exports LLVM symbols. If I use a linker version script to strip out the LLVM symbols, it works. (ref https://www.ibm.com/developerworks/aix/library/au-aix-symbol-visibility/index.html)

[isuruf]

Adding -Wl,--exclude-libs,ALL to LDFLAGS works too. These options are gcc specific though.

[certik]

I see, this only happens if symengine is built with LLVM support.

What is the root of the problem? That even if symengine links to LLVM statically, the symengine.so still exports LLVM symbols that then clash with the cling's llvm symbols? Is this a problem with LLVM, that it exports symbols?

What are the possible solutions? One is to strip out the LLVM symbols from symengine.so (either by a linker script or by a compiler dependent linker flag). Is there another solution?

Perhaps using shared LLVM and link to the same version from both cling and symengine?

[isuruf]

What is the root of the problem? That even if symengine links to LLVM statically, the symengine.so still exports LLVM symbols that then clash with the cling's llvm symbols? Is this a problem with LLVM, that it exports symbols?

I think the issue is that LLVM dynamically looks at the symbols loaded in the current process to figure out which LLVM passes to register and since cling and symengine both have the two symbols corresponding to the same pass, that same pass is added twice which results in an LLVM error when the library is loaded.

What are the possible solutions? One is to strip out the LLVM symbols from symengine.so (either by a linker script or by a compiler dependent linker flag). Is there another solution?

Perhaps using shared LLVM and link to the same version from both cling and symengine?

I think you mentioned all of the options available.

[certik]

@SylvainCorlay you have the most experience with cling: what do you think we should do here?

[SylvainCorlay]

I don't know. cc-ing Vassil Vassilev who is a co-author of cling.

@vgvassilev you may be interested in this!

[vgvassilev]

The error usually shows up when you have linked some of the llvm libraries twice: first in an executable and second in a shared library which gets dlopen-ed by the same executable. This causes reinitialization of the global state and leads to similar set of errors.

PS: dlopen-ing libCling with RTLD_LOCAL may help.