embedding julia in .so assert: jl_TargetMachine

[Orbots]

Summary of situation:
built julia-1.2.0-rc2 from source for centos6 with gcc 6.4.0 and -DGLIBCXX_USE_CXX11_ABI=0
dlopen an .so which embeds some simple julia code. “Lifted” into global namespace with dlopen(RTLD_NOW | RTLD_GLOBAL)
from a maya plugin.

jl_init() results in this assert:
void* jl_init_llvm(): Assertion `jl_TargetMachine && “Failed to select target machine -” " Is the LLVM backend for this CPU enabled?"’ failed.

Any ideas? A toy julia embedding works fine ( not loaded from maya )

versioninfo():
Julia Version 1.2.0-rc2.0
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)

[yuyichao]

Please don't double post without waiting for a reply and without any cross referencing.

https://discourse.julialang.org/t/assertion-jl-targetmachine/26483

This is likely due to linking issues especially when you are loading into a process that's using llvm.

[Orbots]

readelf -d path to maya/lib/*.so | grep LLVM
readelf -d path to maya/lib | grep LLVM
no hits.

[Orbots]

Here's few relevant variables printed from gdb just before the offending call.

TheTriple = {Data = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x203f3638 "x86_64--linux-gnu"}}, Arch = llvm::Triple::x86_64, SubArch = llvm::Triple::NoSubArch, Vendor
= llvm::Triple::UnknownVendor, OS = llvm::Triple::Linux, Environment = llvm::Triple::GNU, ObjectFormat = llvm::Triple::ELF}

TheCPU = (std::basic_string<char, std::char_traits, std::allocator > &) @0x7fff5009cf30: {static npos = 18446744073709551615, _M_d
ataplus = {<std::allocator> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x1eeb84d8 "broadwell
"}}

targetFeatures = {<llvm::SmallVectorImpl<std::basic_string<char, std::char_traits, std::allocator > >> = {<llvm::SmallVectorTemplateBase<s
td::basic_string<char, std::char_traits, std::allocator >, false>> = {<llvm::SmallVectorTemplateCommon<std::basic_string<char,
std::char_traits, std::allocator >, void>> = {llvm::SmallVectorBase = {BeginX = 0x203f4240, EndX = 0x203f43e8, CapacityX = 0x
203f43e8}, FirstEl = {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\000\000\000\000L\000\000"}, }}, }, <
No data fields>}, Storage = {InlineElts = {{<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\000.? \000\000\000"}, }, {<llv
m::AlignedCharArray<8ul, 8ul>> = {buffer = "\200\261\272U\225\177\000"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer =
"/dev/pts"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "/1\000\237\066\000\000"}, }, {<llvm::Alig
nedCharArray<8ul, 8ul>> = {buffer = "\000U\000\000\005\000\000"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\277
000\000\000;\212\000"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\000\003\034\177\025\004\000\001"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\000\021\023\032\000\022\017\027"}, }, {<llvm::AlignedCharArray<8u
l, 8ul>> = {buffer = "\026\000\000\000\000\000\000"}, }}}}

[Orbots]

Here are the same variables from a toy executable dlopening the same libjulia, which works. jl_TargetMachine is non-null.

TheTriple = {Data = {static npos = , _M_dataplus = {<std::allocator> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x72bb08 "x86_64--linux-gnu"}}, Arch = llvm::Triple::x86_64, SubArch = llvm::Triple::NoSubArch, Vendor = llvm::Triple::UnknownVendor, OS = llvm::Triple::Linux,Environment = llvm::Triple::GNU, ObjectFormat = llvm::Triple::ELF}

TheCPU = (std::basic_string<char, std::char_traits, std::allocator > &) @0x7fffffffc390: {static npos= , _M_dataplus = {<std::allocator> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x688228 "broadwell"}}

targetFeatures = {<llvm::SmallVectorImpl<std::basic_string<char, std::char_traits, std::allocator > >> = {<ll
vm::SmallVectorTemplateBase<std::basic_string<char, std::char_traits, std::allocator >, false>> =
{<llvm::SmallVectorTemplateCommon<std::basic_string<char, std::char_traits, std::allocator >, vo
id>> = {llvm::SmallVectorBase = {BeginX = 0x72c6c0, EndX = 0x72c868, CapacityX = 0x72c868}, FirstEl = {<llv
m::AlignedCharArray<8ul, 8ul>> = {buffer = "\000\000\000\000\001\000\000"}, }}, }, }, Storage = {InlineElts = {{<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\000\231r
000\000\000\000"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "\025]\357\366\377\177
000"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "/dev/pts"}, }, {<l
lvm::AlignedCharArray<8ul, 8ul>> = {buffer = "/9\000\237\066\000\000"}, }, {<llvm::AlignedCha
rArray<8ul, 8ul>> = {buffer = "\000\005\000\000\005\000\000"}, }, {<llvm::AlignedCharArray<8u
l, 8ul>> = {buffer = "\277\000\000\000;\212\000"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {
buffer = "\000\003\034\177\025\004\000\001"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffe
r = "\000\021\023\032\000\022\017\027"}, }, {<llvm::AlignedCharArray<8ul, 8ul>> = {buffer = "
\026\000\000\000\000\000\000"}, }}}}

[yuyichao]

These variables are not useful. It's pretty clear that it's the code that you called that's causing the problem and you still need to figure out which version of LLVM it is calling, or if there's any possibly of symbol mixing.

[Orbots]

What do you mean by the code that I called? I called jl_init() in both cases. In both cases jl_init() was from the same libjulia.so. Perhaps you mean the code that called me?

I've ruled out llvm being used by maya to a "depth" of 1 ( direct library dependencies ) and somewhat to the next level. I can wade through the spew of LD_DEBUG although this is getting out of my depth.

Hey. Maybe you can help? \s
Could you guide me on how to figure out which version of LLVM it's calling and how to rule out the possibility of symbol mixing?

[yuyichao]

By "code you called" I mean "which version of LLVM it is calling". No I do not mean the code that calls you.

Hey. Maybe you can help? \s

I've said everything I could based on the information you provide but most of what you said are in some "summary" that doesn't include enough detail for me to judge whether I can trust what you says.

Examples,

built julia-1.2.0-rc2 from source for centos6 with gcc 6.4.0 and -DGLIBCXX_USE_CXX11_ABI=0

What build parameters? In particular what LLVM?

readelf -d path to maya/lib/*.so | grep LLVM
readelf -d path to maya/lib | grep LLVM

Not exactly sure what you are trying to say here since LLVM doesn't have to appear anywhere here (assuming this is where maya keeps it's library). Your graphic drivers could use LLVM and code could statically link LLVM.

I've ruled out llvm being used by maya to a "depth" of 1 ( direct library dependencies ) and somewhat to the next level.

I'm not sure how you come to this conclusion. If it's from the readelf result then I don't think that's true.

Could you guide me on how to figure out which version of LLVM it's calling and how to rule out the possibility of symbol mixing?

OK, this one I assume you at least know how to get started..........
I guess it might not be obvious and indeed I don't think I learnt this from anywhere in particular so that's fair..........
What I meant is just to step into a LLVM call to figure out where the code is. Just look at $pc after stepping into a LLVM call and see where it lands.
You should also be able to figure out what symbols are loaded but I don't remember/know how to do it.

[Orbots]

I ran with
env LD_DEBUG=libs maya
loaded my plugin
after grepping output for llvm

I find four libraries.
10047: 126838: find library=libLLVM-6.0.so [0]; searching
10049: 126838: trying file=path/julia/1.2.0-rc2/lib/julia/tls/x86_64/libLLVM-6.0.so
10050: 126838: trying file=path/julia/1.2.0-rc2/lib/julia/tls/libLLVM-6.0.so
10051: 126838: trying file=path/julia/1.2.0-rc2/lib/julia/x86_64/libLLVM-6.0.so
10052: 126838: trying file=path/julia/1.2.0-rc2/lib/julia/libLLVM-6.0.so
10062: 126838: calling init: path/julia/1.2.0-rc2/lib/julia/libLLVM-6.0.so

Now I have seen this, which I don't call because the executable is maya ( called me ), and my code is a shared library ( maya plugin ). Is this a clue to what is going wrong?
from: https://docs.julialang.org/en/v1/manual/embedding/index.html

JULIA_DEFINE_FAST_TLS() // only define this once, in an executable (not in a shared library) if you want fast code.

I'll check the program counter and see what I see.

[Orbots]

Sorry, that's only one library, the others are where it's searching and not finding.

[Orbots]

after a tedious gdb session checking $pc and info symbol I'm pretty sure there is only one library, one version of LLVM being used. The one included in julia/1.2.0-rc2 that I build from source. libjulia-debug.so -> julia/libLLVM-6.0.so
That's it. No symbol mixing. A single LLVM version.

It ends up failing here:
0x00007fd4a9b082a0 in std::string::assign(char const*, unsigned long)@plt () from path/julia/libLLVM-6.0.so
(gdb)
std::string::assign (this=0x7fff4d769c80, __s=0x7fd4ab16c1a8 "Unable to find target for this triple (no targets are registered)", __n=65)
The triple is included earlier in this thread. Maybe someone who knows about triples and targets can look at this?

Not sure where to go from here. I suppose I can build libLLVM in debug and step in there. Any suggestions?

[c42f]

So, could you compare the triple you get with your toy program to the triple which is used when loaded in maya? Personally I'd be modifying codegen.cpp to start dumping out info like the triple in a readable format rather than (or in addition to) using the debugger.

I think building LLVM in debug mode would be a good step: working a bit on getting your tools / environment right can save hours of confusion in a debugger.

[yuyichao]

The triple should be the same and it's conformed above. In general the input to that function isn't really generated through llvm and that's why I said the dump isn't useful. You can certainly pri~t them to confirm.

That's also why I said it's likely from the llvm side and probably a linking issue since the interaction with llvm is fairly limited up to this point.

[Orbots]

The triples are actually the same:
TheTriple = Data="x86_64--linux-gnu", Arch=llvm::Triple::x86_64, SubArch = llvm::Triple::NoSubArch, Vendor = llvm::Triple::UnknownVendor, OS = llvm::Triple::Linux,Environment = llvm::Triple::GNU, ObjectFormat = llvm::Triple::ELF

So is TheCPU.

targetFeatures look different though. Which are not printed out nicely, so I could do that.

Not sure if there is any internal state in LLVM during the call to the function to map "the triple" or not.

[Orbots]

That's also why I said it's likely from the llvm side and probably a linking issue since the interaction with llvm is fairly limited up to this point.

We run a pretty limited and system, I'm not aware of any other instances of LLVM. Assuming there is only one version of llvm, what other linking errors could arise? In both cases it uses the same llvm and lib though, so it would be some kind of dynamic linking problem.

[c42f]

targetFeatures look different though

Interesting, maybe.

In general the input to that function isn't really generated through llvm and that's why I said the dump isn't useful. You can certainly print them to confirm.

Sure, fair enough (in case it's not obvious to @Orbots, I'd like to point out that @yuyichao knows far more about codegen than myself).

My approach to this kind of thing is kind of simple minded: log the heck out of everything and doggedly verify the assumptions you're making. If you don't know about it already, you may find jl_safe_printf useful for internal contexts where normal IO would be a problem.