add optional RTLD flags argument to dlopen, fixes issue #2312

[stevengj]

As a more general alternative to my #2317 patch (to fix #2312), this patch makes all dlopen flags available in Julia: it adds an optional argument

dlopen(libfile::String [, flags::Integer])

where the second argument is a bitwise-or of RTLD flags, which defaults to RTLD_LAZY|RTLD_DEEPBIND|RTLD_LOCAL.

So, a library can be opened for global access via dlopen(libfile, RTLD_LAZY|RTLD_DEEPBIND|RTLD_GLOBAL).

Note that if a flag is not supported on the running platform (e.g. Windows), it is simply ignored.

[StefanKarpinski]

I like this one. Will wait for @JeffBezanson to take a look though.

[ViralBShah]

I really like this too.

[stevengj]

(Note, by the way, that RTLD_LOCAL is defined to zero because it is not a "real" flag, just a default if RTLD_GLOBAL is not specified, and the latter takes precedence if both are specified, according to POSIX. glibc also defines it to zero.)

[stevengj]

@JeffBezanson, is this okay to merge, since explicit RTLD flags seemed to be your preference in #2317 as well?

[JeffBezanson]

This is fine, since if you're going to call dlopen you might as well be allowed to pass its flags. But, we still need to think about how this affects static compilation. If you use the normal ccall((:fn, :lib), ...) form, it's not good to have to switch all of those to dlsym on discovering that you need RTLD_GLOBAL when using the dynamic loader.

[Keno]

@JeffBezanson What's your opinion on the add_library_mapping trick I'm using in Cairo/Tk. Too hacky or something worth exposing?

[stevengj]

@JeffBezanson, regarding static compilation, I don't see this as such a big problem:

• If the Julia code is explicitly calling dlopen, you can just compile it to code that still calls dlopen. This will be correct, if possibly suboptimal.
• If the author using dlopen and dlsym wants to do something different when static compiling, it should be pretty straightforward to do so just by defining a macro with ccall and then changing the macro for static compiling. I'll probably be using a macro anyway just to cache the results of dlsym, as suggested by Viral.
• Now that the functionality in dlopen is there, you can always modifiy ccall to take an optional triplet (symbol, libname, flags) where the flags gives flags for the dynamic case, or something like that. However, my suspicion is that it may be better to hold off on this kind of thing until you have a prototype of static compilation working—it might be that additional options are needed for static compilation, so that a more general mechanism will be needed.

(I have to use dlopen anyway because the library name for PyCall is not known until runtime. So I will need some alternative way to specify the library name during static compilation anyway, or to just use dlopen in the compiled code.)

[JeffBezanson]

It is of course a good point that statically-compiled programs can use dlopen, C being a prime example :) We still need to be careful about binding time though. If dlsym is called at compile time and its result is cached, that would probably be the wrong thing and we need a way to call it at program startup instead. dlsym could also return something other than a raw Ptr, containing enough information to recompute the address if necessary.

The add_library_mapping hack is fairly innocent, since it doesn't require rewriting client code. If we decided we didn't like it, we could just delete that function.

All this does seem to ask for a better solution though. We should probably build in the pattern of dlopening at run time and caching the result of dlsym for each call site. Basically some way to specify whether a particular library should be bound at compile time or startup time.