Skip to content
This repository

RFC: move implementation of dlopen into Julia #1166

Closed
wants to merge 1 commit into from

4 participants

Jameson Nash Stefan Karpinski Jeff Bezanson Viral B. Shah
Jameson Nash
Collaborator

This should give expanded flexibility, such as allowing for run-time search paths (currently named Base.DL_LOAD_PATH), should allow issue #949 to be implemented entirely in Julia, etc.

Jameson Nash vtjnash commented on the diff August 15, 2012
base/sysimg.jl
@@ -269,6 +272,7 @@ include("char.jl")
269 272
 include("ascii.jl")
270 273
 include("utf8.jl")
271 274
 include("string.jl")
  275
+include("dlload.jl")
1
Jameson Nash Collaborator
vtjnash added a note August 15, 2012

I put include("dlload.jl") here, so that it will be available for loading PCRE, but is after the very useful, basic string manipulation functions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Stefan Karpinski

I dig it. Anything that can be moved into Julia probably should be. The only exception is the function to fork and exec a child process, which I've actually been meaning to move into C for some time now. But that's for performance reasons: in C you can make sure that you don't stomp on the heap at all, which boots the fork/exec out of the "fast lane" in the kernel. Of course, @JeffBezanson should take a look at it too.

Jeff Bezanson
Owner

Looks good, but there is a big wrinkle. The native dynamic linker is worse than this, and does not have the flexibility to specify libraries per-symbol AFAIK. It only supports symbol names and a global list of libraries to search. I was planning to move towards generating native object files, which would require us to go along with this limitation. We can probably specify libraries with absolute paths, in which case this code could be used to help find those paths. But then objects might have to be compiled on the deployment machine (or a clone of it), which may or may not be ok.

If we want to keep our current linking model, we'd have to do our own symbol resolution by emitting a global variable for each symbol, looking up the address of each at startup, and doing indirect calls through those globals. This strikes me as probably not a good idea.

Viral B. Shah
Owner

@JeffBezanson I couldn't understand what you said. Could you elaborate on "current linking model"? Is it possible to have something midway between what we currently have, and what @vtjnash has provided, so that there is more flexibility in the way we locate and use shared libraries?

Jameson Nash
Collaborator

Jeff, I couldn't follow you either. Are you looking for RTLD_DEEPBIND, which was suggested and merged in a pull request a while back?

Jeff Bezanson
Owner

The point is: we cannot continue using dlopen/dlsym in ccall. This results in call sites specialized for addresses in the current process. To generate object code, ccall needs to see the name of the symbol, and then it will be resolved by the dynamic linker as for any executable or shared object.

So if you really want to call dlopen, this code is perfectly fine. But the compiler will have to use some different mechanism. And we will need some declarative way to specify library dependencies, perhaps as part of a module preamble (along with imports and exports).

Viral B. Shah
Owner

Ok, I understand now. Calling dlopen in ccall is quite handy though, the way it is, and a more powerful dlopen would certainly be nice to have for interactive development, and writing one liner ccalls.

Could we continue supporting things the way they are right now, while requiring the user to do extra work in case they want to compile?

Jeff Bezanson
Owner

I don't like the idea of having two mechanisms. The only advantage of what we do now is that it lets you specify which library each symbol comes from, avoiding collisions and working around the standard library search path. Is that what you like about it, or just the syntax?

Viral B. Shah
Owner
Jeff Bezanson
Owner

Well, we can certainly have concise syntax either way. Stefan and I discussed making the syntax ccall((:func, :lib), ...).

Stefan Karpinski

Yes, the idea was that you'd write one of the following:

ccall(:memcpy, ...)               # in the julia binary's main address space
ccall((:lgamma, :libfdm), ...)    # in the dynamically loaded library "libfdm.so"
ccall((:lgamma, :libfdm, 1), ...) # in the dynamically loaded library "libfdm.1.so"

That is no more verbose than using a variable called "libfdm" and it avoids defining all thes global libfoo variables which then have to be redefined after image generation (because libraries get mapped to different locations in memory).

Jameson Nash vtjnash closed this October 12, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Aug 15, 2012
Jameson Nash move implementation of dlopen into Julia, for expanded flexibility (s…
…uch as runtime search paths)
7f01dcb
This page is out of date. Refresh to see the latest.
6  base/base.jl
@@ -110,7 +110,11 @@ const isimmutable = x->(isa(x,Tuple) || isa(x,Symbol) ||
110 110
 
111 111
 dlsym(hnd, s::String) = ccall(:jl_dlsym, Ptr{Void}, (Ptr{Void}, Ptr{Uint8}), hnd, s)
112 112
 dlsym(hnd, s::Symbol) = ccall(:jl_dlsym, Ptr{Void}, (Ptr{Void}, Ptr{Uint8}), hnd, s)
113  
-dlopen(s::String) = ccall(:jl_load_dynamic_library, Ptr{Void}, (Ptr{Uint8},), s)
  113
+jl_uv_dlopen(s::String, h::Ptr{Void}) = ccall(:jl_uv_dlopen, Int32, (Ptr{Uint8}, Ptr{Void}), s, h)
  114
+function dlclose(handle::Ptr{Void})
  115
+    ccall(:uv_dlclose, Void, (Ptr{Void},), handle)
  116
+    c_free(handle)
  117
+end
114 118
 
115 119
 identity(x) = x
116 120
 
63  base/dlload.jl
... ...
@@ -0,0 +1,63 @@
  1
+
  2
+if CURRENT_OS == :OSX
  3
+    const DL_EXTENSIONS = ["", ".dylib"]
  4
+elseif CURRENT_OS == :Windows
  5
+    const DL_EXTENSIONS = ["", ".dll"]
  6
+else
  7
+    const DL_EXTENSIONS = [".so", ""]
  8
+end
  9
+
  10
+const DL_LOAD_PATH = ["$JULIA_HOME/../lib", ]
  11
+
  12
+function dlopen(s::String)
  13
+    handle = ccall(:malloc, Ptr{Void}, (Uint,), 
  14
+        ccall(:jl_sizeof_uv_lib_t, Uint, ()))
  15
+    statbuf = Array(Uint8, ccall(:jl_sizeof_stat, Int, ()))
  16
+    @windows_only begin
  17
+        if s[2] == ':'
  18
+            errno = jl_uv_dlopen(s, handle)
  19
+            if errno == 0
  20
+                return handle
  21
+            end
  22
+        end
  23
+    end
  24
+    @unix_only begin
  25
+        if s[1] == '/'
  26
+            errno = jl_uv_dlopen(s, handle)
  27
+            if errno == 0
  28
+                ## Can't register finalizers on BitsKinds
  29
+                #finalizer(handle, dlclose)
  30
+                return handle
  31
+            end
  32
+        end
  33
+    end
  34
+    for ext in DL_EXTENSIONS
  35
+        for dir in DL_LOAD_PATH
  36
+            path = "$(dir)/$(s)$(ext)"
  37
+            errno = jl_uv_dlopen(path, handle)
  38
+            if errno == 0
  39
+                return handle
  40
+            else
  41
+                # isfile() won't be defined yet, so we make our own
  42
+                err = ccall(:jl_stat, Int32, (Ptr{Uint8},Ptr{Uint8}), path, statbuf)
  43
+                if (err == 0 && ccall(:jl_stat_mode, Uint, (Ptr{Uint8},), statbuf) & 0xf000 == 0x8000)
  44
+                    msg = "could not load module $s: " * bytestring(ccall(:uv_dlerror, Ptr{Uint8}, (Ptr{Void},), handle))
  45
+                    dlclose(handle)
  46
+                    error(msg)
  47
+                end
  48
+            end
  49
+        end
  50
+        path = "$s$ext"
  51
+        errno = jl_uv_dlopen(path, handle)
  52
+        if errno == 0
  53
+            ## Can't register finalizers on BitsKinds
  54
+            #finalizer(handle, dlclose)
  55
+            return handle
  56
+        end
  57
+    end
  58
+    
  59
+    msg = "could not load module $s: " * bytestring(ccall(:uv_dlerror, Ptr{Uint8}, (Ptr{Void},), handle))
  60
+    dlclose(handle)
  61
+    error(msg)
  62
+end
  63
+
5  base/start_image.jl
... ...
@@ -1,7 +1,10 @@
1 1
 # set up non-serializable state
2 2
 
3 3
 # restore shared library handles
4  
-_jl_lib = ccall(:jl_load_dynamic_library,Ptr{Void},(Ptr{None},),C_NULL)
  4
+_jl_lib = ccall(:malloc, Ptr{Void}, (Uint,), 
  5
+        ccall(:jl_sizeof_uv_lib_t, Uint, ()))
  6
+@assert ccall(:jl_uv_dlopen, Int32, (Ptr{Void}, Ptr{Void}), C_NULL, _jl_lib) == 0
  7
+
5 8
 @unix_only _jl_repl = _jl_lib
6 9
 @windows_only _jl_repl = ccall(:GetModuleHandleA,stdcall,Ptr{Void},(Ptr{Void},),C_NULL)
7 10
 
9  base/sysimg.jl
@@ -235,6 +235,7 @@ include("tuple.jl")
235 235
 include("cell.jl")
236 236
 include("expr.jl")
237 237
 include("error.jl")
  238
+include("osutils.jl")
238 239
 
239 240
 # core numeric operations & types
240 241
 include("bool.jl")
@@ -244,8 +245,10 @@ include("promotion.jl")
244 245
 include("operators.jl")
245 246
 include("pointer.jl")
246 247
 
247  
-_jl_lib = ccall(:jl_load_dynamic_library,Ptr{Void},(Ptr{None},),C_NULL)
248  
-_jl_libfdm = dlopen("libfdm")
  248
+_jl_lib = ccall(:malloc, Ptr{Void}, (Uint,), 
  249
+        ccall(:jl_sizeof_uv_lib_t, Uint, ()))
  250
+ccall(:jl_uv_dlopen, Int32, (Ptr{Void}, Ptr{Void}), C_NULL, _jl_lib)
  251
+_jl_libfdm = ccall(:c_dlopen, Ptr{Void}, (Ptr{Uint8},), "libfdm") # dlopen isn't defined yet
249 252
 
250 253
 include("float.jl")
251 254
 include("reduce.jl")
@@ -269,6 +272,7 @@ include("char.jl")
269 272
 include("ascii.jl")
270 273
 include("utf8.jl")
271 274
 include("string.jl")
  275
+include("dlload.jl")
272 276
 include("regex.jl")
273 277
 include("show.jl")
274 278
 include("grisu.jl")
@@ -282,7 +286,6 @@ include("serialize.jl")
282 286
 include("multi.jl")
283 287
 
284 288
 # system & environment
285  
-include("osutils.jl")
286 289
 include("libc.jl")
287 290
 include("env.jl")
288 291
 include("errno_h.jl")
80  src/dlload.c
@@ -27,29 +27,44 @@ static char *extensions[] = { ".so", "" };
27 27
 #include "julia.h"
28 28
 #include "uv.h"
29 29
 
30  
-#define PATHBUF 512
31  
-
32  
-extern char *julia_home;
  30
+DLLEXPORT size_t jl_sizeof_uv_lib_t() { return sizeof(uv_lib_t); }
33 31
 
34  
-int jl_uv_dlopen(const char* filename, uv_lib_t* lib)
35  
-{
36  
-#ifdef RTLD_DEEPBIND
37  
-    dlerror(); /* Reset error status. */
38  
-    lib->handle = dlopen(filename, RTLD_LAZY|RTLD_DEEPBIND);
39  
-    if (lib->handle) {
40  
-        lib->errmsg = NULL;
  32
+DLLEXPORT int jl_uv_dlopen(const char* filename, uv_lib_t* lib) {
  33
+    lib->errmsg=NULL;
  34
+    if (filename == NULL) {
  35
+#ifdef _WIN32
  36
+        if(!GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
  37
+            (LPCSTR)(&jl_uv_dlopen),
  38
+            &lib->handle))
  39
+            jl_errorf("could not load base module", modname);
  40
+#else
  41
+        lib->handle = dlopen(NULL,RTLD_NOW);
  42
+        assert(lib->handle != NULL);
  43
+#endif
41 44
         return 0;
42 45
     } else {
43  
-        lib->errmsg = strdup(dlerror());
44  
-        return -1;
45  
-    }
  46
+#ifdef RTLD_DEEPBIND
  47
+        dlerror(); /* Reset error status. */
  48
+        lib->handle = dlopen(filename, RTLD_LAZY|RTLD_DEEPBIND);
  49
+        if (lib->handle) {
  50
+            lib->errmsg = NULL;
  51
+            return 0;
  52
+        } else {
  53
+            lib->errmsg = strdup(dlerror());
  54
+            return -1;
  55
+        }
46 56
 #else
47  
-    return uv_dlopen(filename, lib);
  57
+        return uv_dlopen(filename, lib);
48 58
 #endif
  59
+    }
49 60
 }
50 61
 
51  
-uv_lib_t *jl_load_dynamic_library(char *modname)
52  
-{
  62
+#define PATHBUF 512
  63
+extern char *julia_home;
  64
+// minimal boot-strapping dlopen() function, until Julia has enough functions to implement
  65
+// the necessary string and array processing for path and extension searching :O
  66
+// (this is only actually used for loading libfdm)
  67
+DLLEXPORT uv_lib_t *c_dlopen(char *modname) {
53 68
     int error;
54 69
     char *ext;
55 70
     char path[PATHBUF];
@@ -57,26 +72,6 @@ uv_lib_t *jl_load_dynamic_library(char *modname)
57 72
     uv_lib_t *handle=malloc(sizeof(uv_lib_t));
58 73
     handle->errmsg=NULL;
59 74
 
60  
-    if (modname == NULL) {
61  
-#ifdef _WIN32
62  
-        if(!GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
63  
-            (LPCSTR)(&jl_load_dynamic_library),
64  
-            &handle->handle))
65  
-            jl_errorf("could not load base module", modname);
66  
-#else
67  
-        handle->handle = dlopen(NULL,RTLD_NOW);
68  
-#endif
69  
-        goto done;
70  
-    }
71  
-#ifdef _WIN32
72  
-    else if (modname[1] == ':') {
73  
-#else
74  
-    else if (modname[0] == '/') {
75  
-#endif
76  
-        error = jl_uv_dlopen(modname,handle);
77  
-        if (!error) goto done;
78  
-    }
79  
-
80 75
     for(i=0; i < N_EXTENSIONS; i++) {
81 76
         ext = extensions[i];
82 77
         path[0] = '\0';
@@ -86,11 +81,11 @@ uv_lib_t *jl_load_dynamic_library(char *modname)
86 81
                 /* try julia_home/../lib */
87 82
                 snprintf(path, PATHBUF, "%s/../lib/%s%s", julia_home, modname, ext);
88 83
                 error = jl_uv_dlopen(path, handle);
89  
-                if (!error) goto done;
  84
+                if (!error) return handle;
90 85
                 // if file exists but didn't load, show error details
91 86
                 struct stat sbuf;
92 87
                 if (stat(path, &sbuf) != -1) {
93  
-                    //JL_PRINTF(JL_STDERR, "could not load module %s (%d): %s\n", modname, error, uv_dlerror(handle));
  88
+                    //warning: this leaks memory:
94 89
                     jl_errorf("could not load module %s: %s", modname, uv_dlerror(handle));
95 90
                 }
96 91
             }
@@ -98,18 +93,15 @@ uv_lib_t *jl_load_dynamic_library(char *modname)
98 93
         /* try loading from standard library path */
99 94
         snprintf(path, PATHBUF, "%s%s", modname, ext);
100 95
         error = jl_uv_dlopen(path, handle);
101  
-        if (!error) goto done;
  96
+        if (!error) return handle;
102 97
     }
103 98
 
104  
-    //JL_PRINTF(JL_STDERR, "could not load module %s (%d): %s\n", modname, error, uv_dlerror(handle));
  99
+    //warning: this leaks memory:
105 100
     jl_errorf("could not load module %s: %s", modname, uv_dlerror(handle));
106  
-    uv_dlclose(handle);
107  
-    free(handle);
108 101
     return NULL;
109  
-done:
110  
-    return handle;
111 102
 }
112 103
 
  104
+
113 105
 void *jl_dlsym_e(uv_lib_t *handle, char *symbol) {
114 106
     void *ptr;
115 107
     int  error=uv_dlsym(handle, symbol, &ptr);
4  src/init.c
@@ -126,7 +126,9 @@ void julia_init(char *imageFile)
126 126
 {
127 127
     jl_page_size = sysconf(_SC_PAGESIZE);
128 128
     jl_find_stack_bottom();
129  
-    jl_dl_handle = jl_load_dynamic_library(NULL);
  129
+	
  130
+    jl_dl_handle = malloc(sizeof(uv_lib_t));
  131
+	jl_uv_dlopen(NULL, jl_dl_handle);
130 132
 
131 133
 #if defined(__linux__)
132 134
     int ncores = jl_cpu_cores();
10  src/julia.expmap
@@ -67,10 +67,7 @@
67 67
     jl_array_grow_beg;
68 68
     jl_array_del_beg;
69 69
     jl_array_ptr;
70  
-    jl_sizeof_off_t;
71  
-    jl_sizeof_ios_t;
72 70
     jl_stdout_stream;
73  
-    jl_sizeof_stat;
74 71
     jl_stat;
75 72
     jl_lstat;
76 73
     jl_fstat;
@@ -128,8 +125,6 @@
128 125
     connect_to_host;
129 126
     open_any_tcp_port;
130 127
     getlocalip;
131  
-    jl_sizeof_fd_set;
132  
-    jl_sizeof_timeval;
133 128
     jl_set_timeval;
134 129
     jl_fd_clr;
135 130
     jl_fd_isset;
@@ -221,7 +216,10 @@
221 216
     jl_defer_signal;
222 217
     jl_register_toplevel_eh;
223 218
     jl_dump_function;
224  
-    uv_exepath;
  219
+    c_dlopen;
  220
+    uv_*;
  221
+    jl_uv_*;
  222
+    jl_sizeof_*;
225 223
   local:
226 224
     *;
227 225
 };
2  src/julia.h
@@ -811,7 +811,7 @@ jl_function_t *jl_get_expander(jl_module_t *m, jl_sym_t *macroname);
811 811
 void jl_set_expander(jl_module_t *m, jl_sym_t *macroname, jl_function_t *f);
812 812
 
813 813
 // external libraries
814  
-DLLEXPORT uv_lib_t *jl_load_dynamic_library(char *fname);
  814
+DLLEXPORT int jl_uv_dlopen(const char *fname, uv_lib_t* lib);
815 815
 DLLEXPORT void *jl_dlsym(uv_lib_t *handle, char *symbol);
816 816
 
817 817
 // compiler
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.