WIP: libuv rewrite and Windows port

[Keno]

WARNING: Do not merge yet. I opened this pull request to have a central place for review/discussion.

The rewrite of Julia based on libuv (supporting Win,Linux and OS X) is getting more and more stable. I need to do some more testing, but now that all major features are implemented, I wanted to open up review/discussion. Fire away!

[nolta]

Wow, looks like a lot of work.

Is this just the libuv patch, or the full Windows patch?

Are all the external/cmake files necessary?

[Keno]

It's the full windows patch. The cmake files are necessary if the normal build system doesn't work (which can happen quite easily on windows and (even more importantly) if we want to move move to MSVC (which has better code generation/optimization on windows). Furthermore cmake makes it quite easy to do out-of-source builds which can be useful at times (e.g. to have a separate set of debug/release libraries - I used this during testing because with certain llvm flags, you can actually debug the JIT [That'l be possible in release mode with the next release of LLVM, too]). So no, not stricly necessary to build, but they don't hurt either and can be very useful if make doesn't work out of the box.

[ViralBShah]

@loladiro what's the simplest way I can try this out on OS X on my setup? Should I fork your repository?

[Keno]

git clone git://github.com/loladiro/julia.git && git submodule update --recursive --init && make

should work just fine

[nolta]

...
make[1]: /scratch/nolta/win32/julia/external/root/../root2/bin/llvm-config: Command not found
    CC src/jltypes.o
In file included from jltypes.c:17:0:
julia.h:7:16: fatal error: uv.h: No such file or directory

Also, external/root2 is empty.

[Keno]

Fixed (I used root2 to switch between debug/release builds of llvm). Thanks

[nolta]

A couple more problems with the Makefile:

• libuv isn't built by default,
• uv.h isn't installed.

After cd'ing into external/libuv, making it by hand, and copying uv.h to external/root/include, i get the following error:

wrapper.c:190:9: error: ‘uv_process_options_t’ has no member named ‘detached’

Here's the full dump:

    CC src/jltypes.o
jltypes.c: In function ‘print_env’:
jltypes.c:1029:20: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
jltypes.c:1031:5: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
    CC src/gf.o
gf.c: In function ‘type_summary’:
gf.c:974:5: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:976:5: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c: In function ‘jl_apply_generic’:
gf.c:1064:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1067:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1068:13: warning: passing argument 1 of ‘type_summary’ from incompatible pointer type
gf.c:968:14: note: expected ‘struct jl_value_t *’ but argument is of type ‘struct _jl_type_t *’
gf.c:1068:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1070:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1105:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1108:17: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1109:17: warning: passing argument 1 of ‘type_summary’ from incompatible pointer type
gf.c:968:14: note: expected ‘struct jl_value_t *’ but argument is of type ‘struct _jl_type_t *’
gf.c:1109:17: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c:1111:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
gf.c: In function ‘jl_f_make_callback’:
gf.c:1405:26: warning: initialization from incompatible pointer type
gf.c:1406:20: warning: initialization from incompatible pointer type
gf.c:1414:49: warning: assignment from incompatible pointer type
gf.c:1448:5: warning: passing argument 1 of ‘jl_box_int32’ makes integer from pointer without a cast
julia.h:597:53: note: expected ‘int32_t’ but argument is of type ‘void *’
gf.c: At top level:
gf.c:1050:13: warning: ‘enable_trace’ defined but not used
    CC src/support/hashing.o
    CC src/support/timefuncs.o
    CC src/support/dblprint.o
    CC src/support/ptrhash.o
    CC src/support/operators.o
    CC src/support/socket.o
socket.c:191:9: warning: "/*" within comment
socket.c: In function ‘connect_to_addr’:
socket.c:169:1: warning: control reaches end of non-void function
socket.c: In function ‘connect_to_host’:
socket.c:150:1: warning: control reaches end of non-void function
socket.c: In function ‘open_any_udp_port’:
socket.c:110:1: warning: control reaches end of non-void function
socket.c: In function ‘open_any_tcp_port’:
socket.c:88:1: warning: control reaches end of non-void function
socket.c: In function ‘open_tcp_port’:
socket.c:61:1: warning: control reaches end of non-void function
socket.c: In function ‘mysocket’:
socket.c:30:1: warning: control reaches end of non-void function
    CC src/support/utf8.o
    CC src/support/ios.o
    CC src/support/dirpath.o
    CC src/support/htable.o
    CC src/support/bitvector.o
    CC src/support/int2str.o
    CC src/support/dump.o
dump.c: In function ‘hexdump’:
dump.c:22:9: warning: implicit declaration of function ‘jl_printf’
    CC src/support/libsupportinit.o
    CC src/support/arraylist.o
    CC src/support/wcwidth.o
    LINK src/support/libsupport.a
    CC src/flisp/flisp.o
    CC src/flisp/builtins.o
    CC src/flisp/string.o
    CC src/flisp/equalhash.o
    CC src/flisp/table.o
    CC src/flisp/iostream.o
    CC src/flisp/julia_extensions.o
    LINK src/flisp/libflisp.a
    CC src/flisp/flmain.o
    CC src/flisp/flisp
    FLISP src/julia_flisp.boot
    FLISP src/julia_flisp.boot.inc
    CC src/ast.o
ast.c: In function ‘jl_init_frontend’:
ast.c:98:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
ast.c:99:9: warning: implicit declaration of function ‘jl_exit’
    CC src/builtins.o
builtins.c: In function ‘jl_errorf’:
builtins.c:43:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
builtins.c:44:9: warning: implicit declaration of function ‘jl_exit’
builtins.c: In function ‘jl_eval_module_expr’:
builtins.c:246:20: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
builtins.c: In function ‘jl_find_file_in_path’:
builtins.c:475:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
    CC src/module.o
module.c: In function ‘jl_checked_assignment’:
module.c:121:20: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
    CC src/codegen.o
    CC src/interpreter.o
interpreter.c: In function ‘eval’:
interpreter.c:86:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
interpreter.c:87:13: warning: implicit declaration of function ‘jl_exit’
    CC src/alloc.o
    CC src/dlload.o
dlload.c: In function ‘jl_load_dynamic_library’:
dlload.c:77:21: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
dlload.c:98:5: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
dlload.c: In function ‘jl_print’:
dlload.c:105:1: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
dlload.c: In function ‘jl_dlsym’:
dlload.c:120:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
dlload.c:121:13: warning: implicit declaration of function ‘jl_exit’
    CC src/sys.o
    CC src/init.o
init.c: In function ‘julia_init’:
init.c:190:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
init.c:192:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
init.c:193:13: warning: implicit declaration of function ‘jl_exit’
init.c:205:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
init.c:214:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
init.c:224:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
init.c:233:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
init.c: At top level:
init.c:121:13: warning: ‘chachedPagesize’ defined but not used
    CC src/task.o
task.c: In function ‘jl_raise’:
task.c:539:13: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
task.c: In function ‘jl_init_tasks’:
task.c:686:50: warning: assignment from incompatible pointer type
    CC src/array.o
    CC src/dump.o
dump.c: In function ‘jl_restore_system_image’:
dump.c:853:9: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
julia.h:984:45: note: expected ‘struct uv_stream_t *’ but argument is of type ‘struct uv_tty_t *’
dump.c:854:9: warning: implicit declaration of function ‘jl_exit’
    CC src/wrapper.o
wrapper.c: In function ‘closeHandle’:
wrapper.c:48:13: warning: implicit declaration of function ‘free’
wrapper.c:48:13: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c:55:17: warning: implicit declaration of function ‘jl_callback_call’
wrapper.c:56:17: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c:58:13: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c:67:17: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c:68:13: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c:73:5: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c: In function ‘jl_return_spawn’:
wrapper.c:93:5: warning: passing argument 1 of ‘uv_close’ from incompatible pointer type
support/../../external/libuv/include/uv.h:388:45: note: expected ‘struct uv_handle_t *’ but argument is of type ‘struct uv_process_t *’
wrapper.c: At top level:
wrapper.c:96:6: warning: conflicting types for ‘jl_callback_call’
wrapper.c:55:17: note: previous implicit declaration of ‘jl_callback_call’ was here
wrapper.c: In function ‘jl_callback_call’:
wrapper.c:100:5: warning: implicit declaration of function ‘malloc’
wrapper.c:100:25: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:105:17: warning: assignment from incompatible pointer type
wrapper.c: In function ‘jl_make_pipe’:
wrapper.c:131:28: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:131:21: warning: unused variable ‘opts’
wrapper.c: In function ‘jl_start_reading’:
wrapper.c:150:28: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c: In function ‘jl_spawn’:
wrapper.c:179:28: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:190:9: error: ‘uv_process_options_t’ has no member named ‘detached’
wrapper.c: In function ‘jl_make_async’:
wrapper.c:239:25: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:242:5: warning: passing argument 3 of ‘uv_async_init’ from incompatible pointer type
support/../../external/libuv/include/uv.h:958:44: note: expected ‘uv_async_cb’ but argument is of type ‘void (*)(struct uv_handle_t *, int)’
wrapper.c: In function ‘jl_idle_init’:
wrapper.c:255:23: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:258:5: warning: return makes integer from pointer without a cast
wrapper.c: In function ‘jl_idle_start’:
wrapper.c:265:29: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:268:5: warning: passing argument 2 of ‘uv_idle_start’ from incompatible pointer type
support/../../external/libuv/include/uv.h:938:44: note: expected ‘uv_idle_cb’ but argument is of type ‘void (*)(struct uv_handle_t *, int)’
wrapper.c: In function ‘jl_idle_stop’:
wrapper.c:275:9: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c: In function ‘jl_free_buffer’:
wrapper.c:282:5: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c: In function ‘jl_puts’:
wrapper.c:287:5: warning: implicit declaration of function ‘strlen’
wrapper.c:287:25: warning: incompatible implicit declaration of built-in function ‘strlen’
wrapper.c: At top level:
wrapper.c:330:5: warning: large integer implicitly truncated to unsigned type
wrapper.c: In function ‘jl_putc’:
wrapper.c:337:27: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:338:9: warning: pointer targets in initialization differ in signedness
wrapper.c:341:25: warning: initialization from incompatible pointer type
wrapper.c: In function ‘jl_write’:
wrapper.c:349:27: warning: incompatible implicit declaration of built-in function ‘malloc’
wrapper.c:354:25: warning: initialization from incompatible pointer type
wrapper.c: In function ‘jl_vprintf’:
wrapper.c:366:5: warning: implicit declaration of function ‘vasprintf’
wrapper.c:371:9: warning: incompatible implicit declaration of built-in function ‘free’
wrapper.c: In function ‘jl_exit’:
wrapper.c:398:5: warning: implicit declaration of function ‘exit’
wrapper.c:398:5: warning: incompatible implicit declaration of built-in function ‘exit’
wrapper.c: In function ‘jl_write’:
wrapper.c:357:1: warning: control reaches end of non-void function
wrapper.c: In function ‘jl_putc’:
wrapper.c:344:1: warning: control reaches end of non-void function
wrapper.c: In function ‘jl_puts’:
wrapper.c:288:1: warning: control reaches end of non-void function
make[1]: *** [wrapper.o] Error 1
make: *** [julia-release] Error 2

[Keno]

Ah yes, I saw that upstream removed that feature temporarily a few hours ago, because it wasn't working yet (we're not using it anyway, but of course we have to tell it not to.)

[nolta]

Ok, after deleting the offending line,

...
    LINK uijulia-release-readline
/scratch/nolta/win32/julia/external/root/lib/libreadline.a(display.o): In function `cr':
/scratch/nolta/julia/external/readline-6.2/display.c:2465: undefined reference to `tputs'
...

so i added -lncurses -lcurses back to READLINE in Make.inc, and now i get

    LINK uijulia-release-readline
    LINK ui/webserverjulia-release-webserver
server.cpp: In function ‘void scgi::read_header(uv_stream_t*, intptr_t, uv_buf_t)’:
server.cpp:544:9: warning: name lookup of ‘i’ changed
server.cpp:491:9: warning:   matches this ‘i’ under ISO standard rules
server.cpp:521:21: warning:   matches this ‘i’ under old rules
    PERL jl/pcre_h.jl
    PERL jl/errno_h.jl
    JULIA sys0.ji
julia: src/unix/stream.c:835: uv_write2: Assertion `(stream->type == UV_TCP || stream->type == UV_NAMED_PIPE || stream->type == UV_TTY) && "uv_write (unix) does not yet support other types of streams"' failed.
/bin/sh: line 1: 32203 Aborted                 ./julia -b stage0.jl
make: *** [sys0.ji] Error 134

[JeffBezanson]

Wow this is amazing. Haven't looked through all of it yet, but I noticed some I/O-related stuff. jl_write takes a pointer to a julia Array and stores it in an I/O request structure, but the Array could easily be freed before the request is processed unless I missed something. The memory management situation is tricky here.

Less important, I'm worried that putc calling malloc could be a big performance problem.

[Keno]

I'll have a look at it tonight

[Keno]

@nolta That assertion failure is part of a problem that is caused by the differences of 32 vs. 64 bit (I was using ming32 on windows). I should have a fix soon.
@JeffBezanson You are quite correct. Regarding the putc issue, I think that would be easy to solve by introducing a buffer of write requests (they are always processed in order, so all we'd have to do is store beginning and end and then give out the pointers as appropriate). Do you think that would work?
Regarding the other issue, I have been thinking, but as far as I see it there are only two solutions:

• Copy the data to a new buffer (we could use a similar technique as for putc here to avoid malloc calls)
• Prevent the data from array from getting freed as long as the write req isn't processed yet.
  • How could the array be freed? Does the GC do so? If so, can we prevent the GC from freeing it (I had a look because I was wondering how Node.JS deals with the problem, and they keep a reference around until the data is written at which point the ref count drops to 0 and the object is freed). Of course the Julia GC isn't ref counted (AFAIK, I haven't looked at the code too closely), but could we do something similar?

Any other ideas?

[JeffBezanson]

The make_callback function seems to be unnecessary; you should just pass a normal function object. Doing the method lookup in advance is a premature optimization at best; the repeated code it involves is not worth it. The malloc'd object also hides references to julia objects.

Yes, the GC could free the array. We will need a way to protect specific objects from GC until they are released. There is a function jl_gc_preserve, but it can only unprotect objects in stack order. We need to add a function to remove specific objects from the preserved list. It will have to be O(n) but it's the best we can do for now. Maybe later we can add a "protect" bit to objects that prevents them from being freed.

[Keno]

The problem here is doing the type inference and multiple dispatch. I wanted you to have a look at that anyway (my solution was just temporary). I rewrote it a little bit, but do you have any ideas how to pass arbitrary Julia functions to C and getting the types right? I'm sure you're way more qualified than I to write that code. Let me know what you think.

-----Original Message-----
From: JeffBezanson [mailto:reply+i-3501333-
30349dd8e15c8ae004ef86f6e250081055119f59-
1291671@reply.github.com]
Sent: Dienstag, 6. März 2012 16:17
To: Keno Fischer
Subject: Re: [julia] RFC: libuv rewrite (#521)

The make_callback function seems to be unnecessary; you should just pass
a normal function object. Doing the method lookup in advance is a
premature optimization at best; the repeated code it involves is not worth it.
The malloc'd object also hides references to julia objects.

Yes, the GC could free the array. We will need a way to protect specific
objects from GC until they are released. There is a function
jl_gc_preserve, but it can only unprotect objects in stack order. We need
to add a function to remove specific objects from the preserved list. It will
have to be O(n) but it's the best we can do for now. Maybe later we can add
a "protect" bit to objects that prevents them from being freed.

---

Reply to this email directly or view it on GitHub:
#521 (comment)

[JeffBezanson]

Any julia function (full generic function, method, whatever) is callable with the exact same jl_apply interface. So looking up the method in advance just creates an opportunity to accidentally call the wrong method. This callback mechanism should naively take a julia function object, construct arguments based on the data it wants to pass (pointers, it seems), and call the function on that.

Code like this:

        v->type = jl_tupleref(cb->types,i);
        *jl_bits_data(v)=va_arg(argp,void*);

is dangerous since it assumes the type is a pointer-sized bits type, but the actual type was chosen elsewhere. Instead it should do something like jl_box_long to construct a valid object.

Also JL_GC_PUSH(argv) won't work since it doesn't know the size of the argv array. You need to use JL_GC_PUSHARGS(argv,n) instead, initialize the array to zero (could use calloc), and put it before you start allocating objects, since any allocation can trigger a GC.

[Keno]

@JeffBezanson
Ok, how about:

    JL_CALLABLE(jl_f_make_callback)
    {
        JL_TYPECHK("make_callback",function,args[0]);
        JL_TYPECHK("make_callback",tuple,args[1]);
        jl_callback_t *cb = malloc(sizeof(jl_callback_t));
        cb->function=(jl_function_t*)args[0];
        cb->types=(jl_tuple_t*)args[1];
        cb->state=nargs>2?args[2]:0;
    #ifdef ENVIRONMENT64
        jl_value_t *t = jl_box_int64((void *)cb);
    #else
        jl_value_t *t = jl_box_int32((void *)cb);
    #endif
        JL_GC_POP();
        return t;
    }

    void jl_callback_call(jl_callback_t *cb,...)
    {
        int l = cb->types->length+((cb->state!=0)?1:0);
        jl_value_t **argv = calloc(l,sizeof(jl_value_t*));
        JL_GC_PUSHARGS(argv,l);
        va_list argp;
        va_start(argp,cb);
        jl_value_t *v;
        int i=0;
        if(cb->state!=0) argv[i++]=cb->state;
        for(i; i<l; ++i) {
            jl_type_t *t = (jl_type_t*)jl_tupleref(cb->types,i-(cb->state>=0?1:0));
            if(t==
    #ifdef ENVIRONMENT64
            jl_pointer_type||t==
    #endif
            jl_int64_type) {
                v = jl_box_int64(va_arg(argp,int64_t));
            } else if(t==
    #ifdef ENVIRONMENT32
            jl_pointer_type||t==
    #endif
            jl_int32_type) {
                v = jl_box_int32(va_arg(argp,int32_t));
            } else {
                jl_error("callback: only Ints and Pointers are supported at this time");
            }
            v->type=t;
            argv[i]=v;
        }
        jl_apply(cb->function,(jl_value_t**)argv,l);
        JL_GC_POP();
    }

[JeffBezanson]

Much better.

But next, observe that making the callback struct is essentially the same as allocating a closure --- a function plus arguments to pass. For example, make_callback(io_callback,(),io) could be just ()->io_callback(io).

The other problem with the jl_callback_t is our GC can't find it and doesn't know its layout. If you use a normal julia object, all you have to do is make sure it's rooted while it might be needed by libuv, and things will be fine.

[Keno]

Yes, I learned about closures after I had already written that originally. I'll rework it to use closures.
How do I tell the GC about jl_callback_t while still making sure that it's not deleted in the middle of a request? Right now, I'm freeing the callback when the request is done (which is of course not ideal, because it doesn't allow reuse of callbacks). I wanted to implement the make_callback function in Julia anyway, I just have to make sure I can use it from the C side. Would the thing to do be adding a new type in jltypes.c and then telling Julia about it in builtins.c?

[JeffBezanson]

Probably not; if necessary you can define the callback type in julia and access it from C using jl_fieldref(obj, i) with the index of the field (currently all objects are just arrays of pointers). But, I'm not convinced a special callback object is actually needed.
Whatever representation you use, you'll have to put any julia object on the preserved list until I/O is done with it, then remove it.

[ViralBShah]

I cloned the repo, and just tried building on OS X. This seems like a simple build issue. Will try fixing it later in the day.

In file included from jltypes.c:17:
julia.h:7:16: error: uv.h: No such file or directory
jltypes.c: In function ‘print_env’:
jltypes.c:1029: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
jltypes.c:1031: warning: passing argument 1 of ‘jl_printf’ from incompatible pointer type
make[1]: *** [jltypes.o] Error 1
make: *** [julia-release] Error 2

[Keno]

Yes, I already fixed it in my local repo. I'll push shortly: I'm just sorting out some callback stuff.

[Keno]

Ok, I'm figuring something out using closures, but the syntax is a mess (I guess I'll write some kind of macro to make it better). I'll probably have a rough prototype of that tomorrow.
Also, @ViralBShah that commit I just pushed is unfortunately broken, because I accidentally pushed some of the experimental work on closures I was working on. I should have a proper version tomorrow though, sry!

[ViralBShah]

@loladiro I tried pushing to your fork, but I didn't have permission. I expected it to generate a pull request. Anyways, here is what I did to get uv to build and install. You can replace what you have with:

## libuv

compile-uv:
    $(MAKE) -C libuv
install-uv: compile-uv
    cp libuv/uv.a $(EXTROOTLIB)
    cp libuv/include/uv.h $(EXTROOT)/include

clean-uv:
    $(MAKE) -C libuv clean
    rm -f $(EXTROOTLIB)/uv.a $(EXTROOT)/include/uv.h
distclean-uv: clean-uv

[Keno]

Done. Also, I gave you, Jeff and Stefan commit access to my fork

[Keno]

@JeffBezanson I've been trying to figure the closure thing out. It works fine if I don't have to pass any parameters back such as

()->io_callback(io)

But then there are also situations in which I do have to pass parameters back. I tried something like:

(handle::PtrSize,exitStatus::PtrSize,termSignal::PtrSize)->process_exited(pp,handle,exitStatus,termSignal)

However, I can't seem to figure out how to access the types of the parameters. I tried f->linfo->sparams, but that does not seem to be set. What would be the best way to get the type information?

[Keno]

@JeffBezanson I'm still trying to figure out how to get the type information (I haven't had much time to work on it due to midterms), would

jl_lam_args(f->linfo)

be the way to do that? And if so, how do I get the type information from the returned array of variables?

[JeffBezanson]

No, I don't think you should do anything like that. If they're pointers, just package them using jl_box_long (for windows we will probably need jl_box_pointer) and do everything else from within the juila callback.

[Keno]

What if I want to have ints on a 64 bit platform? I need to get at least some sort of type information to know what kind of C type is expected. Originally I had an additional type field in jl_callback_t, so that I would know what kind of c type to get: I guess I could pass pointers to the c values to julia, but then there still remains the problem of how many arguments the Julia function expects. Now, I could make that an additional argument to jl_callback_call, but that's unnecessary duplication of information, given that we already of to add another layer of indirection since we can't directly pass ints.

[JeffBezanson]

At the point where jl_callback_call is called in C, for example:

jl_callback_call(cb, x, y)

you know the types of x and y. There's your type information. I understand jl_callback_call is a varargs function, but setting up the extra objects to get the type info from julia I think will actually end up being more complicated, since it requires the special callback_t object and make_callback. That could all be avoided by writing the calls as something like jl_callback_call(cb, 2, CB_INT, x, CB_PTR, y). Looks ugly, but the net amount of code and potential memory-management problems is less.

[Keno]

Yes, we would have to add another field to uv_pipe_t. I am not quite sure how to do it correctly yet, since we are quite clearly overstepping the use case that this part of libuv was originally designed for (basic reading and writing to/from processes). However, that is exactly the reason we have a separate fork of libuv, so we can make the changes need for julia without having to worry about breaking anything upstream. I guess the thing to do here would be to have two fd field, one named fd_in and one fd_out and set them appropriately whenever uv_pipe_t is used inside libuv.

[StefanKarpinski]

Any changes made to our copy of libuv should definitely be considered for merging back upstream though. If we need it, someone else may also, and, of course, it would be better if we didn't have to maintain a separate patched version :-)

[Keno]

Right, once we figure out how do to it properly I'll shoot a quick email to the maintainers of libuv, to see if they can use any of our additions (I doubt it though as it's not used within node, but we'll see).

[pao]

I think the libuv maintainers are well aware that Node is not their only customer, and are all for more users of their library. Just justify it well and pay attention to the review feedback, same as any other upstream. Worst case, we keep the fork.

[jfhbrook]

My understanding is that the spawn/fork situation isn't the most polished part of libuv. They probably need a fix and don't know it yet.

[vtjnash]

It doesn't seem too too bad, it just wasn't made to do anything complex. So it'll take a bit of work to get it ready for our needs. I'm tracking the primary issues here Keno#5, if you care to help comment.

Is the following behavior intentional (i.e. desired) or accidental?
"Finally, we have an example of how you can make a process read from itself:
julia> run(gen | dup | dup)
This example never terminates since the dup process reads its own output and duplicates it to stderr forever." (http://julialang.org/manual/running-external-programs/)

I ask since in the current implementation of the libuv port, this is currently broken (in short, it fails trying to make an infinite pipeline of dup processes). The question is, should it be fixed to duplicate the behavior described in the manual or does it make more sense if this acts like the following (which is more closely what it is currently trying to do):
julia> run(gen | copy(dup) | copy(dup))

-Jameson

On Apr 21, 2012, at 4:43 AM, Joshua Holbrook wrote:

My understanding is that the spawn/fork situation isn't the most polished part of libuv. They probably need a fix and don't know it yet.

---

Reply to this email directly or view it on GitHub:
#521 (comment)

[JeffBezanson]

Just want to jot down a few notes from the meeting yesterday:

• Need interface for wrapping a pointer as an Array (I'm on it)
• Need to do less buffer copying, and avoid the byte-at-a-time i/o currently done by Deserializer. We should perhaps prepend a length to each message so we can read the whole thing into a buffer, and deserialize out of it in one step. On the serialize() side, we should fully hand off the message buffer to libuv without copying it.
• Should get rid of the single-byte functions for AsyncStream, or make them buffer by default.
• At 12,000 lines, this patch is almost the same size as the whole codebase :) We need to start manually merging it over, possibly even a file at a time. I especially want to get the general portability fixes in, so at least things generally compile on windows even if i/o doesn't work at first.

[JeffBezanson]

Another one:

• Try to write gc_mark_libuv_data that traverses libuv's state in order to mark any julia objects we know are in there. Then we don't need any extra data structures in the GC.

[jfhbrook]

It doesn't seem too too bad, it just wasn't made to do anything complex. So it'll take a bit of work to get it ready for our needs.

Yeah, definitely. We use it for process management at work so one of my coworkers had to make it able to detach processes. From what I remember they were definitely open to improvements but most of the core efforts were going towards tcp (natch) so it was/is an area that can use some love.

[jfhbrook]

Also: I heard that the libuv peeps are also talking about fixing the stream issue right now so almost definitely talk to them. :)

[jfhbrook]

See: Keno#5 (comment)

[Keno]

I just updated this pull request against the current master. The diff now contains almost exclusively changes to the Julia codebase itself (The diff dropped from 11,000 lines to ~5,700 lines). Also note that although I will keep this pull request up, it will not be merged but rather I will continue in the current form of slowly adding parts of it to master on a feature-by-feature basis.

[miau]

How can I build ff574dc on MinGW? It seems that the variable WGET_DASH_O isn't set on MinGW and the following error occurs.

/bin/sh: random/dsfmt-2.1.tar.gz: No such file or directory

Setting the variable WGET_DASH_O manually, the following error occurs.

patching file dSFMT.h
/bin/sh: line 2: /c/julia/deps/install-name-WINNT.sh: No such file or directory

[Keno]

sry about that. We recently changed the build system and I did not yet adjust the build scripts for windows. I'll do that tomorrow. If you want to try it today, you can use commit 3aa49c5 , but you'd have to recompile once I fix it tomorrow. Again, I apologize for the inconvenience.

[miau]

Thanks. I tried 3aa49c5 and it has been built successfully. But there are some problems remaining.

• min()/max() on Float64 causes an error.

julia> min(1.0, 2.0)
Error: Symbol Could not be found fmin (-1:127)

• readline() seems to be broken.

julia> f = open("README.md")
IOStream(<file README.md>)

julia> readline(f)
"�h\U-ffdbc0\0\0\x01\0\x01\0\0\0\0\0\x13\0\0\0P\0\0\0\0\0\0\0<a name=\"banner\"
/>\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
(snip)
0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\x01\0\0Ћ\x02\x1f\0\0invalid UTF-8 character index

• Moving cursor with Home/End key would be nice for Windows users.
  Though there are some settings,
  "\e[1~" and "\e[4~" seems to be enough for my console (cmd.exe on Windows 7.)
  Here is a patch.

diff --git a/ui/repl-readline.c b/ui/repl-readline.c
index adfcdf3..071c3ff 100644
--- a/ui/repl-readline.c
+++ b/ui/repl-readline.c
@@ -522,7 +522,9 @@ static void init_rl(void)
         rl_bind_key_in_map('\005',     line_end_callback,   keymaps[i]);
         rl_bind_key_in_map('\002',     left_callback,       keymaps[i]);
         rl_bind_key_in_map('\006',     right_callback,      keymaps[i]);
-        rl_bind_keyseq_in_map("\033[3~",delete_callback,    keymaps[i]);
+        rl_bind_keyseq_in_map("\e[1~", line_start_callback, keymaps[i]);
+        rl_bind_keyseq_in_map("\e[4~", line_end_callback,   keymaps[i]);
+        rl_bind_keyseq_in_map("\e[3~", delete_callback,     keymaps[i]);
         rl_bind_keyseq_in_map("\e[A",  up_callback,         keymaps[i]);
         rl_bind_keyseq_in_map("\e[B",  down_callback,       keymaps[i]);
         rl_bind_keyseq_in_map("\e[D",  left_callback,       keymaps[i]);

[Keno]

The fmin stuff is known. Microsoft does not provide the proper functions in the CRT. We'll switch to our own libm implementation soon. I'll have a look at the readline failure and I'll incorporate the shortcut suggestion. Thanks for your feedback!

[ViralBShah]

Given the huge demand for the windows port, is it possible for us to start merging bits and pieces that will let people try out the windows port, even though not all the featuers may be ready?

[Keno]

I've already been merging bits and pieces all along. There's one bug that's currently blocking a binary release, but as soon as I get that figured out, I'll upload it.

[Keno]

Ignore this. It wasn't actually merged. I accidentally force pushed JuliaLang/master onto my own repository (thus there were no changes anymore). I'll see if I can reopen somehow.

[Keno]

Ok, it seems that I can't reopen. I'll just leave it as is until we're ready to merge and the figure things out from there.

[pao]

It appears that's the case for pull requests. You could put up a new one and crossreference to this issue if you want to keep this visible.

[alanedelman]

Don't know why the windows repl color is always lousy but here's the most recent ugliness:

if this is not visible, i'll just say that the response to the prompt is an equally invisible light shade of gray :-)

[StefanKarpinski]

Can you try ENV["JULIA_ANSWER_COLOR"] = "magenta" and see how that looks?