[2.0 RFC] "pull based" event loop

[saghul]

EDIT: the proposal is now here: libuv/leps#3

Please read the README for information on the LEP process: https://github.com/libuv/leps/blob/master/README.md

[creationix]

I love it. Could we have the option to go one step farther and instead of having uv_backend_dispatch call the C callbacks, instead have a function that returns the next item in the queue till nothing is left and leave dispatching up to the user?

[saghul]

I love it. Could we have the option to go one step farther and instead of having uv_backend_dispatch call the C callbacks, instead have a function that returns the next item in the queue till nothing is left and leave dispatching up to the user?

I'm not sure that would be possible, because not all callbacks are the same, so libuv will actually need to inspect the queued request type and call the callback accordingly. What use case do you have in mind?

[creationix]

My main use case is for efficient FFI from luajit. It may now support C callbacks, but it's a terribly slow and inefficient path in luajit.

If I was able to instead do something like:

for event in uv.next_event() do
  -- Route to lua callback based on properties on event struct
end

Then I think it would be faster. At the C level, I understand how types can get in the way, but I'd be fine with a void* or leaving it out entirely for things like C callbacks since I plan on passing in NULL anyway for this use case.

[txdv]

C calling into lua is slow?

[creationix]

@txdv, C calling into Lua is a little slow, but not too bad. That's what I do currently with my luv bindings.

What I want is the option of pure ffi libuv bindings without writing any C and without using luajit's callback feature in the ffi since it is terribly slow.

http://luajit.org/ext_ffi_semantics.html#callback

[creationix]

From Mike Pall's docs on ffi, he says:

For new designs avoid push-style APIs: a C function repeatedly calling a callback for each result. Instead use pull-style APIs: call a C function repeatedly to get a new result. Calls from Lua to C via the FFI are much faster than the other way round. Most well-designed libraries already use pull-style APIs (read/write, get/put).

[txdv]

I actually would like this too, since I have an idea which requires me to be in total control of the stack.

[saghul]

@creationix, @txdv I don't think that is going to fly. Requests are opaque structures for the most part, and a few of them will be "internal" as in, you cannot create them, but the loop uses them internally. Thus, the dispatch function will need to know how to call the callback associated with that request. This would result in even more FFI calls.

Lets analyze an example (pseudocode):

uv_req_t* uv_backend_dequeue(uv_loop_t* loop);
void uv_backend_dispatch(uv_req_t* req);

/* dispatch all requests */
while ((req = uv_backend_dequeue(loop)) {
    /* this would trigger the callback associated with the request */
    uv_backend_dispatch(req);
}

The above is C pseudocode for how you'd do it manually. Now, if you use FFI to call into C you'd need to call uv_backend_dispatch for each queued request so that its associated callback is called. If we follow my initial proposal, you'd just need to call a single function, which would fire all callbacks.

I'm not convinced I want this, or maybe it's too early to decide. Nevertheless, what you propose (if I understood correctly) is not orthogonal with what I say, so a new proposal could be made once this one has progressed enough to have a clearer picture.

[txdv]

    /* this would trigger the callback associated with the request */

Should be

We call some funtion which returns the data and then we do the callback in our own programming language.

[saghul]

We call some funtion which returns the data and then we do the callback in our own programming language.

Different requests have different data, I'm not sure this can be achieved in a clean way. I'll keep it in mind, but it won't be part of this particular LEP.

[txdv]

We could like make a union and then according to the type https://github.com/txdv/libuv/blob/master/include/uv.h#L161-L169 let the user use whatever he wants in it?

[bnoordhuis]

That's already how it works unless I'm misunderstanding you. Requests have a common base, uv_req_t. You can look at the type field to figure out what kind of request you are dealing with. Do you (generic you) need anything more?

[piscisaureus]

Continues in libuv/leps#3