[RFC] Fixes and improvements for jobs/streams/events/api #853
Conversation
|
The commit message of 4bf36bd says |
I'm guessing that refers to this commit: Also, tangentially, shouldn't the return value of
Possibly it does, it looks quite nice to be able to just call Now I guess one of the bigger differences is the fact that the |
👍 thanks
That makes sense, I will add 'job_alive' function to check status.
True, I have and idea for fixing that |
I just read the source code of
At first glance, the only one we should worry about is 'out of memory' because libuv calls BTW, what do you think of the optional callback that can be passed to |
There seems to be no way to deal with failures when calling `msgpack_unpacker_next`, so this reimplements that function as `msgpack_rpc_unpack`, which has an additional result for detecting failures. On top of that, we make use of the new function to properly return msgpack-rpc errors when something bad happens.
Before this change, any write that could cause a WStream instance to use more than `maxmem` would fail, which is not acceptable when writing big chunks of data. (This could happen when returning contents from a big buffer through the API, for example). Writes of any size are now allowed, but before we check if the currently used memory doesn't break the limit. This should be enough to prevent us from stacking data when talking to a locked process.
tarruda
changed the title
|
picked some commits from the redraw events PR as those will be needed soon |
Pertinent question indeed. Maybe we can do that before the first true release (not in alpha/beta) (the x-wrappers, because there might be a lot of them). I'm not entirely sure of the best way to do that either.
It gives a lot of flexibility but perhaps makes the usage of I'm have some trouble thinking of a way to uncomplicate the typedef struct {
uv_write_t req;
done_cb cb;
void *data;
} WriteDataRaw;
// write n bytes to the stream, `buf` will not be freed.
// you can safely free the buffer if either done_cb is called
// or you have another way of knowing that the stream was done/closed/...
// (for example when the parent job says it's done).
bool wstream_write_raw(WStream *wstream, const char *buf, size_t size)
{
return wstream_write_raw_cb(wstream, buf, size, NULL, NULL);
}
bool wstream_write_raw_cb(WStream *wstream, const char *buf, size_t size, done_cb cb, void *data)
{
// This should not be called after a wstream was freed
assert(!wstream->freed);
// I'm not sure if `curmem` and `maxmem` are relevant
// for raw writes, since the memory is not managed nor allocated by the
// wstream. Which is why I left the memory management out.
uv_buf_t uvbuf = uv_buf_init((char *) buf, size);
WriteDataRaw *wd = xmalloc(sizeof(WriteDataRaw));
wd->req.data = wstream;
wd->cb = cb;
wd->data = data;
return uv_write(&wd->req, wstream->stream, &uvbuf, 1, write_raw_cb) == 0;
}
// write_cb_raw is a simpler version of write_cb that doesn't try to do free the buffer
static void write_raw_cb(uv_write_t *req, int status)
{
WriteDataRaw *wd = (WriteDataRaw *) req;
WStream *wstream = wd->req.data;
wstream_dec_pending(wstream);
if (wb->cb) {
wd->cb(wd->cb_data);
}
free(req);
}
// @return true if the stream was freed, false otherwise
static bool wstream_dec_pending(WStream *wstream)
{
wstream->pending_reqs--;
if (wstream->freed && wstream->pending_reqs == 0) {
// Last pending write, free the wstream;
free(wstream);
return true;
}
return false;
}
// @return true if the buffer was freed, false otherwise
static bool wbuffer_dec_pending(WBuffer *buffer)
{
if (!--buffer->refcount) {
bool should_free = true;
// perhaps not needed anymore...
if (buffer->cb) {
should_free = buffer->cb(buffer->cb_data);
}
if (should_free) {
// Free the data written to the stream
free(buffer->data);
}
// poison the struct
memset(buffer, 0, sizeof(WBuffer));
free(buffer);
return true;
}
return false;
}I think I mostly failed. By the way, from a bit of inspection of the libuv source I think there can never be a race condition where Also, since now |
|
@aktau the wbuffer/wstream interface should be simpler now while still giving a lot of control to callers |
Looks quite nice indeed! |
- Removed 'copy' parameter from `wstream_new_buffer`. Callers simply pass a copy of the buffer if required. - Added a callback parameter, which is used to notify callers when the data is successfully written. The callback is also used to free the buffer(if required) and is compatible with `free` from the standard library.
The name `async` was not appropriate to describe the behavior enabled by the flag.
This function will be used to temporarily change the `defer` flag on rstream instances.
'job_start' returns the id as an out paramter, and the 'job_find' function is now used by eval.c to translate job ids into pointers.
This is has the same effect as the RStream 'defer' flag, but also works for the job's exit event.
This was done to give more control over memory management to job_write callers.
These functions will never be called directly by the user so bugs are the only reason for passing invalid channel ids. Instead of returning silently we abort to improve bug detection.
To make it possible reuse `event_poll` recursively and in other blocking function calls, this changes how deferred/immediate events are processed: - There are two queues in event.c, one for immediate events and another for deferred events. The queue used when pushing/processing events is determined with boolean arguments passed to `event_push`/`event_process` respectively. - Events pushed to the immediate queue are processed inside `event_poll` but after the `uv_run` call. This is required because libuv event loop does not support recursion, and processing events may result in other `event_poll` calls. - Events pushed to the deferred queue are processed later by calling `event_process(true)`. This is required to "trick" vim into treating all asynchronous events as special keypresses, which is the least obtrusive way of introducing asynchronicity into the editor. - RStream instances will now forward the `defer` flag to the `event_push` call.
They were renamed to find_{buffer,window,tabpage}_by_handle to avoid conflicts
with existing functions of the same name.
- Rename a/n/m to items/size/capactity in kvec.h - Add capactity field to Arrays/Dictionaries
Should be done similar to `neomake#utils#getbufvar` instead anyway.
This PR does three things:
@aktau, this should simplify the implementation of a job_sync function, it should be something like this:
I have not tested the above code but it should work, let me know if you have any questions.