[RFC] Reimplement job control patch with libuv #475

tarruda · 2014-04-05T21:41:09Z

This is an improved version of the job control patch I sent to vim mailing list a few months ago, and it will be the base for the new plugin architecture. The job module implemented here will be reused for spawning plugins.

The basic difference between plugins and plain jobs started with jobstart is that instead of invoking an auto commands passing the raw data to the vimscript programmer, plugins will have access to Neovim msgpack API directly

This is still a WIP so I'm still going to fix documentation, formatting, etc(this was basically a copy and paste from the patch)

schmee · 2014-04-05T21:51:11Z

Great to see the new plugin architecture taking form! 🎉

oni-link · 2014-04-05T22:38:52Z

src/os/job.c

+  uv_close((uv_handle_t *)&job->proc_stderr, NULL);
+  shell_free_argv(job->proc_opts.args);
+  free(job);
+}


Looks like the process watcher is not closed.

justinmk · 2014-04-06T00:03:16Z

Couldn't wait, so I just tested it. It almost works :)

In case it's of any value, I get Abort trap: 6 by the following steps:

Create a simple bash script:

$ echo "sleep 3 && exit 42" > test.sh && chmod u+x test.sh

Do the following in nvim:

:au JobActivity * echom string(v:job_data)
:call jobstart('foo', '/Users/justin/test.sh')

tarruda · 2014-04-06T00:38:00Z

@justinmk this last commit should have fixed it. Here's something else you can use to play with it:

let g:nc = jobstart('netcat', 'nc', ['-l', '1234'])
au JobActivity netcat call append(line('$'), v:job_data[1])

Open another terminal and type the following:

nc localhost 1234

and start entering lines. Also calling jobwrite(g:nc, str) should write str to the terminal running the netcat client.

Note that I'm still not handling the redraw issue correctly, for now it's simply calling shell_resized() which is sub-optimal.(Still need to figure out the right functions to call for redrawing only the invalidated columns)

oni-link · 2014-04-06T15:34:29Z

src/os/job.h

+/// @param data Caller data that will be associated with the job
+/// @param cb Callback that will be invoked everytime data is available in
+///        the job's stdout/stderr
+/// @return The job id


job_start can fail. Returns 0 in case of error.

tarruda · 2014-04-06T20:50:59Z

I'm curious, is there some "rule" against usage of goto statements? Many say it's always a bad practice, but personally I think they are a simple way to perform cleanup in a function(Currently using it in f_job_start. One can also extract cleanup code into a separate function like I have done here but I think it's a bit of overkill. Here is an interesting article about the subject.

I think this is something we might want to establish in our code style, as I said I'm ok with gotos for cleanup code.

justinmk · 2014-04-06T20:59:17Z

@tarruda The dogmatism surrounding anti-goto sentiment is cargo-cult adherence to Dijktra's famous "goto considered harmful" essay. It's still the right tool for C occasionally.

The essay was originally written in the context of trying to convince assembly programmers of the 1970s that function dispatch was not going to kill performance.

One can also extract cleanup code into a separate function like I have done here but I think it's a bit of overkill

Exactly. In languages with private functions (and no forward declaration requirement) and RAII or garbage collection, goto is a code smell. But C doesn't have those things, so AFAIK goto is acceptable in some limited cases.

tarruda · 2014-04-07T01:58:31Z

@justinmk thanks for the link, interesting read :)

tarruda · 2014-04-07T03:12:34Z

I'm not satisfied with having to call shell_resized after handling each job event. The problem seems to be that the whole redrawing logic is scattered across edit.c/normal.c(not sure though, I have tried playing with the functions in screen.c but couldn't find anything obvious) and is executed whenever keys are pressed. The shell_resized call can be removed by using the 'events as keys' hack I proposed in #395, this will make vim redraw after asynchronous events more 'naturally'. In any case this is something to be fixed in another PR

Some may be wondering why I'm using job ids instead of opaque pointers(could have made the fields of Job private to the job module for example) and the reason is simple: I couldn't find a way to pass arbitrary pointers to vimscript, the simplest solution was to use integer ids.

I think this is ready for review, but I will only merge it after #476 . After this is merged I can implement the msgpack API hopefully we'll be able to use python plugins.

justinmk · 2014-04-07T14:09:46Z

@tarruda Playing around with netcat worked beautifully. One question so far: there doesn't seem to be any protection from overwriting an existing job with the same job name:

:jobstart('netcat', 'nc', ['-l', '1234'])
:jobstart('netcat', 'nc', ['-l', '1234'])

For a plugin to avoid collisions like this, it would be useful if jobstart() with, say, an empty string as the first parameter returned the next available job id and registered the job without overwriting an existing job. Would it be a bad idea for au JobActivity 1 to match job id 1? That would probably require disallowing job names starting with a number.

tarruda · 2014-04-07T14:46:37Z

@justinmk they won't overwrite each other, I think what happened is that the second call to netcat failed because it tried listening on the same port as the first.

When you pass a name to jobstart you are actually saying to Neovim: 'When that job sends you some data, emit an 'event' with that name'. This is similar to the event emitter pattern in javascript, except that I tried to use the autocommand infrastructure for it. You can also listen for a namespace of events like this:

:let srv1_id = jobstart('netcat-server-1', 'nc', ['-l', '9991'])
:let srv2_id = jobstart('netcat-server-2', 'nc', ['-l', '9992'])

au JobActivity netcat-server-* call append(line('$'), 'message from server '.v:job_data[0].': '. v:job_data[1])

tarruda · 2014-04-07T14:57:48Z

I've updated to use the modifications sent by @philix in #476 but I dont like the way we have to forward declare the Event/Job types to avoid circular dependency problem between the headers. Perhaps we should move typedefs/structs to *_def.h files?

justinmk · 2014-04-07T15:06:33Z

similar to the event emitter pattern in javascript, except that I tried to use the autocommand infrastructure

From a user interface perspective, I think the design works well, so far.

tarruda · 2014-04-07T15:11:18Z

From a user interface perspective, I think the design works well, so far.

There's still some things missing:

Get a list of jobs
Get detailed information about a job, such a pid
Notification about when a job exits

But I think it's better to worry about those after the plugin infrastructure is implemented because there might still be some modifications to the job module.

tarruda · 2014-04-07T15:30:31Z

Rebased/squashed, if no one objects in one hour I will merge it

justinmk · 2014-04-07T15:58:27Z

src/globals.h

@@ -1009,6 +1009,9 @@ EXTERN size_t (*iconv)(iconv_t cd, const char **inbuf, size_t *inbytesleft,
 EXTERN char_u e_invrange[] INIT(= N_("E16: Invalid range"));
 EXTERN char_u e_invcmd[] INIT(= N_("E476: Invalid command"));
 EXTERN char_u e_isadir2[] INIT(= N_("E17: \"%s\" is a directory"));
+EXTERN char_u e_invjob[] INIT(= N_("E900: Invalid job id"));
+EXTERN char_u e_jobtblfull[] INIT(= N_("E901: Job table is full"));
+EXTERN char_u e_jobexe[] INIT(= N_("E902: '%s' is not an executable"));


Probably should use \"%s\" to be consistent with the existing error messages.

tarruda · 2014-04-07T17:05:48Z

@justinmk / @oni-link Thanks for the feedback, these last commits should have fixed the issues reported

oni-link · 2014-04-07T17:34:03Z

src/os/job.c

+    }
+
+    // Still alive
+    while (remaining_tries-- && is_alive(job)) {


Change order of operands, so that we really wait 1s. 20 consecutive jobs, that are not alive anymore at this point, decrease remaining_tries to zero without delay.

- Add a job control module for spawning and controlling co-processes - Add three vimscript functions for interfacing with the module - Use dedicated header files for typedefs/structs in event/job modules

oni-link · 2014-04-07T18:10:12Z

src/os/job.c

+  uv_close((uv_handle_t *)&job->proc_stdin, NULL);
+  uv_close((uv_handle_t *)&job->proc_stderr, NULL);
+  uv_close((uv_handle_t *)&job->proc, NULL);
+  free(job);


Is it save to free job at this point? Are all callbacks that use job canceled (synchronously), or are they waiting for one last run of the event loop?

As far as I know it is safe. Calling uv_close should remove any references from the event loop so no callbacks will be run after this.

felipecrv · 2014-04-08T05:14:31Z

src/os/job.c

+  kBufferLockStderr    ///< Data read from stderr
+} BufferLock;
+
+struct _Job {


Per the style-guide it should be struct job. If job conflicts with something then you could call it job_s or job_struct. Consistent rules for type names is the most useful thing that comes from a coding style guide IMO.

Missed that, sorry

oni-link · 2014-04-18T08:50:41Z

src/os/job.c

+{
+  uv_disable_stdio_inheritance();
+  uv_prepare_init(uv_default_loop(), &job_prepare);
+  uv_prepare_start(&job_prepare, job_prepare_cb);


@tarruda Why not call uv_prepare_start when the first job is started and call uv_prepare_stop after the job table is empty? Otherwise the job_prepare_cb is called every iteration of the uv loop.

Use debian6 package for 32bit llvm+clang, ubuntu one is actually 64bit

oni-link reviewed Apr 5, 2014
View reviewed changes

oni-link reviewed Apr 6, 2014
View reviewed changes

tarruda changed the title ~~[WIP] Reimplement job control patch with libuv~~ [RFC] Reimplement job control patch with libuv Apr 7, 2014

justinmk reviewed Apr 7, 2014
View reviewed changes

oni-link reviewed Apr 7, 2014
View reviewed changes

Implement job control

4b063ea

- Add a job control module for spawning and controlling co-processes - Add three vimscript functions for interfacing with the module - Use dedicated header files for typedefs/structs in event/job modules

oni-link reviewed Apr 7, 2014
View reviewed changes

tarruda mentioned this pull request Apr 7, 2014

Why has all the FEAT_CLIPBOARD code been removed? #483

Closed

tarruda merged commit 4b063ea into neovim:master Apr 7, 2014

felipecrv reviewed Apr 8, 2014
View reviewed changes

tarruda mentioned this pull request Apr 8, 2014

[RFC] Run clint.py in some files #489

Merged

oni-link reviewed Apr 18, 2014
View reviewed changes

This was referenced Jun 2, 2014

libcall feature is not supported in neovim #795

Closed

[RFC] system() prefer pipes over tmpfiles #807

Closed

Grimy pushed a commit to Grimy/neovim that referenced this pull request Jan 7, 2015

Merge pull request neovim#475 from jmesmon/fix474

f347885

Use debian6 package for 32bit llvm+clang, ubuntu one is actually 64bit

LeifW mentioned this pull request Mar 3, 2015

Idris not started? idris-hackers/idris-vim#34

Closed

dwb pushed a commit to dwb/neovim that referenced this pull request Feb 21, 2017

Update Stylelint formatter (neovim#475)

414214b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Reimplement job control patch with libuv #475

[RFC] Reimplement job control patch with libuv #475

tarruda commented Apr 5, 2014

schmee commented Apr 5, 2014

oni-link Apr 5, 2014

justinmk commented Apr 6, 2014

tarruda commented Apr 6, 2014

oni-link Apr 6, 2014

tarruda commented Apr 6, 2014

justinmk commented Apr 6, 2014

tarruda commented Apr 7, 2014

tarruda commented Apr 7, 2014

justinmk commented Apr 7, 2014

tarruda commented Apr 7, 2014

tarruda commented Apr 7, 2014

justinmk commented Apr 7, 2014

tarruda commented Apr 7, 2014

tarruda commented Apr 7, 2014

justinmk Apr 7, 2014

tarruda commented Apr 7, 2014

oni-link Apr 7, 2014

tarruda Apr 7, 2014

oni-link Apr 7, 2014

tarruda Apr 7, 2014

felipecrv Apr 8, 2014

tarruda Apr 8, 2014

oni-link Apr 18, 2014

tarruda Apr 18, 2014

[RFC] Reimplement job control patch with libuv #475

[RFC] Reimplement job control patch with libuv #475

Conversation

tarruda commented Apr 5, 2014

schmee commented Apr 5, 2014

Choose a reason for hiding this comment

justinmk commented Apr 6, 2014

tarruda commented Apr 6, 2014

Choose a reason for hiding this comment

tarruda commented Apr 6, 2014

justinmk commented Apr 6, 2014

tarruda commented Apr 7, 2014

tarruda commented Apr 7, 2014

justinmk commented Apr 7, 2014

tarruda commented Apr 7, 2014

tarruda commented Apr 7, 2014

justinmk commented Apr 7, 2014

tarruda commented Apr 7, 2014

tarruda commented Apr 7, 2014

Choose a reason for hiding this comment

tarruda commented Apr 7, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment