Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Reimplement job control patch with libuv #475

Merged
merged 1 commit into from Apr 7, 2014

Conversation

tarruda
Copy link
Member

@tarruda tarruda commented Apr 5, 2014

This is an improved version of the job control patch I sent to vim mailing list a few months ago, and it will be the base for the new plugin architecture. The job module implemented here will be reused for spawning plugins.

The basic difference between plugins and plain jobs started with jobstart is that instead of invoking an auto commands passing the raw data to the vimscript programmer, plugins will have access to Neovim msgpack API directly

This is still a WIP so I'm still going to fix documentation, formatting, etc(this was basically a copy and paste from the patch)

@schmee
Copy link
Contributor

schmee commented Apr 5, 2014

Great to see the new plugin architecture taking form! 🎉

uv_close((uv_handle_t *)&job->proc_stderr, NULL);
shell_free_argv(job->proc_opts.args);
free(job);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the process watcher is not closed.

@justinmk
Copy link
Member

justinmk commented Apr 6, 2014

Couldn't wait, so I just tested it. It almost works :)

In case it's of any value, I get Abort trap: 6 by the following steps:

Create a simple bash script:

$ echo "sleep 3 && exit 42" > test.sh && chmod u+x test.sh

Do the following in nvim:

:au JobActivity * echom string(v:job_data)
:call jobstart('foo', '/Users/justin/test.sh') 

@tarruda
Copy link
Member Author

tarruda commented Apr 6, 2014

@justinmk this last commit should have fixed it. Here's something else you can use to play with it:

let g:nc = jobstart('netcat', 'nc', ['-l', '1234'])
au JobActivity netcat call append(line('$'), v:job_data[1])

Open another terminal and type the following:

nc localhost 1234

and start entering lines. Also calling jobwrite(g:nc, str) should write str to the terminal running the netcat client.

Note that I'm still not handling the redraw issue correctly, for now it's simply calling shell_resized() which is sub-optimal.(Still need to figure out the right functions to call for redrawing only the invalidated columns)

/// @param data Caller data that will be associated with the job
/// @param cb Callback that will be invoked everytime data is available in
/// the job's stdout/stderr
/// @return The job id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

job_start can fail. Returns 0 in case of error.

@tarruda
Copy link
Member Author

tarruda commented Apr 6, 2014

I'm curious, is there some "rule" against usage of goto statements? Many say it's always a bad practice, but personally I think they are a simple way to perform cleanup in a function(Currently using it in f_job_start. One can also extract cleanup code into a separate function like I have done here but I think it's a bit of overkill. Here is an interesting article about the subject.

I think this is something we might want to establish in our code style, as I said I'm ok with gotos for cleanup code.

@justinmk
Copy link
Member

justinmk commented Apr 6, 2014

@tarruda The dogmatism surrounding anti-goto sentiment is cargo-cult adherence to Dijktra's famous "goto considered harmful" essay. It's still the right tool for C occasionally.

The essay was originally written in the context of trying to convince assembly programmers of the 1970s that function dispatch was not going to kill performance.

One can also extract cleanup code into a separate function like I have done here but I think it's a bit of overkill

Exactly. In languages with private functions (and no forward declaration requirement) and RAII or garbage collection, goto is a code smell. But C doesn't have those things, so AFAIK goto is acceptable in some limited cases.

@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

@justinmk thanks for the link, interesting read :)

@tarruda tarruda changed the title [WIP] Reimplement job control patch with libuv [RFC] Reimplement job control patch with libuv Apr 7, 2014
@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

I'm not satisfied with having to call shell_resized after handling each job event. The problem seems to be that the whole redrawing logic is scattered across edit.c/normal.c(not sure though, I have tried playing with the functions in screen.c but couldn't find anything obvious) and is executed whenever keys are pressed. The shell_resized call can be removed by using the 'events as keys' hack I proposed in #395, this will make vim redraw after asynchronous events more 'naturally'. In any case this is something to be fixed in another PR

Some may be wondering why I'm using job ids instead of opaque pointers(could have made the fields of Job private to the job module for example) and the reason is simple: I couldn't find a way to pass arbitrary pointers to vimscript, the simplest solution was to use integer ids.

I think this is ready for review, but I will only merge it after #476 . After this is merged I can implement the msgpack API hopefully we'll be able to use python plugins.

@justinmk
Copy link
Member

justinmk commented Apr 7, 2014

@tarruda Playing around with netcat worked beautifully. One question so far: there doesn't seem to be any protection from overwriting an existing job with the same job name:

:jobstart('netcat', 'nc', ['-l', '1234'])
:jobstart('netcat', 'nc', ['-l', '1234'])

For a plugin to avoid collisions like this, it would be useful if jobstart() with, say, an empty string as the first parameter returned the next available job id and registered the job without overwriting an existing job. Would it be a bad idea for au JobActivity 1 to match job id 1? That would probably require disallowing job names starting with a number.

@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

@justinmk they won't overwrite each other, I think what happened is that the second call to netcat failed because it tried listening on the same port as the first.

When you pass a name to jobstart you are actually saying to Neovim: 'When that job sends you some data, emit an 'event' with that name'. This is similar to the event emitter pattern in javascript, except that I tried to use the autocommand infrastructure for it. You can also listen for a namespace of events like this:

:let srv1_id = jobstart('netcat-server-1', 'nc', ['-l', '9991'])
:let srv2_id = jobstart('netcat-server-2', 'nc', ['-l', '9992'])

au JobActivity netcat-server-* call append(line('$'), 'message from server '.v:job_data[0].': '. v:job_data[1])

@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

I've updated to use the modifications sent by @philix in #476 but I dont like the way we have to forward declare the Event/Job types to avoid circular dependency problem between the headers. Perhaps we should move typedefs/structs to *_def.h files?

@justinmk
Copy link
Member

justinmk commented Apr 7, 2014

similar to the event emitter pattern in javascript, except that I tried to use the autocommand infrastructure

From a user interface perspective, I think the design works well, so far.

@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

From a user interface perspective, I think the design works well, so far.

There's still some things missing:

  • Get a list of jobs
  • Get detailed information about a job, such a pid
  • Notification about when a job exits

But I think it's better to worry about those after the plugin infrastructure is implemented because there might still be some modifications to the job module.

@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

Rebased/squashed, if no one objects in one hour I will merge it

@@ -1009,6 +1009,9 @@ EXTERN size_t (*iconv)(iconv_t cd, const char **inbuf, size_t *inbytesleft,
EXTERN char_u e_invrange[] INIT(= N_("E16: Invalid range"));
EXTERN char_u e_invcmd[] INIT(= N_("E476: Invalid command"));
EXTERN char_u e_isadir2[] INIT(= N_("E17: \"%s\" is a directory"));
EXTERN char_u e_invjob[] INIT(= N_("E900: Invalid job id"));
EXTERN char_u e_jobtblfull[] INIT(= N_("E901: Job table is full"));
EXTERN char_u e_jobexe[] INIT(= N_("E902: '%s' is not an executable"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should use \"%s\" to be consistent with the existing error messages.

@tarruda
Copy link
Member Author

tarruda commented Apr 7, 2014

@justinmk / @oni-link Thanks for the feedback, these last commits should have fixed the issues reported

}

// Still alive
while (remaining_tries-- && is_alive(job)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change order of operands, so that we really wait 1s. 20 consecutive jobs, that are not alive anymore at this point, decrease remaining_tries to zero without delay.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

- Add a job control module for spawning and controlling co-processes
- Add three vimscript functions for interfacing with the module
- Use dedicated header files for typedefs/structs in event/job modules
uv_close((uv_handle_t *)&job->proc_stdin, NULL);
uv_close((uv_handle_t *)&job->proc_stderr, NULL);
uv_close((uv_handle_t *)&job->proc, NULL);
free(job);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it save to free job at this point? Are all callbacks that use job canceled (synchronously), or are they waiting for one last run of the event loop?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know it is safe. Calling uv_close should remove any references from the event loop so no callbacks will be run after this.

@tarruda tarruda merged commit 4b063ea into neovim:master Apr 7, 2014
kBufferLockStderr ///< Data read from stderr
} BufferLock;

struct _Job {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the style-guide it should be struct job. If job conflicts with something then you could call it job_s or job_struct. Consistent rules for type names is the most useful thing that comes from a coding style guide IMO.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed that, sorry

{
uv_disable_stdio_inheritance();
uv_prepare_init(uv_default_loop(), &job_prepare);
uv_prepare_start(&job_prepare, job_prepare_cb);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tarruda Why not call uv_prepare_start when the first job is started and call uv_prepare_stop after the job table is empty? Otherwise the job_prepare_cb is called every iteration of the uv loop.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Grimy pushed a commit to Grimy/neovim that referenced this pull request Jan 7, 2015
Use debian6 package for 32bit llvm+clang, ubuntu one is actually 64bit
dwb pushed a commit to dwb/neovim that referenced this pull request Feb 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants