Add support for rpcrequest from vim script

[boxofrox]

Make the write socket available to the dispatch thread so an rpc
response may be sent back to the requesting vim script.

Rename cb to notify_handler.

Add request_handler to start_event_loop_cb. Similar to
notify_handler, but returns a result to be sent in the rpc response.

Flush buffered write sockets, otherwise vim will wait indefinitely for
the rpc response.

Not certain whether this is a direction you wanted to go. If not, let me know what you had in mind and I'll try to rewrite this PR to accommodate.

[boxofrox]

This patch lays the foundation for the next PR: using jobstart in vimscript to execute a rust plugin and establish an RPC session over that process's stdin/stdout. feature/session-from-neovim-parent-process

[daa84]

Hello, thanks for patch. Looks good. I think maybe better to use Option here for functions? Idea is: if i don't need one of this two callbacks, i don't need to write any addition code like empty callback.

[boxofrox]

Sounds good. I'll make it happen.

[boxofrox]

I'm trying to create a plugin where vimscript sends requests to the rust plugin, and the rust plugin responds to requests. I want vimscript to deliver a "quit" command over RPC to kill the rust plugin (among other commands). "quit" is arbitrary and demonstrates a situation where state is shared between the callback and main thread.

To facilitate the "quit" command, I have a is_done : Arc<Mutex<bool>> = false shared with the notify_handler (cannot rely on free function here), so it can flip the boolean when vimscript sends the "quit" event, which subsequently causes the main thread in Rust (which monitors is_done) to clean up and die, terminating the plugin.

The problem I'm having is organizing this [complex?] handler in a closure in such a way that my code remains legible, and I'm not bashing myself against "Expected Type<X>, found Type<closure>" compiler errors in Rust when I refactor my plugin.

It kind of sucks that closures require I define them at the time they're passed as arguments. E.g.

    session.start_event_loop_cb(
        Some(|name, args| {
            // lots of code here
        }),
        Some(|name, args| {
            // lots of code here
            Ok(Value::String("ok".to_owned()))
        }));

A Trait for the callbacks might alleviate my issue, but that adds more Type complexity due to sharing across threads that affects everyone. I imagine some users of neovim-lib would get by with simple closures, so I'm not sure a Trait is the right approach.

pub trait NeovimHandler {
    fn handle_notify (&mut self, name: &str, args: Vec<Value>) {}
    fn handle_request (&mut self, name: &str, args: Vec<Values>) -> Result<Value, Value> {
        Err(Value::String("Not implemented".to_owned()))
    }
}

The only other option I know of are trait objects Box<closure>, and I understand their limitations less right now.

Thoughts?

[boxofrox]

I created a version of this patch with a Handler trait for reference https://github.com/boxofrox/neovim-lib/commits/handler-trait.

I think this might work out well with a bit more... work. To mitigate complications with shared state, custom Handler objects will be owned by dispatch_thread. Shared state can be wrapped up in Arc<Mutex<>> as a field in the custom Handler object, and the user defined functions can manage the mutex locks.

For example:

// imagine a complex object with lots fields to share
struct SharedState {
    // insert much shared state here
}

struct MyHandler {
    shared: Arc<Mutex<SharedState>>
}

impl Handler for MyHandler {
    fn handle_notify (&mut self, name: &str, args: Vec<Value>) {
        let shared = self.shared.lock().unwrap();
        // do stuff with `shared`
    }
}

Anyone not working with shared state isn't forced to mess with Arc<Mutex<>>'s.

I'd also like to keep support for closures by added a ClosureHandler type that takes the closures from start_event_loop_cb and encapsulates them. The Handler trait would have it's own start_event_loop_handler function in session.rs.

Sorry for writing so much. I've spent quite some time with this, and it's hard for me to tackle various solutions without losing track of the problems I've had.

[daa84]

I think about it a bit and Optional is bad solution for generics as None does not provide type information. So solution with trait maybe ok.
Another way is to create number of functions with two or one arguments. I think a bit more about this...

[daa84]

Ok i think trait with default implementations of handler's ok for given situation.

To provide support for closures maybe better to use builder pattern for ClosureHandler without generics pointer, but using dynamic dispatch?

ClosureHandler::new().notify_handler(|| {
  // something here
});

Or maybe ClosureHandler just not needed :) as implementation of trait is not a big problem, but gives static dispatch as a result.

[boxofrox]

If you're happy dropping the closure callbacks, I'm happy. Since start_event_loop is a one-time call, we don't gain much from closures, and it's trivial to create a Handler implementation, and that only needs be done once, if at all.

I also don't mind keeping support for closures, if desired. It's an interesting exercise to keep the API backward compatible. I'm not sure if the builder pattern gains anything, though maybe it'll drop the match Option statement from the Handler trait functions.

I'll update the PR to use the Handler trait without closures for now.

[boxofrox]

@daa84 did you want to drop the Option<H> here and use H instead? In order to do so, DefaultHandler must be exposed to the user, so they can pass along that value when they have no need of callbacks.

[daa84]

My first idea with Option was bad, because Option::None for generics does not work good. So i prefer to use two functions with and without handler as was before but with trait as a handler.

[boxofrox]

Ok, I think I got it this time. Option is now out.

[daa84]

One last change, can you move out wr.flush() from write_value to encode function? I have plans to move back to rmp value again and think to remove this code :) this is because previously i think Value is removed from rmp value permanently but they are moved this code to separate crate

[boxofrox]

Certainly. I'll knock that out sometime Monday.

[boxofrox]

I moved flush() to the model.encode function. Let me know if it's the wrong spot.

[boxofrox]

One more thing to consider. A user has to define their handler before they can start the session's event loop, and only then can they create the Neovim object.

If a handler needs to ask Neovim for more information, how might that be done? The Neovim object owns the Session object, which owns the Handler object. A circular dependency forms when the Handler needs to communicate with Neovim before completing its task.

As it stands, it seems the Handler is limited to waiting for Neovim to trigger it. All the Handler processing is done exclusively in Rust, then a response is returned. It's quite a limitation, and I'm wondering if a solution may require more changes to this PR.

One solution might be to use channel so the handler can communicate with the Rust scope holding onto the Neovim object, and that scope will process requests (e.g. getpos) and send the results back to the handler so it can finish it's response to Neovim.

Not sure what to do here, or if we can worry about this later, but I just ran into this issue with my test plugin. I can put this in the issue tracker if you prefer.

[boxofrox]

It just occurs to me that a single dispatch thread won't be able to process responses from Neovim while that same thread is executing a Handler method.

[daa84]

Yes, your right, the idea is: special thread waits for response from neovim and if this thread blocks - response can't be taken. So to process make neovim call it is need to make response to be processed. Currently only sync call implemented, so it's not possible to make call and then go to next instruction.

Solution is: take event in event process thread -> send this event (also with id) to channel (don't wait here for response, so thread can continue execution) -> do some process in main thread -> send response with given id in main thread. This way it will work i think :)

[daa84]

I don't think previously of such use case :(

[boxofrox]

No worries. 😄

I tried spawning threads to handle events in dispatch_event, but I found Neovim replies to subsequent commands with a WouldBlock error. I'm not sure if this is because my plugin is trying to communicate with Neovim while Neovim is in insert mode, or something else.

If I figure something out that patches neovim-lib, I'll offer up another PR for consideration.

Cheers

[boxofrox]

Sending events over channels is working very well. Thanks for the suggestion.

[daa84]

Thanks