Rpc over Stdin/stdout for plugins

[natefinch]

I saw you mention plugins that communicate over pipes... Not sure exactly what you mean by pipes, but I wanted to float another suggestion - using json RPC over stdin/stdout for a plugin process. It obviates the need for any system resources, and anyone can write code that talks to stdin/stdout.

I made a go package specifically for this here: HTTPS://github.com/natefinch/pie. Obviously that's go, not rust, but it may help explain the idea.. And if you do go in that direction, people who wanted to make a plugin in go could easily use that package.

[raphlinus]

Yes, I meant stdin/stdout specifically, for the reasons you state: to make it super easy and low friction.

I'll look at the pie stuff - at first glance, it looks pretty similar to what I was thinking about. What I'd love to do is have a protocol so simple you can write a single-task plugin with 30 lines of code or so and no external libraries. But then there probably will be a library that handles async behavior in a more sophisticated manner, like providing caches and managing invalidation for you. Go will be a high priority for such a library because of its combination of high performance and low developer friction.

[developit]

FWIW, I've used a plugin architecture nearly identical to that PIE example a number of times, and found it scales well (in terms of complexity). Particularly if you simply standardize on JSON-RPC, since libraries for that are near-universally available.

[natefinch]

Ahh, awesome. Yeah, if you want to write a plugin in Go, you don't even really need pie... it's only a few lines of code.. it's the pluggable application code that is really more complicated, and the plugin side is just there for symmetry.

This is a complete go program that will create a json rpc service over stdin/stdout, with service name "Plugin" and a method "Plugin.Hello".

package main

import (
    "io"
    "net/rpc"
    "net/rpc/jsonrpc"
    "os"
)

func main() {
    s := rpc.NewServer()
    s.Register(Plugin{})
    s.ServeCodec(jsonrpc.NewServerCodec(rwCloser{os.Stdin, os.Stdout}))
}

// Plugin's methods become RPC calls.
type Plugin struct{}

func (Plugin) Hello() string { return "hello world!" }

// rwCloser just merges a ReadCloser and a WriteCloser into a ReadWriteCloser.
type rwCloser struct {
    io.ReadCloser
    io.WriteCloser
}

func (rw rwCloser) Close() error {
    err := rw.ReadCloser.Close()
    if err := rw.WriteCloser.Close(); err != nil {
        return err
    }
    return err
}

[developit]

Yup. Roughly the same in node (though I'd have to go check the registry bit):

import rpc from 'rpc-stream';
process.stdin.pipe(rpc({
  Hello: cb => cb('hello world!')
})).pipe(process.stdout);

[raphlinus]

I just took a closer look at json-rpc. It's obviously pretty similar to what I'm already doing. The binary framing is already causing issues in non-binary-clean transports (see issue #29). I'm open to the idea, esp. for plugins.

One thought: 2.0 looks problematic because it defines strict server and client roles, and the client can't send requests to the server. I think that's very limiting for plugins. So maybe just standardizing on 1.0 is the answer here.

The idea of having plugins start from a tiny amount of code because there is already an rpc implemention is compelling.

[developit]

Agreed regarding fluid client/server. Otherwise you would end up using notifications to essentially tunnel RPC over RPC, which seems backwards.

[natefinch]

I can't actually think of when the plugin would need to initiate a request. When the main program loads the plugin, it can call an endpoint to get the plugin's configuration (what events it wants to hook into, etc). After that, it's all the main program calling methods on the plugin to give it a chance to react to events.

[raphlinus]

There definitely are cases, for example the plugin could be watching for external changes. But it's not just that, the plugin has to be able to do stuff too, like add spans, edit the text, or pop up a dialog.

Having slept on this, I don't think going back JSON-RPC 1 is the best thing, I can mix and match a bit, and say that the syntax is 2 but that both directions are supported. Keep in mind, this is stdin/stdout, not http, so conforming to an exact existing spec doesn't seem as important - it should be possible to find and adapt some existing code quickly.

Btw, I poked around at the go rpc library, and I'm not sure it can be used out of the box, though it's close - among other things, it enforces a Service.Method syntax (dot included), which I'm not sure is appropriate. But I think the amount of go code needed is pretty small.

Btw, people on this thread will probably be interested in my notes on plugin architecture, #43.

[raphlinus]

Closing this, as the plugin architecture is landing, and it's based on json-rpc over stdin/stdout, pretty much as suggested.