Plug-and-play plugins

[bobheadxi]

Current plugins require code to actually be committed to ubclaunchpad/rocket. Following #84 and the option for a generic self-hosted Rocket, this won't be a very good experience for plugins.

What if you could actually drop binaries in as plugins?

https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

Tucked among the multitude of significant features introduced in Go version 1.8 is a new Go plugin system. This feature let programmers build loosely coupled modular programs using packages compiled as shared object libraries that can be loaded and bound to dynamically at runtime.

Example: https://github.com/vladimirvivien/go-plugin-example

Pretty neat! Thoughts @bfbachmann ?

[bfbachmann]

Yea this looks nice. It could also allow us to enable and disable plugins on the fly. We'd still have to build the image with all the plugin files in it, though.

[bobheadxi]

They can be mounted in as well, for any plugins that aren't included in the core rocket image. One way might be to mount a config file:

# rocket.toml
[[plugin]]
    name = "welcome"
    binary = "welcome.so"
    # other settings?
[[plugin]]
    name = "game"
    binary = "game.so"

And mount everything:

sudo docker run -d \
    -v "$(pwd)"/config:/app/plugins \
    -v /binaries:/app/plugins \
    ubclaunchpad/rocket $ARGS

Then based on the config file, rocket could attempt to load the plugins on startup

[bobheadxi]

Even cooler, maybe that config file could specify a download link and the Rocket image will automatically download and use the specified plugins 😮

[bfbachmann]

Yikes! That sounds like remote code execution! I definitely would not run any plugins on Rocket that I had not reviewed myself.

[bobheadxi]

That's fine - it's an absolutely manual thing, and you would only add plugins you trust and actually want to use, hence the configuration file.

For example, if some org decides to use Rocket, they would probably want to make a bunch of internal plugins - in that case, download + load would be fine, because they know what they're getting.

[bfbachmann]

If you're talking about just mounting/adding plugin files to Rocket's container then that's reasonable. I thought you meant that Rocket would download and run plugins at runtime.

[bobheadxi]

Rocket would download and run plugins at runtime.

I'm noting that it's a possible feature, to make plugins easier to share, but only if the user specifically configures rocket to do so via the hypothetical config file, and not by default.

[bfbachmann]

I guess I just don't see the point in Rocket downloading stuff at runtime. If you have the means to develop plugins, host a server that serves those plugins securely to Rocket and deploy and configure your own Rocket instance then you surely have the means to just update and restart Rocket? In fact, I'd say just updating Rocket with the plugin files and restarting the container would be easier than hosting plugin files that Rocket downloads. Plugins are fine, but the download stuff just seems unnecessary - it introduces extra security concerns for us and users and it's more complicated than the alternative.

[bfbachmann]

Also, you don't need Rocket to download plugins to make them easier to share. They're just files! Send them to your friend, or use Git!

[bobheadxi]

If you have the means to develop plugins, host a server that serves those plugins securely to Rocket and deploy and configure your own Rocket instance then you surely have the means to just update and restart Rocket?

It's not meant to be a replacement for manually loading plugins by mounting binaries, but an alternative with a better user experience.

Would you rather:

I've made an amazing plugin! To use this plugin, just add x line to your config and go! 🚀

or:

I've made an amazing plugin! To use this plugin, just download my thing, make sure my thing is mounted to the right place, add it to your config, then... go 😢

I don't see the security concerns with rocket loading plugins for you - the onus is entirely on the user to choose trustworthy plugins, which they have to do anyway if they download manually.

As I pointed out in the other comment that we'll likely need a configuration file format anyway to enable this well, so why not support more ways to load plugins?

[bfbachmann]

After having given this more thought, here are the drawbacks I see in compiling Rocket plugins to shared object files that are loaded by Rocket at runtime:

1. If you're developing a plugin like this you have to compile the plugin anyway, so it's no less work than just adding the plugin to Rocket and re-compiling Rocket as we currently do. Moreover, when you add a plugin you have to put the .so file somewhere Rocket can find it, and point to it in Rocket's config. Then you'd need to restart Rocket so it loads the new config.
2. Loading plugins from .so files introduces a whole new set Rocket config which can be broken in it's own special ways. The plugin config file could be wrong, or loading plugins could fail in other ways at runtime (you also have to do a bunch of ugly type-assertions and the like). You have no guarantee that the plugin is even compatible with the Plugin interface since you're compiling your plugin separately from Rocket, and in the case that your plugin doesn't fit the interface, you only find out at runtime rather than at compile time.
3. The point in Rocket plugins is that they're modular and they don't require much knowledge of Rocket to build/maintain. We don't actually gain anything meaningful from using plugins in this way - they are still just as modular as they always were. I think the guy who wrote that article you reference got the wrong idea of Go plugins. I feel like, for the most part, if you "have relied on multiple out-of-process designs such as OS exec calls, sockets, and RPC/gRPC (and so on) to achieve code modularization" you don't know how to write good code. You definitely don't need to compile modules to shared object files and then dynamically load them at runtime to achieve modularity. That seems like an insane argument to me. Plugins (as implemented by go) are used by OS’s to have one shared instance of something like libc, not dynamically loading components at runtime.
4. You mention that loading plugins this way could be an alternative to the current method rather than a replacement, and you say "why not support more ways to load plugins?". I think that having two methods of handling plugins is unnecessarily complex and would undoubtedly confuse contributors. It's more code to maintain that doesn't do anything new and it needlessly introduces a new class of Rocket failures.

[bobheadxi]

If you're developing a plugin like this you have to compile the plugin anyway, so it's no less work than just adding the plugin to Rocket and re-compiling Rocket as we currently do. Moreover, when you add a plugin you have to put the .so file somewhere Rocket can find it, and point to it in Rocket's config. Then you'd need to restart Rocket so it loads the new config.

Imagine if say, Slack was a self-hosted service, and the docs say "right if you want plugins you gotta fork the source code, add your code, and recompile this whole thing"

Loading plugins from .so files introduces a whole new set Rocket config which can be broken in it's own special ways.

By that logic, no application should ever allow configuration!

The plugin config file could be wrong, or loading plugins could fail in other ways at runtime (you also have to do a bunch of ugly type-assertions and the like)...

Not on us - as with any plugin, it's on the user.

The point in Rocket plugins is that they're modular and they don't require much knowledge of Rocket to build/maintain.

By literally requiring you to have the entire Rocket codebase, you've already invalidated any legitimacy to the "modular" claim, and to some degree the "don't require much knowledge of Rocket to build/maintain" claim as well, since you'll have to dig in and attach your plugin to the various entrypoints.

We don't actually gain anything meaningful from using plugins in this way - they are still just as modular as they always were

As I noted in the original issue, this is for other users of Rocket. Are you seriously going to wait for a PR approval to a tool's parent repository in order to use a plugin only your organization will use? And requiring you to fork to build your plugin + Rocket from source is a truly awful experience.

You definitely don't need to compile modules to shared object files and then dynamically load them at runtime to achieve modularity.

Yes you do! Otherwise it's not modular! Think about any application you use, and now imagine having to build the entire codebase yourself in order to add a small plugin - that's no longer a "plugin", thats literally just modding the source code.

You mention that loading plugins this way could be an alternative to the current method rather than a replacement, and you say "why not support more ways to load plugins?". I think that having two methods of handling plugins is unnecessarily complex and would undoubtedly confuse contributors. It's more code to maintain that doesn't do anything new and it needlessly introduces a new class of Rocket failures.

This isnt for contributors, this is for users. Not everyone wants the same plugins. Not everyone wants to PR to add a plugin only they will use - in fact, very likely no one wants to. This ticket is an extension of #84 with the hopes of extending Rocket's capabilities to more general use cases, which is a great way to potentially draw a wider audience for this project - but first we need to consider what it's like to deploy Rocket with custom functionality. As it is right now, the plugins aren't really "plugins", theyre just part of the source code.

We don't actually gain anything meaningful from using plugins in this way - they are still just as modular as they always were.

Think about this - is it really modular if you need to change source code? I'd say absolutely not.

[bobheadxi]

I feel like, for the most part, if you "have relied on multiple out-of-process designs such as OS exec calls, sockets, and RPC/gRPC (and so on) to achieve code modularization" you don't know how to write good code.

I feel this is a bit of a ridiculous notion as well - that is a perfectly valid way to write code IMO if you feel like you need the kind of flexibility a monolithic codebase cannot possibly provide... for example, for extensions and plugins maybe? Imagine having to unscrew your laptop in oder to use a USB-powered DVD player, or to connect to a webcam!

Now that I think of it, USB-level modularity is the kind of modularity you should strive for when you claim something is "modular" and is a "plugin" tbh. Whether you do this through this approach (.so binaries), or through OS exec calls, sockets, and RPC/gRPC, or through REST APIs or websockets... doesn't matter, it just actually has to be modular.