Implement plugin system

[mthenw]

The goal of plugins is to extend the behavior of the Event Gateway core. Examples include authentication, integration with external identity management systems, event validation systems, event persistence.

They should be implemented in any language as a local sidecar that adheres to the plugin calling interface. It listens on a local TCP socket for a nice mix of interoperability and performance.

[zdne]

@mthenw is currently any sort of the middleware functionality implemented?

[xtuc]

Just my two cents.

I think the TCP-based solution suffer from a few issues:

• A TCP socket is by definition statefull.
• The network is unreliable (especially cloud provider ones).
• If you use multiple Docker containers with the bridge driver, it will introduce a big overhead when talking through the network.
• People could sneak in the network and sniff the plugin's traffic.

I would use an executable-based solution, which has the following advantages:

• Communitcations are through IPC which is available in all languages.
• No latency.
• Stateless.
• Unix permissions on files.
• Mount the file as a volume (open mode is even configurable).
• NPM-like plugin system.
• administration is much simpler.
• With Docker, inter-container IPC is possible.

Or shared Unix sockets, basically same as the executable-based solution. The plugin author can expose a high-level API such as REST.

Note that you can still implement a TCP server on top of the executable-based solution.

Imagine you need a unique instance of one plugin across your cluster. While the TCP based solution is possible, it will suffer from all the issue we have in distributed systems. Ideally the plugin author shoudn't have to care about the system topology.

Since you'll need proper communication for your state within your cluster, I think the plugins should be part of that. I could imagine at least two master nodes in the cluster managing all the stateful plugins and making sure the information were correctly transmitted.

I'm curious to hear what you think about that approach.

[mthenw]

@zdne nothing is implemented yet. Though it's high in our backlog https://github.com/serverless/event-gateway/projects/2#card-4080295 so we should start implementing this pretty quickly. Could you write more about your use case or about your expectations from plugin system?

[mthenw]

@xtuc thanks for your input. My current thinking is that a plugin needs to run on the same host so there are no issues around unreliable networking. EG should be responsible for running plugin process so there is no need to run it in e.g. separate Docker container. Plugin same as EG instance has to be stateless. Communication could be done via some RPC (gRPC?). What do you think?

[xtuc]

I'm not a fan of the TCP socket for local communication only. There are still a few issues with the network and security (You could accidentally open the server to the outside. localhost could be resolved to another IP than the loopback 😄. The network interface could be misconfigured. Etc). Do you absolutely need it to be through some network?

RPC/RMI frameworks are quite different in the format they use for communication. You cannot really take the reference RPC framework of your language and use it in the plugin system out of the box.

I forgot to mention it but with the executable-based solution you could talk over stdin/stdout. In that case I would use a JSON stream (JSON objects separated by new lines).

[mthenw]

Summary

Major use-cases:

• supporting a different kind of authorization mechanisms
• persisting events to some durable storage (e.g. exporting every request to S3)
• validating events

Technical requirements:

• ability to write plugins in different languages
• plugins failure should not cause EG instance failure
• calling plugin should be fast (no remote communication)

Golang has its own plugin system but it forces to write plugin in golang and those plugins work only on Linux (no Windows support).

I think that the best solution is to use Hashicorp's plugins system for golang that they use in their products. A plugin is run by the gateway on the same instance. Plugin communicates via grpc on localhost (fast & reliable). In case of running EG in multiple containers, each instance has a copy of the plugin.

Benefits:

• used by production-grade systems (Vault, Nomad)
• plugins can be written in any language supported by gRPC (python, ruby, nodejs, java, c#, php and more)
• existing examples - should be easy to start

Implementation

Event Gateway needs to expose the following system events:

• gateway.event.received - when event was received by Events API
• gateway.function.creating - before config change
• gateway.function.created - after config update
• gateway.function.invoking - before invoking function
• gateway.function.invoked - after invoking function
• gateway.function.invocationFailed - rename existing gateway.info.functionError
• gateway.subscription.creating - before config change
• gateway.subscription.created - after config update

and allow plugin so subscribe to those events in two ways, blocking and non-blocking. Blocking/sync means that we need to get a response back to proceed, non-blocking/async means that plugin response is ignored.

Interface to implement by a plugin:

• Subscriptions() returns []PluginSubscription - EG calls this method during startup and subscribe plugin to listed events
• Invoke(event Event) returns error - EG calls it for every event that plugin subscribed to.

Model

PluginSubscription{
  Event string,
  Type SubscriptionType // can be Async Or Sync 
}

To enable plugin event-gateway needs to be run with new CLI param -plugin=<path to plugin>. This param can be used multiple times.