Golang-based plugin system

[saitho]

This issue examines a structure for a new plugin system. This will supersede the modules system from StackHead v1.

Module: a self-contained piece of software (e.g. types, data structure, functions), that can be combined with other modules to construct a more complex software.
Plugin: a plugin is a ready to use software component that can be added to an existing software to change adds features.

Therefore, "plugin" is the correct term, thus we are talking about "StackHead Plugins".

Requirements

• StackHead executes the plugin.
• Two-way asynchronous method calls
  • StackHead needs to wait until asynchronous process triggered on plugin is finished ("await")
  • When plugin triggers asynchronous processes in StackHead, it needs to be able to wait for a response ("await")

Approach 1: Golang plugins

https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

• .so files need to be placed at a special location on the executing system (Ubuntu: ~/stackhead/plugins/)
• plugins are expected to be hosted on GitHub. We may provide a plugin search endpoint which is a cached GitHub repository results view with a repository tag
• two way asynchronity ????
• Disadvantage: Both StackHead and the plugin need to use exactly the same dependencies, i.e. the same commit of https://github.com/getstackhead/pluginlib.

Approach 2: Golang plugins + gRPC FAILED

• Idea: pluginlib provides Protobuf definitions which both StackHead and plugins generate and implement.
• While StackHead and plugins need to use the same major version, there is no connection to individual commits.
• However, version requirement are bound to protobuf/gRPC version to use, which is fine for now
  • dependency upgrades will cause breaking change in StackHead!
• Dealing with asynchronous callbacks is done naturally.
• This requires 1+n RPC servers to run (n is number of plugins)
• Disadvantage: No check if versions of Protobuf definitions are compatible
• To test: Compatible with Golang plugins? Idea: run RPC server in main(), pass port via symbol to StackHead

FAILED: gRPC dependencies are also contained in .so file => loading conflict as dependencies are loaded twice (or n+1-ice with n plugins)

Approach 3: Scriggo

• Idea: Use Scriggo to dynamically execute Go code from a directory
  • SECURITY: Allows arbitrary code execution if plugin dir is compromised. Security checks need to be added, e.g. comparing the hash of the files in the directory with hashes provided with the release on GitHub/GitLab/Gitea etc (go-releaser). Issue warning to user and require confirmation if checks are not successful or not possible (e.g when no release is available due to working on a branch)
• StackHead injects fake packages via Scriggo to allow the plugin to communicate with StackHead
  • pluginlib may provide mock methods to make plugin development easier via gomod file. However there is no dependency of StackHead or the plugin during runtime
  • Dealing with asynchronous callbacks is done naturally via Golang code
• Disadvantage: Custom Go dependencies cannot be used

Approach 4: gRPC

• Idea: pluginlib provides Protobuf definitions which both StackHead and plugins generate and implement.
• While StackHead and plugins need to use the same major version of the definition, there is no connection to individual commits.
• Dealing with asynchronous callbacks is done naturally.
• This requires 1+n RPC servers to run (n is number of plugins)
• Disadvantage: No check if versions of Protobuf definitions are compatible => responsibility of developer?
• StackHead Plugins would compile into a regular binary that needs to be executed on initial StackHead use
  • First start local StackHead RPC server on fixed port; then plugins which register at the server

[saitho]

Due to limitations on Go plugins, a StackHead plugin is required to use the same version (= even same commit) of the StackHead Plugin API as the StackHead version. That means a StackHead plugin needs to be built specifically for a StackHead Plugin API version. In order to reduce the number of releases and needs to rebuild plugins every time, the StackHead Plugin API is a separate repository: https://github.com/getstackhead/pluginlib

Using newer version of pluginlib would consequently mean a breaking change in the main repository to signalize plugins need to be upgraded to the new version and rebuilt.

[saitho]

The idea of having a plugin system is scrapped, as Go does not provide pretty nice ways to do so.
The Scriggo-way worked, but was limited to one main file.

Instead, the modules will be included in StackHead itself.