Extensions System

[avelino]

Looking to improve how it is possible to "extend" pREST we are starting an extension system using the Go Language Plugin package that allows to load operating system libraries (.so or .dll).

This way it is possible to develop pREST extensions in any language that generates .so, example:

• Go
• C
• C++
• Rust
• …

But today it is not possible to extend pREST as a framework?

Yes (we have in the documentation how to do this), but my feeling is to teach people how to hack pREST, it's created "a new software" using the pREST packages, i.e. not necessarily using the pREST product.

The proposal of the extension’s ecosystem is to support extending pREST without modifying the distributed binary (prestd') and prestd' loading extensions configured to extend specific parts of pREST.

Will framework support be discontinued?

No, if you want to use pREST as a framework to develop APIs (or any web software) feel free - the MIT license allows you to do so. But we won't be able to support what's outside our domain (what was developed by you or your team).

What can be extended?

Some points that I think it is important to extend:

• Endpoints: create custom endpoints that are part of your API domain
• ETL in existing pREST endpoints: it is possible to change the return data objects (tables, views, functions, etc)

Subject that I think is important to have your contribution and I will return the question to you: what would you like to extend by writing an extension?

How would the extension be created?

Here would not be the best place to talk about implementation, but follows an example of how to use the Go plugin package:

Plugin with filename exts/hello.go

package main

// Hello plugin
func Hello() (ret string) {
	ret = "Hello plugin caller!"
	return
}

It must be compiled using buildmode=plugin:
go build -o exts/lib/hello.so -buildmode=plugin exts/hello.go
will generate a hello.so file.

Loading the .so file into main.go:

package main

import (
	"fmt"
	"plugin"
)

func main() {

	p, err := plugin.Open("./exts/lib/hello.so")
	if err != nil {
		panic(err)
	}

	f, err := p.Lookup("Hello")
	if err != nil {
		panic(err)
	}
	ret := f.(func() string)()
	fmt.Println("ret plugin:", ret)
}

Running main.go:

❯ prestd (master) ✘ go run main.go
ret plugin: Hello plugin caller!

Example of generating OS Library ".so" in:

• Java

Draft: #475

[sebastianwebber]

what about using webassembly ? Envoy do something similar: https://containerjournal.com/features/extending-the-envoy-proxy-with-webassembly/

[DiegoSantosWS]

I liked the idea and think it's good to think of webassembly

[felipeweb]

@avelino go server can run webassembly file using something like https://github.com/wasmerio/wasmer-go.
but I'm in favor of https://github.com/hashicorp/go-plugin.

Why?

With go-plugin we can use the power of GRPC to provide an extension. here we have a docs about how to use without Go.

How do I think this feature?

Download plugin -> Start plugin as an internal process-> establish the RPC communication -> provide the new feature as a HTTP endpoint making a call to the plugin behind the scenes.

[tiaguinho]

I agree with @felipeweb.
I think that create plugins using RPC communication is the best way to go.
Using RPC will also make pREST ready to be used as a microservice inside the company environment.

[avelino]

I see opportunity for both solutions, companies with teams of engineers with much knowledge software development the solution with RPC will be the best option

For companies that do not have teams with low engineering knowledge the RPC will be a closed lock, so thinking about giving a formula to develop plugins and versioning the code in 1 git single repository (Wordpress wing)

[avelino]

With an operating system library .so it is possible to distribute ready-made extensions (marketplace), making it possible to install extensions distributed by other users - for open source project we are fostering the community

[avelino]

I have implemented the first version of the plugin system based on an operating system library (.so), #659

Following Felipe's proposal (which was sensational): opened an issue to implement the plugin via grpc protocol #662