Table of Contents

• Features
• How it works
  • Headers
  • The Function
    • In Normal mode
    • In HTTP mode
    • In HTTP+HTML mode
  • Return value
• HTTP handler
• Installation
  • Docker container
  • NixOS service
  • Building it
• Clustering
  • Adding Scripts
• Command line flags
  • Cli options
    • Commands
    • dev
    • devhttp
    • Command line options
  • Server options
• Executors
  • Lua
    • Plugin system
    • Libraries
    • Web "framework" library
  • WASM
  • Podman
• Contributing
• Support

TLDR: msgscript is what you could call a poor man's Lambda-like function/application server. It can run functions and even some small web based applications.

Features

• Single binary
• Nearly no overheads
• Good enough performances (RTT of around 10ms for the hello example)
• Runs Lua functions, WASM binaries and Podman containers (no docker)
• Can use reusable libraries in Lua
• Can add Go based plugins in Lua
• HTTP handler
• Script storage either as flat files or in etcd
• Can be scaled up with multiples instances through locking (requires etcd)

How it works

Let's start with an example script:

--* subject: funcs.hello
--* name: hello

function OnMessage(subject, payload)
    local response = "Processed subject: " .. subject .. " with payload: " .. payload

    return response
end

For more technical details, a flow chart explains what happens when a message is sent and the server sees a match and than executes a script.

Headers

The headers are formed with the pattern of --* <header>: <value>. There are multiple possible headers:

• subject: The subject the script is associated with
• name: The name of the script. Multiple scripts can be associated with the same subject
• http: Used to return HTML responses
• require: Used to load a library script. It comes from the library "repository" of scripts and is prepended to the script that will be executed.

Each script is a Lua file that gets executed when the server receives a message that matches a pattern. The pattern is defined in the subject field. The files also contains a name field. Multiple scripts can be associated with the same subject.

The Function

In Normal mode

Normal mode is the default mode. The server executes the script when the message matches the subject and name fields. The function called is named OnMessage(). That function is fed 2 arguments, which can be named whatever you want. The first is the subject and the second is the payload. The payload is what the originating NATS message contains.

It's possible to call this mode both through NATS or with the HTTP handler.

In HTTP mode

Example, for a GET request:

--* subject: http.hello
--* name: http_get
--* http: true

function GET(url, body)
    return "Hello, " .. body .. "!", 200, { ["Content-Type"] = "text/plain" }
end

The function executed will have the same name as the HTTP verb of the originating HTTP request.

This mode is only available with the HTTP handler.

In HTTP+HTML mode

If you want to use the HTTP handler and want to return HTML, you can do so by setting the http header to true.

Just like in HTTP mode, the function executed will have the same name as the HTTP verb of the originating HTTP request.

Return value

The function is expected to return a string. If it does not, the server will log a warning: Script returned no response.

In NATS mode: it will return the string as-is.

In HTTP mode: it will return the following JSON document:

{"calculate":{"http_code":0,"error":"","http_headers":{},"is_html":false,"payload":"SGkhIEknbSBlbmNvZGVkIQo="}}

Each keys at the root level is named after the name field of the script. The value is a table with the following keys:

• http_code: The HTTP status code. It defaults to 200
• error: If the script returns an error, it will be set here
• http_headers: A map of HTTP headers
• is_html: Whether the response is HTML
• payload: The payload of the message. It is base64 encoded

In HTTP+HTML mode: you can return 3 different values:

• The HTML as a string
• The HTTP code (200 is missing)
• The HTTP headers (empty if missing)

HTTP handler

If you have a application that cannot reach nats by itself (say a webhook), it's possible to use the included HTTP handler.

The server listens to port 7643 by default (it can be changed through the command line). You can push messages by doing a POST request to http://serverIP:7643/<subject> where the <subject> is any subjects that you have scripts registered to it.

Example using curl (if you are running locally and for the subject of the example above):

curl -X POST -d 'John' http://127.0.0.1:7643/http.hello

Installation

Docker container

A container is avaible through the ghcr.io container registry. You can use it as so:

docker pull ghcr.io/numkem/msgscript:latest

It takes either server or cli as a first parameter.

NixOS service

You can enable it in your NixOS configuration using the provided module (once included from either the flake or importing the module using something like niv or manually):

services.msgscript.enable = true;

The options are defined in the nix/modules/default.nix file.

Building it

Being a standalone Go binary, you can build each of the binaries like so:

# Clone the msgscript repository
git clone https://github.com/numkem/msgscript.git
cd msgscript
go build ./cmd/server # Generates the server binary
go build ./cmd/cli    # Generates the CLI binary

It requires the btrfs headers (podman), gpgme (podman) and wastime (wasm) as dependancies.

Clustering

When msgscript is running in cluster mode, it's possible to use it with etcd. You can use the etcd backend to do that.

The cli binary provides some additional commands to manage the scripts stored inside etcd.

Adding Scripts

You can add Lua scripts to etcd using the msgscriptcli command. Here's an example:

msgscriptcli add -subject funcs.pushover -name pushover ./examples/pushover.lua

This command adds the pushover.lua script from the examples directory, associating it with the subject funcs.pushover and the name pushover.

The -subject and -name flags are optional. If they are not provided, they will be read through the headers contained in the file.

Command line flags

Cli options

The cli binary helps with managing the scripts in the message store and for helping with developing scripts.

Commands

add Add a script to the backend by reading the provided lua file completion Generate the autocompletion script for the specified shell dev Executes the script locally like how the server would devhttp Starts a webserver that will run only to receive request from this script help Help about any command lib library related commands list list all the scripts registered in the store rm Remove an existing script

The commands that manages scripts (add, list, rm) are not really useful when using the file base store.

dev

Useful for developing scripts that aren't http based (webhooks).

First argument is the script in question. It will return the script's output.

devhttp

Useful for developing scripts that are http based (webhooks).

First argument is the script in question. You can then reach your script at http://localhost:7634/<subject>/. The script is reloaded from the store on every HTTP request so you don't have to restart the command each time.

Command line options

Flags: -b, --backend string The name of the backend to use to manipulate the scripts (default "etcd") -e, --etcdurls string Endpoints to connect to etcd (default "localhost:2379") -x, --executor string Which executor to use. Either lua or wasm (default "lua") -h, --help help for msgscript -L, --log string set the logger to this log level (default "info") -u, --natsurl string NATS url to reach (default "nats://localhost:4222")

Server options

You can download the binary from the release page. There are 2 binaries available: server and cli. The server is what most people will want. The cli is only useful when paired with the etcd backend.

The server has the following options:

• -backend: The backend to use. Currently supports etcd or file. file is the default.
• -etcdurl: The URL of the etcd server. It can be multiple through a comma separated list.
• -library: The path to a library directory. It has no defaults. It can be an absolute path or a relative path.
• -log: The log level to use. The options are: debug, info, warn, error. It defaults to info.
• -natsurl: The URL of the NATS server.
• -plugin: The path to the plugin directory. It has no defaults. It can be an absolute path or a relative path.
• -port: The port to listen on. It defaults to 7643.
• -script: The path to a script directory. It defaults to the current working directory. It can be an absolute path or a relative path.

Executors

Msgscript supports many different executors or more simply, a way to execute a script. The following shows how to use them.

Lua

This is the default executor.

When writing Lua scripts for msgscript, you have access to additional built-in modules:

• etcd: Read/Write/Update/Delete keys in etcd source
• http: For making HTTP requests source
• json: For JSON parsing and generation source
• lfs: LuaFilesystem implementation source
• nats: For publishing messages back to NATS source
• re: Regular expression library source

these can be included using the built-in require() Lua function.

The following example shows how to deserialize a JSON payload:

--* subject: example.json
--* name: json
local json = require("json")

-- Assuming the payload contains: 
-- {"name": "John"}
function OnMessage(_, payload)
    local data = json.decode(payload)
    
    return "Hello, " .. data.name .. "!"
end

Some examples scripts are provided in the examples folder.

Plugin system

While there is already a lot of modules added to the Lua execution environment, it is possible to add more using the included plugin system.

An example plugin is available. The plugins can be loaded using the --plugin flag for both the server and cli.

Example using the hello plugin:

--* subject: example.plugins.hello
--* name: hello
local hello = require("hello")

function OnMessage(_, _)
    return hello.print()
end

Plugins currently included in this repository:

• scrape: An http parser source
• db: SQL access to MySQL, SQLite and PostgreSQL source
• gopher-lua-libs: Various modules from the gopher-lua-libs:
  • cmd
  • filepath
  • inspect
  • ioutil
  • runtime
  • strings
  • time

NOTE: The plugin file needs to have the .so extension.

Libraries

Libraries are Lua files that gets prepended to the script that needs to be run. These libraries can be used within other scripts using the require header like this:

--* require: foo

In this case, it will load the library named foo and prepend it to the running script.

Some example libraries are available here.

Web "framework" library

The web.lua library contains a very simple web framework. It's used in the example below:

--* subject: example.libs.web
--* name: web
--* require: web
local json = require("json")

local router = Router.new()

router:get("/plain", function(req, _)
    return "Hello, World!", {}, 200, { ["Content-Type"] = "text/plain" }
end)

router:get("/json", function(req, _)
    return nil, { name = "John" }, 200
end)

router:get("/path/<foo>", function(req, _)
    return [[
    <p>{{ param }}</p>
    ]], { param = req.params.foo }, 200
end)

router:post("/post", function(req, _)
    doc = json.decode(req.body)
    return [[ Hello {{ name }}! ]], { name = doc.name }, 200
end)

While not extensive, these examples shows how to use the library.

Namely:

• The function handling the endpoint returns 4 values: the mustache template, the data for the template, the HTTP code and HTTP headers
• If the template is either empty ("") or nil, it will be assumed that it's returning a JSON document. The Content-Type will be set as such.

WASM

Binaires compiled targetting WASM can be used. Some examples are provided in the examples/wasm directory.

The format required in the store looks like this:

--* subject: funcs.wasm
--* name: wasm
--* executor: wasm
/home/numkem/src/msgscript/examples/wasm/c/c.wasm

The import parts are subject, name which are common with all other executors. The executor key needs to be set to wasm. The content is the path to the WASM executable.

Podman

The format requires in the store looks like this:

--* subject: funcs.hello
--* name: podman
--* executor: podman
{
    "image": "hello-world"
}

Like for WASM, it takes the same important keys. The content of the file can be:

Key	Description
image	Container image name
mounts	List of mounts in the same format you would write them on the command line
privileged	true/false if the container should run with more permissions

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

If you encounter any problems or have any questions, please open an issue on the GitHub repository.