RFC: Script Loader

[ccollie]

RFC: scriptLoader

A replacement for the script loading mechanism in bullmq

See WIP branch here and tests here

Why ?

To allow modularity and composability of lua scripts, with recursive dependency handling and path mapping.

Example

isNil.lua

local function isNil(x)
  return x == nil or x == cjson.null
end

string.lua

--- @include "isNil"
local function safeLen(x)
  return not isNil(x) and #x or 0
end

math.lua

--- @include "isNil"
local function sign(x)
  if isNil(x) then return nil end
  if (x == 0) then return 0 end
  return x < 0 and -1 or 1
end

utils.lua

--- @include "string"
--- @include "math"

app.js

const script = await loadScript('path/to/utils.lua')

result

local function isNil(x)
  return x == nil or x == cjson.null
end
local function safeLen(x)
  return not isNil(x) and #x or 0
end
local function sign(x)
  if isNil(x) then return nil end
  if (x == 0) then return 0 end
  return x < 0 and -1 or 1
end

All children are loaded transitively and in dependency order to any depth.

Path mapping is supported, so we can do the following:

utils.lua

--- @include "<includes>/math"
--- @include "<includes>/string"
--- @include "<includes>/jobs"

app.ts

addScriptPathMapping('includes', 'relative/path/to/includes');
const script = await loadScript('utils.lua');

The ~ character at the beginning of an include maps to the project root (location of the project's package.json)

For convenience, we can do glob includes

utils.lua

--- @include "<includes>/util-*.lua"

API

loadScript() function

Signature:

export declare function loadScript(filePath: string, cache?: Map<string, ScriptInfo>): string;

Parameters

Parameter	Type	Description
filePath	string	path to the top-level script we want to load
cache	Map<string, ScriptInfo>	an optional map to return the metadata or all files gathered to fulfil the request

Returns:

string

The interpolated script

addScriptPathMapping() function

Add a script path mapping. Allows @includes of the form <includes>/utils.lua where includes is a user
defined path.

Signature:

export declare function addScriptPathMapping(name: string, relativePath: string): void;

Parameters

Parameter	Type	Description
name	string	the name of the path mapping
relativePath	string	the path relative to the caller of this function

Returns:

void

[roggervalf]

Looks pretty good 🚀, do you want to open a pr for this?. cc: @manast

[ccollie]

@roggervalf Will do. Just give me a day or so to cleanup

[manast]

This looks indeed amazing and would help us a lot in refactoring all the lua scripts we have.

[ccollie]

Had some git issues. I'll push the PR tomorrow.

[roggervalf]

Closing it as the feature was released already 🚀