Lua Integration

rhis page details the integration of Lua with zippylog.

Lua Overview

Lua is a lightweight functional programming language. Two of its attributes that make it attractive to zippylog are that it is fast and it is designed to be embedded.

Lua interpreters are extremely lightweight (you can create over 100,000 per second in a single thread on modern hardware). Each interpreter is also completely independent. There is no shared state and no locking to disrupt thread concurrency. By default, each interpreter is enabled with only the core features of the language: the standard libraries have to be explicitly enabled. This means abilities like I/O, loading other Lua files, printing to output, etc are all absent by default. This is a desirable security trait. It means that a Lua interpreter can take Lua source code from an untrusted source (e.g. the user) and run that code without worrying too much since the Lua interpreter is limited in abilities. The worst thing malicious code can do is allocate tons of memory or chew up many CPU cycles, and the Lua C API provides a mitigation strategy for both (custom memory allocator and a callback executed every N instructions). zippylog processes that accept user-supplied Lua often institute limiting policies to curtail this abuse.

Use in Zippylog

Process Configuration

Configuration of processes is performed by executing Lua files and extracting specifically named global variables from them. It was easier to utilize Lua for this purpose than to parse argv or introduce a separate library (YAML, XML, etc) just for the purposes of process configuration.

Envelope and Message Processing

Lua performs many tasks related to envelope and message processing.

Searching and filtering is one such functionality. For these operations, a Lua function is defined. This function takes as input a protocol buffer message (typically a zippylog envelope) to be operated on. The function inspects the envelope and returns a result.

For server-side streaming of messages (see protocol), a client uploads a Lua function definition to the server. The server instantiates a new Lua interpreter and calls the user-supplied function for each envelope it sees. If the function returns true, the server sends the envelope to the client.

Lua interaction with protocol buffer messages is performed via the lua-protobuf binding, which is written by the same author as zippylog.

Server Processing API

Various client requests contain fields that can contain Lua code. If Lua code is specified, specifically named functions within can provide extra features for the request.

zippylog_get_options()

This function returns the functionality that your Lua code block defines. It returns a table that can have the following string keys:

• enveloper_filter - function name string or function that defines a filter for envelopes. If the defined function returns true, the envelope passes the filter. If not, the envelope doesn't pass the filter. If not defined, zippylog looks for the global function zippylog_envelope_filter.

If this global function is not defined, zippylog will use all default values for all table elements. In many cases, if a function does not exist, functionality will be skipped.

zippylog_enveloper_filter(envelope, properties)

This function (or whatever function you define via zippylog_get_options()) serves as a filter for received envelopes. The first argument is a zippylog envelope protocol buffer message type. The second argument is a table with metadata describing the envelope. This table may have the following keys:

• bucket - Bucket the envelope belongs to
• set - Stream set envenlope belongs to
• stream - Stream envelope belongs to
• path - Stream path

If the function returns true, the envelope passes the filter and it will be delivered to whatever is next in the chain.