Lunatik v5.0

Lunatik v5.0

Lunatik 5.0 brings eBPF into Lua scripts, with pinned maps and Traffic Control and sched_ext programs that call Lua, along with filesystem notification, a netlink stack written in Lua, per-CPU runtimes and a bytecode compiler. A review of every module fixed crashes, hangs and leaks, made interrupt-context runtimes refuse what would sleep, and made the APIs consistent across modules; scripts written for 4.4 need the changes listed under Changes to existing APIs.

Highlights

• eBPF maps: bpf opens a pinned eBPF map with one constructor per map type, and bpf.map turns it into a table a script indexes, assigns and iterates, or into a queue with push, pop and peek.
• Traffic Control: with tc, a TC/eBPF program calls Lua at the ingress and egress layers of the network stack; examples/sniclassify classifies egress traffic by its TLS server name.
• sched_ext: with scx, a sched_ext program calls Lua for task scheduling.
• Filesystem notification: fsnotify places marks on files, directories and mounts and hands their events to a Lua callback, permission events included; examples/fsmonitor logs what changes in a directory, and examples/execguard refuses an exec whose name is not on an allowlist.
• Netlink in Lua: the netlink namespace speaks generic netlink, routing (rt: links, addresses, routes and rules) and nl80211 (interfaces, stations, access points) from a script, and netlink.channel multicasts to user space from softirq; examples/linkflap and examples/netfailover react to links going up and down.
• Per-CPU runtimes: lunatik.percpu and lunatik run --percpu create one runtime per CPU sharing one registration, so a hook runs on the CPU it fires on.
• Bytecode: lunatik compile builds chunks the kernel loads, with the luac of the kernel's own Lua configuration, for a target of either byte order.
• More modules: task for the Linux task interface, struct for fixed C struct layouts, and class for object-style Lua modules.
• Lua 5.5.1.
• New examples: dropreason answers "why is my packet dying?" from the kernel's drop reason, and the six above.
• Documentation: a site at https://luainkernel.github.io/lunatik, with a guide, the reference of every module and the C API.

Fixes

• Crashes: skb:forward() panicked on a packet whose MAC header sits past skb->data (#1291); skb:data() in a tc callback could free the head the calling BPF program still points into (#1369); an object left in lunatik._ENV outlived its module and was released into freed module memory at unload (#1390); thread:stop read a runtime its body's end released (#1428); a view of a copy's data outlived the copy (#1275); a runtime could clone lunatik_env after its release (#1387); a recursion through C ran past the kernel stack, and now raises "C stack overflow" (#1669).
• Sleeping in interrupt context: once armed, a softirq or hardirq runtime refuses a require that opens a file or loads a binding its body did not load (#1282, #1729), require("io") (#1421), linux.netns (#1283), linux.schedule (#1194), hid.register (#864), darken.run (#1256) and a netfilter registration (#1276), and a monitored method allocates with its object's gfp under the object's lock (#1487).
• Callbacks: probe and notifier callbacks get their arguments inside a protected call (#1188); netfilter no longer leaks a packet a callback answers with STOLEN, REPEAT or STOP (#1280); a hid callback that raises no longer leaves its data object on the kernel's buffer (#1361); a callback's errno from the kernel's internal block no longer reaches io_uring (#1737).
• Leaks and warnings: aead frees its output buffer when pushing the result fails (#1759); darken refuses a ciphertext past kmalloc's largest block without the allocator's warning (#1744); netfilter refuses a family or hook with no hook table instead of reaching a kernel warning (#1745); tostring of a shared data object reads it under its lock (#1504).
• Values: skb:data("net") and skb:data("mac") are two views, so taking one no longer moves the other (#1289); autogen's unsigned 32-bit constants reach Lua without sign extension (#1285); netlink.rt.addr's address is the local one, and a point-to-point address reports its peer (#1286).
• CLI: lunatik reload unloads every loaded module, and refuses to replace a module loaded from another build (#737, #1381).

Before upgrading

• Lunatik supports Linux 6.6 and later; the 5.x kernel line is no longer built.
• Unload the previous release with its own CLI before installing 5.0, or reboot: a driver loaded by a release before 5.0 fails every command of the 5.0 CLI with loaded from another build.
• 32-bit ARM is not supported in 5.0: the OpenWrt build fails on missing division helpers (#418), and autogen drops the linux.* constants there (#1349).
• On Linux 7.1 and later, luacrypto.ko loads only on a kernel that builds a user of the crypto rng API, such as CONFIG_CRYPTO_DRBG (#1415).
• xdp and tc built without the running kernel's BTF (make btf_install before make) load with their kfunc unregistered and missing module BTF in the log, and an eBPF program that calls the kfunc then fails to load (#1607).

Changes to existing APIs

• Outcomes and failures (#1301, #1327): an expected outcome, a timeout, a full queue, nothing to read or an absent device, returns nil, with the errno's name as a second value when the kernel gave one, or false; a failed kernel call raises the errno's name.
• Hook decisions (#1296, #1298): a hook callback's decision is its return value in every hook, xdp, tc and sched included; a device or hid callback fails an operation by returning a negative errno.
• stop, close and __close (#1313): a registration stops with stop and a resource closes with close, both idempotent, and every such handle offers __close; netfilter and hid gain stop. Dropping a handle stops nothing, a child runtime whose script registered a hook included (#1042).
• No positional booleans (#1302): runner, probe and socket take a table of options where a boolean changed a call's behaviour or arity.
• Waits (#1757): a wait's timeout is a millisecond count from 0 to INT_MAX, or none for no bound.
• Callbacks (#1754, #1758): device, hid and probe refuse a callback field that is not a function, and a callback's errno from the kernel's own block is refused and logged.
• Class names (#1299): a class is named by its module and type, crypto.aead, bpf.hash, lunatik.runtime, as a type error quotes it.
• Requiring (#1423, #1729): a binding is found before a Lua file of its name, the driver installs as lunatik/driver.lua, the top-level names Lunatik reserves are listed, and an armed interrupt-context hook requires only what its body loaded.
• netfilter (#1294, #1745): a hook registered without a mark sees every packet, and register refuses, with out of bounds, a family netfilter keeps no hook table for and a hook past its family's table.
• notifier (#1746): a callback's return reaches the chain only as a linux.notify code.
• skb (#1295, #1539, #1325, #1747, #1748): skb:data() starts at skb->data, data("net") at the network header and data("mac") at the MAC header, and a view whose header is absent is nil; resize answers whether it resized and keeps a checksum the stack still fills; mark, priority and connmark read when given nil; skb.attr is removed.
• probe (#1311): a handler receives the target and one regs object.
• hid (#1312, #1751): one table per device, the same in every callback, and a remove callback; the report a callback is handed is closed once it returns, also when it raises.
• socket (#1304, #1305, #1317, #1755): an AF_PACKET address in host order, the same shape in and out; socket.raw.new builds the socket and closes it when its bind raises; a value out of its range raises; socket.new answers nil and ESRCH for a pid no task holds, as linux.netns does.
• signal (#1324): kill only, with the pid resolved in the initial pid namespace.
• darken (#1323): an authenticated format, gcm(aes).
• cpu (#1309, #1749): cpu.maxid and the iterators cpu.possible, cpu.present and cpu.online; cpu.stats answers nil for a CPU that is not online.
• rcu (#1322): rcu.map is rcu.foreach, and its callback's return is ignored.
• crypto (#1321, #1750): rng:seedsize, and hkdf without a public hmac; aead:decrypt answers nil and EBADMSG for a message that does not authenticate.
• data (#1756): getstring answers an empty string, and setstring writes nothing, for a zero length up to the end.
• fsnotify (#1753): a mark ends with stop, which a second call and __close take as well.
• linux, thread, data, bpf (#1303): linux.difftime, thread.current, data:getnumber/setnumber and # on a bpf map are removed, and linux.ifaddr is linux.hwaddr.
• linux.* (#1297, #1308): one table per kernel family; each hook family's verdict table is action, and linux.bpf.map_type stands apart from the update flags.
• runner (#1314): list returns a sequence of names.
• util (#1320): the test harness moves to tests.lib, and the hex codec returns one value.
• fib is removed; netlink.rt.rule supersedes it.
• CLI (#1315): the CLI still reads the 4.4 command grammar, and the /dev/lunatik protocol is internal to it.

Known issues

Each is tracked in the issue it names, and the documentation of its module names the first five (#1727, #1752):

• A sched_ext callback runs under its CPU's runqueue lock, where print, a completion's complete or a mailbox's send can hang the CPU (#1718).
• A netdevice callback's STOP or BAD on an event the kernel does not veto stops the chain (#1739).
• An rcu table a probe handler writes deadlocks against a writer that holds it with interrupts on (#1740).
• A probe handler runs Lua in NMI, where its allocation can take a lock its CPU holds (#1741).
• A netlink channel used from a hardirq runtime reaches a warning through the nlmon tap (#1742).
• skb:data() and skb:resize() write into a head a clone shares (#1260).

Contributors

Lourival Vieira Neto (@lneto), Ashwani Kumar Kamal (@sneaky-potato), Harshdeep Singh (@Harshdeep-creator) and Prath (@prath47); @prath47 and @Harshdeep-creator made their first contributions in #654 and #741.

Full Changelog: v4.4.1...v5.0