Integrating Zeek with systemd
When using Zeek's supervisor framework, Zeek acts as a basic service manager. This puts it in conflict with other service managers on the system. On Linux systems, the most common of these today is systemd. This page collects ideas for aligning/integrating the supervisor with systemd, which'll pay off in the forthcoming, supervisor-driven cluster management framework.
This page is not about running current Zeek clusters under systemd. This is relatively straightforward to do, e.g. by defining each node in the cluster as a separate service, and tying them together via a target unit.
We will never require systemd for running Zeek. Not only do people run Zeek in production on platforms other than Linux, at also aligns poorly with more lightweight container-driven deployments.
Scopes are a type of systemd unit that entities other than systemd can manage programmatically. They represent sets of processes, and enable a subset of the control features that systemd provides for service units. Since processes created by Zeek's supervisor aren't managed by systemd, scopes provide a good compromise. Under the hood, systemd creates a cgroup for the scope and puts the requested processes into it.
The main features enabled by scopes include resource control for CPU, memory, and I/O, and the introduction of a process inspection granularity that naturally fits Zeek (e.g, show resource usage of all worker nodes).
Here's an example of what this looks like in a cluster with two worker processes. Given this cluster ...
zeek(1751562)---zeek.stem(1751563)-+-zeek.controller(1751574)
|-zeek.instance-1(1751573)
|-zeek.logger-01(1752216)
|-zeek.manager(1752219)
|-zeek.worker-01(1752217)
`-zeek.worker-02(1752218)
... a scope for the workers looks like this:
$ systemctl status zeek-cluster-worker.scope
● zeek-cluster-worker.scope
Loaded: loaded (/run/systemd/transient/zeek-cluster-worker.scope; transient)
Transient: yes
Active: active (running) since Mon 2021-11-08 13:14:18 PST; 1min 15s ago
Tasks: 20 (limit: 38341)
Memory: 16.0K
CGroup: /system.slice/zeek-cluster-worker.scope
├─1752217 zeek -C -j test-a2c.zeek
└─1752218 zeek -C -j test-a2c.zeek
Nov 08 13:14:18 swinetrek systemd[1]: Started zeek-cluster-worker.scope.
Once established, unit files with [Scope] sections can now apply constraints
to the process set. Systemd's drop-in mechanism applies to these files.
For tinkering, you can currently do this manually via busctl. If not already defined, you may need to set environment variables as follows to use busctl.
$ export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$UID/bus
$ export XDG_RUNTIME_DIR=/run/user/$UID
To establish or update scopes, calls into libsystemd (see below) need to provide the set of processes to have in the scope. If we add plugin hooks for the Supervisor's main process management operations, we'd define natural points for doing so and can keep systemd functionality out of Zeek itself. A plugin could then hook in and do the libsystemd calls.
Scopes also naturally group the stdout/stderr logs of the processes in the scope. We could make the logs more granular for journalctl use by logging explicitly via libsystemd. For details, see here. This could apply to a subset of Zeek logs (reporter logs, say), again via a plugin.
Systemd provides a library that third parties can use to communicate with systemd. Under the hood, this uses d-bus, which brings up questions regarding synchronous vs asynchronous I/O. For simple use cases in the supervisor synchronous communication seems fine, for more complex settings we could use an I/O source in Zeek.