Skip to content

emfor 0.6.0

Latest

Choose a tag to compare

@vanillahsu vanillahsu released this 16 Nov 23:30
· 8 commits to main since this release

The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged: clients and plugins of 0.5.0 keep working with this daemon. Upgrade
libemfor and the clients before you use drain = true, so that an older client
does not choose a drained backend (see below). A config that 0.5.0 accepted is
still accepted.

Manual chapter 7 is split in two (both languages). 07-dns.md keeps the DNS
server, its answers, SRV records, forwarding and caching; the recipes for
nginx, HAProxy, Caddy, Prometheus and pf and the clients that ask for SRV
records themselves moved to 07-dns-recipes.md (chapter 7b), so links to
07-dns.md#using-it-with-nginx and the like now point there. The chapter
numbers do not change.

examples/deploy/emfor-drain.sh is safer. wait-ready no longer returns
at the first healthy: a backend that has just stopped keeps that state for
fall x interval (6 s by default), so a drained backend that was stopped
could be put back dead. It now needs a check newer than the command and
healthy for SETTLE seconds in a row (third argument, default 10). on and
off refuse a backend line they cannot change (it must read
{ addr = "..." }, with a space before the }) without touching the file, and
put the old file back when check-config refuses the new one. New:
EMFOR_CONFIG (passed as --config) and EMFOR_SERVICE (the service, when
two have the same address). tests/deploy has a case with the default
timings, and one for each refusal. Manual chapter 8 (both languages) lists
what to set up first, and the corrections of the review of chapters 7, 8 and
10: emfor status needs write access to the lock file, the drained (checks: …) marker comes last, the upgrade order (clients first, check-config before
the restart), the metrics tables, and in chapters 7 and 10 the SERVFAIL cases,
the 1232-byte UDP cap, include_dir before the first table, rise and fall
of 1 instead of Consul's 0, and weights from 1 to 100.

Using emfor with HAProxy. Manual chapter 7 (both languages) has a section
for HAProxy: a server-template on emfor's SRV answers (resolvers with
nameserver 127.0.0.1:5300) fills its servers, and health and drain reach
HAProxy within a few seconds without a reload. It also says what HAProxy does
not take from the answer: SRV priorities and weights are ignored (so
failover, weighted, locality and fastest do not behave as for nginx),
and that after SERVFAIL HAProxy drops its servers after its hold other
time (30 s) and answers 503, where nginx keeps its last list. The slots of the
template limit the number of backends. tests/haproxy/run.sh [image] checks
all of it, the two ignored features included, and the Integrations workflow
runs it on haproxy:lts and haproxy:2.8. Checked with HAProxy 3.4.7 and 2.8.31.
The section also covers mode tcp (no difference), TLS to the backends (set
sni: without it 3.4.7 refuses the certificate and 2.8.31 accepts any
certificate the CA signed; verifyhost is not reliable) and several name
servers (HAProxy sends each query to all of them and uses the first reply;
with different answers the result varied from run to run, and requests got 503
after a slot came up with port 0, which on HAProxy 3.4.4 (FreeBSD) happened
with one name server as well; use one local nameserver);
tests/haproxy/more.sh checks the agreeing case, run by the same run.sh.

Using emfor with nginx. Manual chapter 7 (both languages) has a section
for nginx: from 1.27.3 the open-source nginx can take its upstream from emfor's
SRV answers (server api.svc.emfor service=http resolve; with resolver 127.0.0.1:5300), so health, drain and weights reach it within a few
seconds without a reload; the section also says what nginx does when emfor
answers SERVFAIL (it keeps its last list) and gives the variable
proxy_pass recipe for older nginx. tests/nginx/run.sh [image] checks it
on a given image, and the Integrations workflow runs it on nginx:stable and
nginx:mainline. That workflow runs every week and when emfor's DNS or
candidate code or these tests change, not on every push.
Checked with nginx 1.30.6, 1.31.8 and (older recipe only) 1.26.3. Not tried:
HTTPS upstreams, nginx Plus, several workers.
The section now also covers the stream module (TCP): the same upstream
works in a stream block, a connection already open to a drained backend is
not cut (it is cut when the backend is stopped), and a new connection to a
stopped backend goes to the next server. And it says that after a lookup has
failed (emfor answered SERVFAIL), nginx asks again only after
resolver_timeout, 30 seconds by default (about 26 seconds to follow emfor's new
answers), so the recipe now sets resolver_timeout 1s (about a second), for
both http and stream, with nginx 1.30.6 and 1.31.8. tests/nginx/run.sh
runs tests/nginx/stream.sh after inside.sh. Not tried for stream: UDP,
ssl_preread, proxy_protocol, nginx Plus. On FreeBSD 15.1 (nginx 1.30.6
package) both scripts passed by hand (about 25 seconds with the default
resolver_timeout, about 1 with resolver_timeout 1s); the package builds stream as a dynamic
module, so nginx.conf needs load_module /usr/local/libexec/nginx/ngx_stream_module.so;.

Rolling deploy with drain (manual chapter 8, both languages).
examples/deploy/emfor-drain.sh drains a backend in a conf.d file (one
backend per line), checks the config, reloads, and waits until emfor shows it
drained (on), waits until the connections to a port are quiet (wait-idle,
Linux, needs ss), waits until the checks say healthy (wait-ready), and
undrains (off). The chapter says what the waiting is for and what was seen:
a request in flight finishes after a drain and is cut after a plain stop;
nginx 1.30 keeps idle connections to its upstream, so "no connection" never
comes. tests/deploy/run.sh [image] checks it on nginx 1.30.6 and 1.31.8
(also in the Integrations workflow). On FreeBSD 15.1 it passed by hand with
wait-idle replaced by a fixed wait (ss is Linux only; the script says so
there). Not tried: HAProxy, HTTPS, clients with their own connection pool,
several emfor hosts.

Clients that ask for SRV records themselves (manual chapter 7, both
languages).
MIT Kerberos (dns_lookup_kdc, the realm is the service name in
capitals), OpenLDAP (ldapsearch -H 'ldap:///dc=ldap%2Cdc=svc%2Cdc=emfor', the
commas must be %2C) and PyMongo (mongodb+srv://, TLS is on by default) find
their servers through emfor's SRV answers and leave a server that fails its checks
or is drained; a _udp name gets NXDOMAIN; with balance = "failover" Kerberos
tries the primary first. tests/srv/run.sh checks them with the Debian
bookworm packages, against plain listeners. Not tried: a real KDC, LDAP or
MongoDB server, LDAPS, sssd, Samba, Active Directory names, other drivers.

Nagios, Icinga and Zabbix (manual chapter 8, both languages).
examples/nagios/check_emfor.sh gives the Nagios status of what emfor status --json reports (CRITICAL for a stale or stopped daemon or a service with fewer
than -c healthy backends, WARNING below -w, UNKNOWN when emfor or the store
cannot be read; drained backends are left out of the counts; perfdata per
service). examples/nagios/emfor-exec.sh runs an existing Nagios plugin as an
exec check, adding -H and -p from EMFOR_IP and EMFOR_PORT; for emfor a
plugin's WARNING is down (--warning up changes that) and UNKNOWN is an error.
tests/nagios/run.sh checks both with a stand-in plugin. Not tried: the real
Monitoring Plugins, a Nagios or Icinga core, NRPE, Zabbix, FreeBSD.

Using it with pf (FreeBSD; manual chapter 7, both languages).
examples/pf/emfor-pf-table.sh asks emfor's DNS server for the SRV records of a
service every second and makes a pf table equal to the targets with the lowest
priority, so that an rdr ... -> <api> round-robin rule spreads connections
over the backends emfor offers, with no proxy. Health and drain reach the
table, a connection that is already open is not cut (pf keeps its state),
priorities give the table its failover, SRV weights are not used, an empty
answer empties the table (pf then drops new connections, so the client waits for
its own timeout; EMPTY puts an address where nothing listens there instead),
and when emfor does not answer the table is left alone. tests/pf/run.sh
checks it on FreeBSD 15.1 (by hand, as root; not in the CI). Not tried: IPv6,
source-hash, daemon(8), many clients.

Using it with Prometheus (manual chapter 7, both languages).
dns_sd_configs of type SRV on emfor's answers gives Prometheus the healthy
backends of a service as scrape targets. The section says what that means: a
backend that fails its checks or is drained is no longer a target, so its up
series ends and up == 0 never fires (alert on emfor's metrics, or keep a
static list for the scrape); a failover backup is scraped too; a
backup = true backend only when no primary is healthy; and the host
resolver must forward the whole zone, since dns_sd_configs cannot be given a
name server or port. tests/prometheus/run.sh checks it on Prometheus 3.16.0
(also in the Integrations workflow); on FreeBSD 15.1 the same script passed by
hand with Prometheus 2.55.1. Not tried: dnsmasq or unbound in front of emfor,
other versions, HTTPS targets, many targets.

Using it with Caddy (manual chapter 7, both languages). reverse_proxy
with dynamic srv takes its upstreams from emfor's SRV answers, so health and
drain reach Caddy without a reload. The section gives the Caddyfile and says
what each part is for: resolvers in both dynamic srv and transport http
(the second dials the names of the targets), grace_period (without it every
request fails when emfor answers SERVFAIL), lb_try_duration (without it a
backend that has just stopped loses requests). Caddy does not follow the SRV
weights, and follows priorities only with lb_policy first.
tests/caddy/run.sh [image] checks it on Caddy 2.11.9 (also in the
Integrations workflow). Not tried: HTTPS upstreams, Caddy's own health checks,
other policies, other versions, FreeBSD.

deb and rpm packages, and a packages/ directory. The example FreeBSD
port moved from examples/freebsd/ to packages/freebsd/ (the same files;
update a link or a script that names the old path). packages/build.sh
builds a deb and an rpm in a container with an old glibc (Debian 11 and
later, Ubuntu 20.04 and later, RHEL 9 and relatives) into packages/dist/:
emfor and emfor-check under /usr/bin, the systemd unit, a config that
only sets include_dir = "conf.d" plus an empty /etc/emfor/conf.d, the
example config, and the emfor user (the deb enables and starts the
service, the rpm does not). They are built from this repository and not
published; they are not signed. They depend on ca-certificates: the http
check builds its client with the system's root certificates even for
scheme = "http", and fails with "cannot build http client" without them.
Tested on arm64 in containers (Debian 12 and Rocky Linux 9): install, user
and modes, the daemon as the emfor user, remove, purge, upgrade over a
modified config; the deb also under systemd 252 as PID 1 in a Debian 12
container (enable and start by the postinst, the hardened unit with tcp, http,
exec and dns-plugin checks, reload, a refused conf.d file, restart, remove,
purge). Not tested: the rpm under systemd, x86-64 packages, a real host.

drain = true: take a backend out of every choice. A backend table can
set drain = true; after a reload the backend is never chosen: by no
balance method (sticky and failover included), not in the last tier of
an open service, not in DNS A, AAAA and SRV answers, not in a fastest
pool. Its checks keep running, so emfor status shows drained (checks: healthy) and removing drain brings it back at once, in the state the
checks gave, without counting rise again. Draining and undraining start no slow
start themselves; a backend that recovered while drained ramps from that
recovery as usual. emfor lookup --explain lists it
as drained, and the metric emfor_backend_drain is 1 for it. When every
backend of a service is drained, lookups fail with ALL_DOWN and DNS
answers SERVFAIL, open or not. drain belongs to the host: it is not
sent to peers and not part of the service definition they compare. The
store format stays version 2: a drained backend is written as unhealthy
with the reason drained, plus the optional fields drain and
checked_state; an older client therefore skips it too, except in the last
tier of an open service when every backend is down, where it can still
choose it (upgrade libemfor first). There is no run-time command: edit the
config and reload.

Fixes from a review of the whole system. (1) include_dir: a
directory or fragment owned by anyone but root or the user running emfor is
refused (a link is judged by the file it points to), as one writable by
group or others already was; the file is checked after it is opened, not by
name. Run emfor check-config as the user the daemon runs as, or keep the
files owned by root. (2) A reload or shutdown no longer counts the plugin
requests it cancels as "no answer", so a healthy plugin shared by several
services is not killed and restarted when one of them is reloaded away. (3)
emfor check-config (exit 1) and emfor lookup (exit 3) report a write
error on standard output with a status instead of panicking (exit 101); a
closed pipe is not an error. (4) A peers secret_file that group or others can write is refused.
(5) A plugin's validate answer is bounded in bytes of its JSON line, so that a rejected reload cannot
make the daemon kill the running plugin. (6) The check-config --show
comments escape control characters in file names. (7) boot in a peer
message must be lowercase hex, as specified; a peer's down reason is cut
and escaped before it is logged; meta/peers writes no longer pile up while
commits fail; the DNS rate limiter sweeps a full table once a second, not
once per packet. (8) The metrics documentation lists the seventh peer reject
reason, plaintext. (9) Client::for_store (new in this release, for emfor lookup) exists only
with the daemon-internal feature, and Client::new is not public (the Rust
Client type itself was not exported before this release). Not changed, on purpose: the RADIUS secret_file,
password_file and tls_key_file still have no permission check (adding one
would stop installs that work today from starting); the Perl client hashes a
byte-string affinity key as Latin-1 text (documented: decode it first).

emfor check-config --show: the merged config and where each value
comes from.
With include_dir the config is in several files; --show
checks it as check-config does and prints the merged config as TOML on
standard output, each value followed by # <file>, the file that last set
it (a list is set as a whole). The config ok and read lines go to
standard error, and include_dir is written as a comment (its files are
merged already), so the output can be saved anywhere and checked again; an invalid
config prints nothing on standard output and exits 2. Without include_dir
the output starts with one comment naming the main file. Built-in defaults
are not added and nothing is redacted (a plugin command is printed as
written). Without --show nothing changes. The comment format is for
people, not for scripts.

include_dir: the config in several files. include_dir = "conf.d" in
emfor.toml makes the daemon read the main file and then every *.toml file
of that directory in filename order (byte order: write 01-, 10-, 99-),
a later file overriding an earlier one. Tables merge key by key; anything
else, lists such as backends included, is replaced by the later value. A
later file cannot remove a service or backend. [store] and include_dir
only work in the main file, because emfor status and emfor lookup read
only that file. The directory and its files must not be writable by group or
others (a fragment can declare a plugin command), and the directory must
exist, or the config is refused: at start the daemon exits 2, on a reload it
keeps the old config. Errors name their file (file:line:column for TOML
syntax errors, the file that set the key for the others, best effort when two
files contribute), emfor check-config lists the files it read when
include_dir is set, and the daemon logs the list at start and reload. The
main config file is not permission-checked, as before. Without include_dir
nothing changes. The FreeBSD port creates an empty etc/emfor/conf.d. Not
tested yet: the port build (make stage check-plist), macOS and FreeBSD
test runs.

A chapter for people who come from HAProxy or Consul. Chapter 10 of the
manual (both languages) translates an HAProxy backend and a Consul service
with an HTTP check into emfor.toml, option by option, says what has no
equivalent (least connections, retries, passive checks, taking a backend out
for maintenance, registration), and describes how to try emfor next to what
is running. The configuration examples were checked with
emfor check-config; the HAProxy and Consul options were checked against
their documentation as far as it could be read (the HAProxy weight range and
the server option defaults beyond inter, rise and fall were not).

emfor lookup: what an application would get, and why. A new command,
emfor lookup <service> [--key <k>] [--explain] [--json] [--dns A|AAAA],
prints the backend a client library would give, for shell scripts and cron
jobs without a client. --explain shows whether the store is stale, the tier
the candidates come from, the reason of every backend that is out
(unhealthy, unknown, last check failed, backup held back, not in the pool, not at this site, stale), slow start steps, and how balance
chooses; it does not advance any rotation. failover and sticky with a key
name their backend; the other methods list the candidates, since each run is
a new process with a random rotation start. --dns A|AAAA explains the
answer of the daemon's DNS server from the store (up to about a second
behind the daemon). --json is stable and only gains fields; exit status 0
answer, 1 no backend, 2 no such service or bad usage, 3 store unusable.
SRV answers are not explained.

Inside: the tiers of healthy_or_all are now computed by
emfor_format::rules::explain_tiers, which also gives the reasons (the
candidates are unchanged); emfor-client makes Client public with
Client::for_store, explain and lookup_explained; emfor-daemon depends
on emfor-client. No file is installed that was not before. The lookup
call, its error codes, libemfor and the other language clients are
unchanged.

emfor status shows more of what decides a lookup. The text output
(which is not a stable interface; --json and the exit status are
unchanged) has three additions:

  • A backup backend is marked backup.
  • A backend that is still healthy but whose last check failed is marked
    last check failed, before its reason. Lookups usually skip such a
    backend (not when every healthy backend is failing), so a healthy row
    with a reason no longer reads as "healthy and in use".
  • When this host and the live members that have not left a service out
    number fewer than three, the group cannot vote, and every reason is this
    host's own. The peers: line then ends with too few live members to vote (only when this holds for every shown service that takes part in
    the group; assigned_checkers in --json has the per-backend answer).

The manual and docs/deploy/daemon.md now list every marker of a backend
row (local-up and voted were missing from the lists) and the peers
and assigned_checkers fields of emfor status --json, and say that the
age of a voted backend is the time its result was applied, not the time
this host last connected to it.

Known limits

  • FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests, the
    rc.d script, the port with make stage check-plist install, the fork
    tests, and the nginx, HAProxy and pf recipes by hand. Not tested:
    poudriere, 13.x, 14.x and amd64 by hand (the CI job runs the tests on FreeBSD 14.5 and 15.1 on GitHub).
  • HAProxy 3.4.4 on FreeBSD answered 503 for a while after a server-template
    slot came up with port 0, also with one name server; the cause is not
    known (the same test passes on Linux).
  • The deb and rpm have been installed, upgraded and removed in containers
    (arm64) without systemd; not tested on x86-64 or under systemd, and they
    are not signed.
  • The packages and the port are not in any repository: they are built from
    the source tree.
  • Everything in 0.5.0's known limits that is not listed here has not
    changed: no forward secrecy for peer messages; the ldap plugin checked
    against Active Directory only without a certificate; lookup throughput
    across threads levels off at about two million a second on Linux
    (docs/performance.md); with [store] sync = false a
    host crash can damage the store; a host announces at most 80 services to
    the peer group. FreeBSD's libthr fork problem is worked around by
    libemfor as in 0.5.0
    (docs/freebsd/).