Repository navigation
Releases
emfor 0.6.0
The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged: clients and plugins of 0.5.0 keep working with this daemon. Upgrade
libemfor and the clients before you use drain = true, so that an older client
does not choose a drained backend (see below). A config that 0.5.0 accepted is
still accepted.
Manual chapter 7 is split in two (both languages). 07-dns.md keeps the DNS
server, its answers, SRV records, forwarding and caching; the recipes for
nginx, HAProxy, Caddy, Prometheus and pf and the clients that ask for SRV
records themselves moved to 07-dns-recipes.md (chapter 7b), so links to
07-dns.md#using-it-with-nginx and the like now point there. The chapter
numbers do not change.
examples/deploy/emfor-drain.sh is safer. wait-ready no longer returns
at the first healthy: a backend that has just stopped keeps that state for
fall x interval (6 s by default), so a drained backend that was stopped
could be put back dead. It now needs a check newer than the command and
healthy for SETTLE seconds in a row (third argument, default 10). on and
off refuse a backend line they cannot change (it must read
{ addr = "..." }, with a space before the }) without touching the file, and
put the old file back when check-config refuses the new one. New:
EMFOR_CONFIG (passed as --config) and EMFOR_SERVICE (the service, when
two have the same address). tests/deploy has a case with the default
timings, and one for each refusal. Manual chapter 8 (both languages) lists
what to set up first, and the corrections of the review of chapters 7, 8 and
10: emfor status needs write access to the lock file, the drained (checks: …) marker comes last, the upgrade order (clients first, check-config before
the restart), the metrics tables, and in chapters 7 and 10 the SERVFAIL cases,
the 1232-byte UDP cap, include_dir before the first table, rise and fall
of 1 instead of Consul's 0, and weights from 1 to 100.
Using emfor with HAProxy. Manual chapter 7 (both languages) has a section
for HAProxy: a server-template on emfor's SRV answers (resolvers with
nameserver 127.0.0.1:5300) fills its servers, and health and drain reach
HAProxy within a few seconds without a reload. It also says what HAProxy does
not take from the answer: SRV priorities and weights are ignored (so
failover, weighted, locality and fastest do not behave as for nginx),
and that after SERVFAIL HAProxy drops its servers after its hold other
time (30 s) and answers 503, where nginx keeps its last list. The slots of the
template limit the number of backends. tests/haproxy/run.sh [image] checks
all of it, the two ignored features included, and the Integrations workflow
runs it on haproxy:lts and haproxy:2.8. Checked with HAProxy 3.4.7 and 2.8.31.
The section also covers mode tcp (no difference), TLS to the backends (set
sni: without it 3.4.7 refuses the certificate and 2.8.31 accepts any
certificate the CA signed; verifyhost is not reliable) and several name
servers (HAProxy sends each query to all of them and uses the first reply;
with different answers the result varied from run to run, and requests got 503
after a slot came up with port 0, which on HAProxy 3.4.4 (FreeBSD) happened
with one name server as well; use one local nameserver);
tests/haproxy/more.sh checks the agreeing case, run by the same run.sh.
Using emfor with nginx. Manual chapter 7 (both languages) has a section
for nginx: from 1.27.3 the open-source nginx can take its upstream from emfor's
SRV answers (server api.svc.emfor service=http resolve; with resolver 127.0.0.1:5300), so health, drain and weights reach it within a few
seconds without a reload; the section also says what nginx does when emfor
answers SERVFAIL (it keeps its last list) and gives the variable
proxy_pass recipe for older nginx. tests/nginx/run.sh [image] checks it
on a given image, and the Integrations workflow runs it on nginx:stable and
nginx:mainline. That workflow runs every week and when emfor's DNS or
candidate code or these tests change, not on every push.
Checked with nginx 1.30.6, 1.31.8 and (older recipe only) 1.26.3. Not tried:
HTTPS upstreams, nginx Plus, several workers.
The section now also covers the stream module (TCP): the same upstream
works in a stream block, a connection already open to a drained backend is
not cut (it is cut when the backend is stopped), and a new connection to a
stopped backend goes to the next server. And it says that after a lookup has
failed (emfor answered SERVFAIL), nginx asks again only after
resolver_timeout, 30 seconds by default (about 26 seconds to follow emfor's new
answers), so the recipe now sets resolver_timeout 1s (about a second), for
both http and stream, with nginx 1.30.6 and 1.31.8. tests/nginx/run.sh
runs tests/nginx/stream.sh after inside.sh. Not tried for stream: UDP,
ssl_preread, proxy_protocol, nginx Plus. On FreeBSD 15.1 (nginx 1.30.6
package) both scripts passed by hand (about 25 seconds with the default
resolver_timeout, about 1 with resolver_timeout 1s); the package builds stream as a dynamic
module, so nginx.conf needs load_module /usr/local/libexec/nginx/ngx_stream_module.so;.
Rolling deploy with drain (manual chapter 8, both languages).
examples/deploy/emfor-drain.sh drains a backend in a conf.d file (one
backend per line), checks the config, reloads, and waits until emfor shows it
drained (on), waits until the connections to a port are quiet (wait-idle,
Linux, needs ss), waits until the checks say healthy (wait-ready), and
undrains (off). The chapter says what the waiting is for and what was seen:
a request in flight finishes after a drain and is cut after a plain stop;
nginx 1.30 keeps idle connections to its upstream, so "no connection" never
comes. tests/deploy/run.sh [image] checks it on nginx 1.30.6 and 1.31.8
(also in the Integrations workflow). On FreeBSD 15.1 it passed by hand with
wait-idle replaced by a fixed wait (ss is Linux only; the script says so
there). Not tried: HAProxy, HTTPS, clients with their own connection pool,
several emfor hosts.
Clients that ask for SRV records themselves (manual chapter 7, both
languages). MIT Kerberos (dns_lookup_kdc, the realm is the service name in
capitals), OpenLDAP (ldapsearch -H 'ldap:///dc=ldap%2Cdc=svc%2Cdc=emfor', the
commas must be %2C) and PyMongo (mongodb+srv://, TLS is on by default) find
their servers through emfor's SRV answers and leave a server that fails its checks
or is drained; a _udp name gets NXDOMAIN; with balance = "failover" Kerberos
tries the primary first. tests/srv/run.sh checks them with the Debian
bookworm packages, against plain listeners. Not tried: a real KDC, LDAP or
MongoDB server, LDAPS, sssd, Samba, Active Directory names, other drivers.
Nagios, Icinga and Zabbix (manual chapter 8, both languages).
examples/nagios/check_emfor.sh gives the Nagios status of what emfor status --json reports (CRITICAL for a stale or stopped daemon or a service with fewer
than -c healthy backends, WARNING below -w, UNKNOWN when emfor or the store
cannot be read; drained backends are left out of the counts; perfdata per
service). examples/nagios/emfor-exec.sh runs an existing Nagios plugin as an
exec check, adding -H and -p from EMFOR_IP and EMFOR_PORT; for emfor a
plugin's WARNING is down (--warning up changes that) and UNKNOWN is an error.
tests/nagios/run.sh checks both with a stand-in plugin. Not tried: the real
Monitoring Plugins, a Nagios or Icinga core, NRPE, Zabbix, FreeBSD.
Using it with pf (FreeBSD; manual chapter 7, both languages).
examples/pf/emfor-pf-table.sh asks emfor's DNS server for the SRV records of a
service every second and makes a pf table equal to the targets with the lowest
priority, so that an rdr ... -> <api> round-robin rule spreads connections
over the backends emfor offers, with no proxy. Health and drain reach the
table, a connection that is already open is not cut (pf keeps its state),
priorities give the table its failover, SRV weights are not used, an empty
answer empties the table (pf then drops new connections, so the client waits for
its own timeout; EMPTY puts an address where nothing listens there instead),
and when emfor does not answer the table is left alone. tests/pf/run.sh
checks it on FreeBSD 15.1 (by hand, as root; not in the CI). Not tried: IPv6,
source-hash, daemon(8), many clients.
Using it with Prometheus (manual chapter 7, both languages).
dns_sd_configs of type SRV on emfor's answers gives Prometheus the healthy
backends of a service as scrape targets. The section says what that means: a
backend that fails its checks or is drained is no longer a target, so its up
series ends and up == 0 never fires (alert on emfor's metrics, or keep a
static list for the scrape); a failover backup is scraped too; a
backup = true backend only when no primary is healthy; and the host
resolver must forward the whole zone, since dns_sd_configs cannot be given a
name server or port. tests/prometheus/run.sh checks it on Prometheus 3.16.0
(also in the Integrations workflow); on FreeBSD 15.1 the same script passed by
hand with Prometheus 2.55.1. Not tried: dnsmasq or unbound in front of emfor,
other versions, HTTPS targets, many targets.
Using it with Caddy (manual chapter 7, both languages). reverse_proxy
with dynamic srv takes its upstreams from emfor's SRV answers, so health and
drain reach Caddy without a reload. The section gives the Caddyfile and says
what each part is for: resolvers in both dynamic srv and transport http
(the second dials the names of the targets), grace_period (without it every
request fails when emfor answers SERVFAIL), lb_try_duration (without it a
backend that has just stopped loses requests). Caddy does not follow the SRV
weights, ...
emfor 0.5.0
Stricter validation: a config that 0.4.0 accepted can now be refused.
emfor check-config, and so a start or a reload, now refuses options that
could never work and that 0.4.0 let through, to fail later: an interval
above 24h, a timeout below 10ms, an http path with # or \, an
exec or plugin command with a NUL, a user or password with a NUL where
the protocol cannot carry one (mysql, pgsql, amqp), an amqp password too
long for AMQP's first frame, a diameter list too long for one message, a
ca_file with no usable CA, and more (see Daemon and emfor-check). Run
emfor check-config with the new binary before you upgrade a host.
Peer messages have a second encrypted format, EMP3
(XChaCha20-Poly1305, which has no practical limit on the number of
messages per secret). Each member is sent the format it understands, so a group of
0.4.0 and 0.5.0 members works in any upgrade order without a
configuration change (see Daemon).
The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon; its files are now libemfor.so.0.5 and
libemfor.0.5.dylib.
The main additions are EMP3, [peers] accept_plaintext and
previous_secret_file, a DNS server that limits UDP responses per client
network and keeps answering while a commit is slow, plugins that are
restarted when they stop answering, and clients that no longer end the
application with SIGBUS on a truncated store or hang a forked child on
FreeBSD. The rest are fixes, most of them in emfor-check and the DNS
server.
Daemon
- Peer messages have a new encrypted format,
EMP3: XChaCha20-Poly1305
with a 24-byte random nonce and a key of its own derived from the
secret.EMP2's 12-byte random nonce is safe for about 2^32 messages
per secret, which a large group can send within days; a repeated nonce
would let an observer read and forge messages. Withencrypt = truethe
format is picked per member: everyEMP2message now carries
"emp3": truein its encrypted payload, and a member is sentEMP3once
the latest message accepted from it isEMP3or such anEMP2, and
EMP2otherwise (also after it sendsEMP1), so members of this
release useEMP3between each other andEMP2with 0.4.0 members. A
group needs no configuration change to upgrade or to go back; a host
going back to 0.4.0 dropsEMP3messages for about its first second (see
Peer group). - A store file shorter than LMDB records it (changed in place, or its
tail lost in a power loss) is moved to<path>.corruptat start like
other damage, with a warning sayingtruncated. Before, reading it to
check it ended the daemon withSIGBUS, again at every restart.
After every commit the daemon extends the file with zero bytes to the
length LMDB records, which LMDB itself can leave a few pages short, so
clients never take a healthy store for a truncated one. A store an
older emfor left short that way is moved aside on the first start
after the upgrade (only the state history is lost). - A plugin that stops answering is restarted. Until now only a plugin
that exited or closed its stdout was restarted: one that kept running
but answered nothing (stuck in a loop, deadlocked, stopped) left every
check of its backendserrorfor good, and the requests for it queued
up without bound. Now, when the callers of 3 requests in a row give up
waiting and no answer to any request arrives in between, the daemon
logs that the plugin stopped answering, kills it and restarts it with
the usual backoff. A plugin that answers withintimeout_msis never
affected (see Plugin protocol). - A single key or value in the store that is not UTF-8 no longer blocks
the daemon. Undersvc/it made every commit fail from start-up on,
so clients saw a stale store for good; underhist/it kept every
service's history from loading. Such entries are now deleted, or
discarded with a warning, on their own. httphost_headermust be a host with an optional port
(api.internal,api.internal:8443,[2001:db8::1]). Values with
spaces, a path or a scheme passedcheck-configbut made every check
down, or silently checked whatever host the name resolved to instead
of the backend.- State changes already queued when one is written are committed in the
same transaction, so a burst of changes (a network split) costs one
commit, not one per backend. - The queues from checks to the peer task and to a plugin's stdin are
bounded. A peer task that falls more than 4096 messages behind drops new
downandresultmessages instead of holding them (counted in the new
emfor_peer_queue_dropped_total, logged once a minute); checks never
wait for it. A plugin that stops reading its stdin gets at most 1024
waiting requests; further checks areerrorat once with
plugin <name> busy: it is not reading its requestsinstead of each
waiting for its timeout while memory grows. tls_cert_fileandtls_key_file(thediameterplugin) are parsed
again only when they change, likeca_file: an unchanged file costs one
statper check instead of a read and a parse, and a renewed one still
takes effect at the next check. Parsedca_file, certificate and key
files that no check used for 10 minutes are released, so files removed
from the config by reloads no longer stay in memory.- The DNS server limits UDP responses to each client network (IPv4 /24,
IPv6 /56) to[dns] rate_limita second, 100 by default (0turns it
off). An SRV answer can be many times the size of its query, so a server
reachable from other hosts could be used to flood a third party with
forged-source queries. Past the limit every other response is an empty
truncated one, which makes a real client retry over TCP, and the rest
are dropped;emfor_dns_rate_limited_totalcounts them. Loopback
clients and TCP are never limited. At start the daemon now warns about
each[dns] listenaddress that is not a loopback address. - A slow commit no longer freezes DNS answers and
/metrics. While the
state store waited for the disk ([store] sync = true, or the kernel
writing back), the daemon stopped applying check results, so DNS kept
answering with the states from before the commit and the results
queued up in memory. Results are now applied as they arrive; the
changes made meanwhile are written together by the next commit, which
starts as soon as the slow one ends. - A reload that changes
intervalorjitterapplies to the wait
already under way. Before, going from10mto2stook up to 11
minutes to apply. - The 200ms checks while a backend is
unknownstop afterrise + fall
in a row; results that never reach either count no longer check about
5 times a second for good. intervalis at most24handtimeoutat least10ms. Larger
intervals could panic the check task (with a hugefall), and a 1ms
timeout gave plugins 0ms.[store] map_sizeis rounded up to a multiple of 64KiB. A size LMDB
refuses (such as"1048577") passedcheck-configand then stopped
the daemon from starting.- A check task that is restarted after a bug now records the backend's
move tounknownas a transition (reasoncheck task restarted);
before, the stored state changed silently. - The DNS server's view of a service is updated per backend instead of
copying the whole service on every check. - DNS over TCP serves at most 128 connections at once and closes a
connection with no query for 5 seconds, so clients can no longer use
up the daemon's file descriptors. - If a DNS listener stops while the daemon runs, the daemon logs it and
exits with status 70, so the service manager restarts it; before, DNS
went silent. - DNS answers BADVERS to EDNS versions above 0 and REFUSED to classes
other than IN; before, a CH query got IN records. - DNS compares names label by label: a label containing a
.byte no
longer puts a name outside the zone inside it. - Over UDP, a reply whose single record or SOA does not fit (names near
255 bytes) is truncated by emfor itself and keeps its EDNS record. - Plugins run in a process group of their own. When a plugin exits, is
stopped, or is killed because it stopped answering, the processes it
started are killed too; before, they could outlive it. - A plugin's stderr lines and invalid stdout lines are rate limited: up
to 100 at once, then 10 a second each. The log notes how many lines
were dropped, so a plugin writing in a loop can no longer fill the
log. - A
ca_fileis parsed again only when it changes (inode, size or
times), not on every check. A replaced file still takes effect at the
next check. - http clients for checks (one per backend with a domain
host_header)
are built off the async worker threads, because building one reads the
system CA store. The peersecret_fileis read off them too. - Documented: a connection that fails because this host has no route to
the backend, or its own firewall blocks it, isdown, noterror.
That has always been the behaviour. - When a check task is restarted after a bug while its backend is
unknown, the backend's latency is now cleared and its reason set to
check task restarted, in the store and in DNS. - A late result from the check task of a backend removed by a reload can
no longer mark that backend healthy after a later reload adds it back.
Each check task now has its own generation, and a backend keeps its
state exactly while its task keeps running. check-confignow refuses an httppathwith#,\or a control
character. The request URL cut or rewrote these, so another path was
c...
emfor 0.4.0
Breaking: the store format is version 2. The daemon writes
meta/schema_version 2, which adds the backend field backup. Clients
and emfor status from this release read versions 1 and 2; older ones
fail every lookup on a version 2 store with INCOMPATIBLE. Upgrade the
clients on a host before the daemon (see
Store format version).
Peer messages are encrypted by default. Members running 0.3.0 cannot
read them and drop them as malformed. To upgrade a peer group, set
encrypt = false in [peers] on each host as it is upgraded; once every
member runs this release, remove the line and reload each daemon.
Breaking, on FreeBSD only: examples/rc.d/emfor is gone, and the
example port keeps the store in /var/db/emfor (see Deployment).
The main additions are backup backends, slow start, a preference for
backends whose last check passed, sticky lookups that honour weight,
encrypted peer messages, and the smtp, amqp, mongodb, grpc and
quic plugins of emfor-check. The plugin protocol (1) is unchanged.
As with every release, install libemfor from the same release as the
daemon.
Daemon
-
Peer group messages are encrypted (
EMP2: ChaCha20-Poly1305 with a key
derived fromsecret_fileby HKDF-SHA256 and a random nonce per
message), so backend names, addresses and reasons no longer travel in
the clear. New[peers] encrypt(defaulttrue, applied on reload):
falsesends the signed plaintextEMP1of 0.3.0. Both formats are
accepted either way; withencrypt = truethe daemon warns once a
minute per member that still sendsEMP1. A message that does not
decrypt counts asauthinemfor_peer_messages_rejected_total.
Secret rotation works as before. There is no forward secrecy: whoever
later gets the secret can read recorded messages, so rotate it
regularly. See[peers]. -
Backup backends: a backend table takes
backup = true. Backups are
used only while no other backend passes its checks: clients and the DNS
server take the primaries whose last check passed, else the backups
whose last check passed, else the healthy primaries, else the healthy
backups, andbalanceworks within that set;fastestdecides which
backends are fast among the primaries and among the backups
separately. When no backend is
healthy,on_all_downdecides as before and backups count like the
others. SRV answers list only the chosen set. At least one backend of a
service must not be a backup, andbackupis applied on reload. See
Backup backends. -
Clients and the DNS server leave out a healthy backend whose last
check failed (itsreasonis set) while another healthy backend passed
its last check, so a backend stops being chosen at its first failed
check instead of afterfallof them. When every healthy backend is
failing, all of them are used as before. The store format is
unchanged. -
slow_start(per service, or in[defaults];"0s", off, by
default, else1sto1h): a backend that turns from unhealthy to
healthy gets a growing share of traffic over that time, in 10 steps,
instead of its full share at once. It applies to client lookups and
DNS A/AAAA answers for every method butfailoverandstickywith a
key, and to SRV weights. The store gainsslow_start_mson services
andrecovered_aton backends; clients that do not know them (as
emfor-client 0.3.0 reading a version 1 store) give a recovered backend
its full share at once. See
Slow start. -
With
on_all_down = "open"and no healthy backend, clients and the DNS
server now use theunknownbackends (not decided yet, or whose checks
only give errors) before theunhealthyones, which failedfall
checks in a row: first theunknownones whose last check passed, then
everyunknownone, then every backend as before.closedand a stale
store are unchanged, and so is the store format; older clients still
choose among all backends. -
The
httpcheck takesexpect_body: text the response body must
contain, for health endpoints that answer 200 either way. The body is
read only when the status matches, up to 64 KiB. -
On macOS,
execcheck commands and plugins no longer inherit the
daemon's sockets and pipes. A plugin could keep a check connection
open for as long as it ran, so the backend never saw it closed. -
Peer groups notice a failure that only a member which is not one of the
backend's assigned checkers sees sooner (the known limit of 0.3.0).
Such a member now checks the backend itself at least every third
intervalinstead of every fifth, and when its own check fails while
the assigned checkers see the backend up it starts a fast recheck run
(every 200 ms, up tofallchecks), as after a peer'sdownreport.
With the defaults the backend is down there after at most about 6.4
seconds instead of about 14 (estimated from the schedule; 6 without a
group). In exchange a backend gets about 3 + (N − 3) / 3 checks per
intervalfrom N members instead of 3 + (N − 3) / 5. Measured with
interval = "2s"andfall = 3: 0.8 to 5.8 seconds, 3.3 seconds
median, over 12 failures started at different points of the schedule.
Clients
- Behaviour change:
balance = "sticky"with an affinity key now
honoursweight: a backend of weight w gets w shares of the keys. Its
value is the highest ofMD5(key + id)andMD5(key + id + "#" + i)
for i from 1 to w − 1, integers only, so every host and language still
picks the same backend for a key. With every weight 1 nothing changes.
A sticky service that already sets other weights moves some keys when
its clients are upgraded, and old and new clients pick differently for
those keys meanwhile, so upgrade the clients of such a service
together. Raising a weight only moves keys to that backend; slow start
does not apply, and a weight above 100 in the store counts as 100. A lookup costs one MD5 per unit of weight: 0.8 µs for 10
backends of weight 1, 72 µs for 10 of weight 100 (Apple M4 Max). The
Rust API addstesting::rendezvous_pick_weighted. See
sticky.
emfor-check
role = "replica"formysqlandpgsql, and for all three database
plugins it now means a replica that is replicating: a redis replica
whose link to its master isconnected, a MySQL or MariaDB replica
whose IO and SQL threads both run, a PostgreSQL standby with a WAL
receiver. Otherwise the backend is down with areplication:reason.
A redis replica that was up withrole = "replica"while its link
was down is now down.max_lag: withrole = "replica", the seconds a replica may be behind
before it is down. See Replicas for
what each plugin measures and the rights the checks need.emfor-check smtpchecks SMTP servers: the greeting must be 220 and
EHLOmust answer 250, on a new connection, optionally upgraded with
STARTTLS(starttls = true) or TLS from the start (tls = true,
port 465); then it sendsQUIT. It never logs in or sends mail. It is
down withgreeting:,ehlo:ortls: STARTTLS not offered. See
docs/deploy/plugins.md.emfor-check amqpchecks AMQP 0-9-1 brokers such as RabbitMQ: a
PLAINlogin and Connection.Open ofvhoston a new connection, over
plaintext or TLS, then a normal Connection.Close. It is down with
auth: 403 ACCESS_REFUSED - …for a rejected login and
vhost: 530 NOT_ALLOWED - …for a vhost the user cannot open. See
docs/deploy/plugins.md.emfor-check mongodbchecks MongoDB 4.4 and later:helloon a new
connection, plaintext or TLS, an optional SCRAM-SHA-256 login (user,
password_file,auth_source), androle = "primary"(a replica set
primary, a standalone server or mongos) orrole = "secondary", so
balance = "failover"follows the replica set's primary. See
docs/deploy/plugins.md.emfor-check grpcchecks gRPC services with the standard health
service:grpc.health.v1.Health/Checkover HTTP/2, plaintext or TLS
with ALPNh2, is up when it answersSERVING. It is down with
status: NOT_SERVING,grpc: 12 UNIMPLEMENTED(no health service) or
grpc: 5 NOT_FOUND: …(an unknownservice). See
docs/deploy/plugins.md.emfor-check quicchecks QUIC servers over UDP: a new QUIC version 1
handshake on every check, without 0-RTT, offering the ALPN values of
alpn(default["h3"]) and verifying the certificate (tls_verify,
server_name,ca_file; always TLS 1.3). It is up when the handshake
completes. Withhttp3 = trueit also sends an HTTP/3GETforpath
and judgesexpect_statusandexpect_bodylike the built-inhttp
check. It is down withquic: version negotiation failed,tls: …
(for exampletls: no application protocol (ALPN)),
quic: closed by peer: <code>[: <reason>],quic: stateless reset,
query: HTTP/3: …,unexpected status <n>orexpect_body: got "…".
UDP has no "connection refused", so a closed port or a firewall that
drops UDP is atimeout. The ...
emfor 0.3.0
Breaking, on FreeBSD only: the rc.d script keeps its pid files in
/var/run/emfor/. Scripts that read /var/run/emfor.child.pid must read
/var/run/emfor/emfor.child.pid, and installed copies of
examples/rc.d/emfor must be replaced (see
Deployment). Nothing else needs a change: configs, commands and the
--json output of 0.2.0 work as before.
The main addition is peer groups: the daemons of one site can pass
backend failures on, vote on a backend's state and split the checks
between them. Without a [peers] table the daemon behaves as in 0.2.0.
The store format (meta/schema_version 1) and the plugin protocol (1) are
unchanged; the new store key meta/peers is read only by emfor status.
As with every release, install libemfor from the same release as the
daemon.
Daemon
- New optional
[peers]table: the emfor daemons of onesitethat list
each other form a peer group. Members send each other a heartbeat every
second over UDP, signed with HMAC-SHA256 using the secret in
secret_file. When a member sees a backend go from healthy to unhealthy
it tells the others, which recheck that backend at once and every 200 ms
(up tofallchecks) instead of waiting for their next interval. A
report never changes a backend's state by itself. Reports are shared
only for services whosecheck,check_optionsandtimeoutmatch.
New metricsemfor_peer_up,emfor_peer_messages_rejected_totaland
emfor_peer_notifications_total. Without[peers]nothing changes.
See
docs/manual/03-configuration.md
and docs/deploy/daemon.md. - Majority vote: with three or more live members, each backend has three
assigned checkers, chosen by rendezvous hashing over the live members.
They send their own check results to the others, and twodownvotes
mark the backend down on every member, also where its own check passes;
the reason then readspeers 2/3: …, and a failure only this host sees
readslocal: …. With fewer members or no recent vote each daemon
decides from its own checks as before.emfor statusshows apeers:
line, members whose check definition differs, and the markerlocal-up;
--jsonhaspeers, from the new store keymeta/peers(format
version stays 1, clients ignore it). New metrics
emfor_peer_results_totalandemfor_peer_disagreements. - Split checks: a member that is not one of a backend's assigned checkers
no longer checks it every interval. While two assigned checkers agree
their results decide (up, ordownwithpeers 2/3: …); it still
checks the backend itself, with the service's own check, at least every
fifth interval, every interval while its own check fails, and whenever
the votes do not agree. A failure only such a member sees is noticed up
to about five intervals later. New metric
emfor_peer_checks_skipped_total. - New setting
peers(defaulttrue), per service or in[defaults]:
peers = falseleaves a service out of the peer group on this host. It
sends no reports or results for it, ignores those it receives, and
decides from its own checks only; a reload applies it and keeps the
backends' state.meta/peerslists such services inservices_off.
The host announces those services in its heartbeats (off: the first
4 bytes of SHA-256 of each name, as 8 hex digits, at most 80; with more
it logs a warning), and the other members leave it out when choosing
those services' assigned checkers, so each host can set it on its own.
Each entry ofmembersinmeta/peershasservices_offwith the
services that member leaves out, andemfor statususes it forvoted
andassigned_checkers.
emfor status
emfor statusmarks backends this daemon is not an assigned checker of
withvoted, addschecks N/M backends hereto thepeers:line when
there are any, and--jsonhasassigned_checkers(each backend's
assigned checkers, worked out frommeta/peers).emfor statusshows, after thepeers:line, one line per member that
leaves shown services out of the peer group:
peers: 10.0.0.6:5302 leaves out api.svc.emfor db.svc.emfor. Only the
services shown are listed (so--servicenarrows it);--jsonis
unchanged.emfor status --watchscrolls when the services and recent changes do
not fit in the terminal: thestore:anddaemon:lines stay at the
top, PgUp/PgDn (or Space) move one screen, Up/Down one line, Home/End
to the start or end, and the last row shows which lines are on screen.
Lines wider than the terminal are cut instead of wrapped.qor Ctrl-C
ends the watch, and the terminal is restored also after a crash. When
it cannot install its signal handlers it now exits 2 instead of 0.
emfor-check
emfor-check dnschecks DNS servers: it sends one query (name,
type, over UDP or TCP) and is up when the response code is one of
expect_rcode(defaultNOERROR) and, withexpect_answer = true, the
answer has a record of the asked type. Over UDP the query is sent again
once at half of thetimeout. Reasons start withrcode:,answer:,
query:,connect:ortimeout. See
docs/deploy/plugins.md.emfor-check radiuschecks RADIUS servers: it sends a Status-Server
(RFC 5997) by default, or an Access-Request with PAP for a test
account, with Message-Authenticator first, and is up when the server
answers with an accept signed with the shared secret from
secret_file. Reasons start withauth:,query:,connect:or
timeout; servers that check Message-Authenticator, such as
FreeRADIUS, do not answer a request made with the wrong secret, which
then shows astimeout. See
docs/deploy/plugins.md.emfor-check sipchecks SIP servers with an OPTIONS request over UDP,
TCP or TLS, and is up when the final status code is in
expect_status, by default 200 to 499: a proxy that answers 404 or
407 is working, while 503 (overload, maintenance) and other 5xx and
6xx codes are down withstatus: <code> <reason>. See
docs/deploy/plugins.md.emfor-check diameterchecks Diameter nodes (HSS, PCRF, OCS, DRA)
over TCP or TLS: a capabilities exchange and a device watchdog on a
new connection, then a Disconnect-Peer-Request. It is down with the
Result-Code and its RFC 6733 name, such as
cea: 3010 DIAMETER_UNKNOWN_PEERor
cea: 5010 DIAMETER_NO_COMMON_APPLICATION. TLS can present a client
certificate (tls_cert_file,tls_key_file), which nodes such as
freeDiameter require. See
docs/deploy/plugins.md.
Deployment
- The FreeBSD rc.d script failed to start emfor with any
emfor_user
other thanroot, such as the port's defaultemfor:daemon: pidfile ``/var/run/emfor.child.pid'': Permission denied. rc.subr runs the whole
command as${name}_user, the daemon(8) supervisor included, and that
user cannot write/var/run. The pid files are now in/var/run/emfor/,
whichservice emfor startcreates owned byemfor_user, and daemon(8)
no longer gets-u(it needs root). Replace installed copies of
examples/rc.d/emfor; scripts that read
/var/run/emfor.child.pidread/var/run/emfor/emfor.child.pidnow.
Documentation
- A user manual in English (docs/manual) and
Traditional Chinese (docs/manual.zh_TW):
concepts, installation, configuration, health checks, backend
selection, clients, DNS, operations and troubleshooting. - SECURITY.md (how to report a vulnerability),
CONTRIBUTING.md and
docs/compatibility.md, which says which
interfaces stay stable across releases.
Known limits
- Unchanged from 0.2.0: FreeBSD has not been tested on a host yet, peer
groups included (tests/resolver/freebsd.sh
checks the daemon, the rc.d script and the port example there). - Unchanged from 0.1.0: lookup throughput across threads levels off at
about two million a second on Linux
(docs/performance.md). - Unchanged from 0.2.0: with
[store] sync = false(the default), a host
crash can leave the store damaged; the daemon then sets it aside at the
next start and the state history is lost. - In a peer group, a failure that only a member which is not one of the
backend's assigned checkers sees is noticed there up to about five
intervals later than without a group. - A host announces at most 80 services left out of the peer group; the
other members still count it as a candidate for the rest.
emfor 0.2.0
Breaking: the daemon is started with emfor server. emfor without a
subcommand now only prints its usage and exits 2, so running it to see the
options no longer starts a second daemon. When upgrading from 0.1.x, change
the service definition before restarting:
| 0.1.x | 0.2.0 |
|---|---|
emfor --config X |
emfor server --config X |
emfor --check-config --config X |
emfor check-config --config X |
--check-config now only prints --check-config was replaced by "emfor check-config" and exits 2. emfor status is unchanged.
examples/rc.d/emfor,
examples/systemd/emfor.service and the
FreeBSD port example use the new commands; copies of the 0.1.x versions
must be replaced. See docs/deploy/daemon.md.
The store format (meta/schema_version 1) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon: the daemon and the clients must also agree on the lock
file's reader slots (below).
Daemon
- The store is no longer flushed to disk on every commit. The new
[store] syncsetting defaults tofalse(LMDB'sMDB_NOSYNC): a
stalled disk could make the heartbeat late and every client on the host
treat the store as stale. The store is rewritten on every start, so
nothing in it needs to survive a host crash. Setsync = trueto keep the
old behavior. Commits now run on a separate thread. - When the daemon starts and finds the store damaged (
MDB_CORRUPTED,
MDB_PAGE_NOTFOUNDorMDB_INVALID, as after a power loss without
flushing), it moves the file to<path>.corrupt, deletes the lock file,
and starts with a new store instead of exiting. The state history of
emfor status --historyis lost with it. See
docs/deploy/daemon.md. - An application process killed in the middle of a lookup (SIGKILL, OOM,
a crash) left its LMDB reader slot taken. Each such slot kept every page
freed after it from being reused, so the store grew until commits failed
withMDB_MAP_FULL, every client then saw a stale store, and restarting
the daemon did not help while applications kept the store open. The
daemon now frees these slots when it opens the store, about once a minute
after that, and right after a failed commit, and logs how many it freed. - The store's lock file has 1024 reader slots instead of LMDB's default
of 126. The daemon and every client ask for the same number, since the
process that opens the store first sizes the lock file. - While a backend's state is
unknown(after a start, a reload that adds
it, or a run of errors) and a check returnsupordown, the next check
runs aftermin(interval, 200ms)instead ofinterval, and the first
check starts within that time too. Withinterval = "2s"andrise = 2,
backends are known within about half a second of a start instead of 2 to
4 seconds, during which services withon_all_down = "closed"answered
ALL_DOWN. - Check reasons are escaped and bounded before they reach the store, the
logs andemfor status: tab, CR and LF become\t,\r,\n, other
control characters\u{1b}and the like, and a reason longer than 512
characters is cut and ends in…. A backend or plugin could send
terminal escape sequences, or reasons large enough to slow every lookup. - A plugin line longer than 65536 bytes no longer grows the daemon's memory
without bound: on stdout the plugin is stopped and restarted as if it had
exited; on stderr the rest of the line is discarded. - Signals:
emfor serverinstalls its signal handlers before it loads the
config. Loading starts plugins and can take seconds; a SIGHUP in that
time (aservice emfor reloador config management reloading right after
a start) used to kill the daemon, and now reloads once it has started. A
SIGTERM or SIGINT while loading at start, or during a reload, ends the
loading at once (a reload is then not applied) and the daemon exits 0,
instead of waiting up to about 10 seconds for slow plugins. - If the state writer stopped while the daemon ran, the daemon kept running
without a heartbeat and every client saw a stale store; it now exits with
status 70 (EX_SOFTWARE), which the systemd unit restarts. A backend
check task that stopped left its backend in its last state; it is now
started again fromunknown, at once and then after 1s, 2s, 4s and so on
up to 60s if it keeps stopping. Both only happen through a bug (a panic). - A
[dns] zonelonger than 242 characters made the daemon panic on every
query in the zone: the SOA namehostmaster.<zone>went past the
253-character limit. Such a zone is now a config error. /metricsreports the store and the plugins:
emfor_store_last_commit_timestamp_seconds,emfor_store_used_bytes
andemfor_store_map_size_bytes(commits fail withMDB_MAP_FULLwhen
the used bytes reach the map size), andemfor_plugin_upand
emfor_plugin_restarts_totalper plugin. Before, a crashing plugin showed
only aserrorchecks.- The
httpchecker shares one HTTP client among a service's backends
unlesshost_headeris a domain name (which needs one client per
backend to connect to it), and no longer builds clients while holding a
lock. emfor status --watchrestored the terminal only on Ctrl-C. Killed, or
ended by a closed ssh session (SIGHUP), it left the terminal on the
alternate screen with the cursor hidden.
Clients
- Applications kept reading the old store file after it was replaced (the
daemon moving a damaged store aside, or the file deleted and created
again), so they saw a stale store until they were restarted. A lookup
that finds the store stale, or fails with an error other thanALL_DOWN
from a fresh store, now checks whether the store path holds another file,
at most once every 100 ms per process, and switches to it. When the new
file cannot be opened yet, lookups keep using the old one. This covers
every client library. - Lookups take read locks only, so lookups on many threads no longer queue
on a mutex, and a round-robin lookup no longer allocates the service name
each time. Failing and stale lookups run about as fast as successful
ones. - The Python client looked for
libemfor.so.0, a name libemfor is never
installed under, so with only the runtime library installed (no
libemfor.sosymlink, not in theldconfigcache)import emfor
failed. It now looks for the versioned names derived from
emfor.__version__:libemfor.so.0.2andlibemfor.0.2.dylibfor 0.2.x.
Deployment
- The FreeBSD rc.d script has
daemon(8)startemforagain 2 seconds
after it exits, likeRestart=on-failurein the systemd unit. The
supervisor's pid is in/var/run/emfor.pidand emfor's in
/var/run/emfor.child.pid.service emfor reloadchecks the config first
and fails without touching the daemon when it is invalid. service emfor configtest, which the rc.d script also runs before
starting, checked the config as root and so started the config's plugins
as root even withemfor_userset. It now runs asemfor_user.
Replace copies of examples/rc.d/emfor.- The example FreeBSD port installs the rc.d script with
USE_RC_SUBR
fromfiles/emfor.in; a test keeps it in step with
examples/rc.d/emfor. - The systemd unit and docs/deploy/daemon.md use
/usr/local/bin/emfor, wherecargo installand the FreeBSD port put
it, instead of/usr/local/sbin/emfor. Change the unit's paths if you
installed tosbin. - The docs spell out what the default
[store] lock_file_mode = "0666"
allows: any local user can write the lock file and so stop the daemon's
commits or disturb lookups. See "Who can use the store" in
docs/deploy/daemon.md for
limiting the store to a group. The default is unchanged.
Known limits
- FreeBSD has not been tested on a host yet: neither the daemon nor the
rc.d script and the port example have run there
(tests/resolver/freebsd.sh checks them). - Unchanged from 0.1.0: lookup throughput across threads levels off at
about two million a second on Linux
(docs/performance.md). - With
[store] sync = false(the default), a host crash can leave the
store damaged; the daemon then sets it aside at the next start and the
state history is lost.
emfor 0.1.1
The repository moved to github.com/embark-systems/emfor. The daemon,
emfor-check, the store format and the clients behave as in 0.1.0. As
with every release, install libemfor from the same release as the
daemon.
Packaging
cargo install --path crates/emfor-daemoninstalls onlyemfor. In
0.1.0 it also installedemfor-test-plugin, a plugin used only by
emfor's own tests; it is now built only with thetest-pluginfeature.- docs/deploy/daemon.md gives the
cargo install
commands forcrates/emfor-daemonandcrates/emfor-check, including
for packaging systems that runcargo installper crate. The repository
root is a workspace, socargo install --path .fails.
Clients
- The Go module path is
github.com/embark-systems/emfor/clients/go
(wasgithub.com/vanillahsu/emfor/clients/go). Its versions are tags
clients/go/vX.Y.Z:
go get github.com/embark-systems/emfor/clients/go@v0.1.1.
Development
- RELEASING.md lists the steps of a release; a test checks
that the Python, Perl, PHP and Go clients carry the workspace version. tests/clients/Dockerfiledescribes the Debian image the local checks
run in.
Known limits
- Unchanged from 0.1.0: FreeBSD has not been tested on a host yet, and
lookup throughput across threads levels off at about two million a
second on Linux (docs/performance.md).
emfor 0.1.0
The first release. The daemon, emfor-check and every client come from
this one version; libemfor and the clients built on it must be upgraded
together with the daemon (they share its LMDB build).
Daemon
- Health checks:
tcpwith optionalsend/expectand TLS,
http/https,execfor Nagios-style scripts, and long-running
plugins speaking protocol v1. Per-service
interval,timeout,rise,fallandjitter, with[defaults]. - State store: a local LMDB file, record format version 1, with a
heartbeat so readers notice a daemon that stopped. Backend state,
latency, the last failure reason and the last 20 state changes per
backend. - Balance methods:
round-robin,sticky(rendezvous hashing on an
affinity key),failover,weighted,locality(samesitefirst) and
fastest(backends within a latency margin of the quickest).
on_all_down = "open"or"closed"decides what an all-down service
returns. - DNS server (optional
[dns]): A, AAAA and SRV answers for its own zone
(defaultsvc.emfor), meant to sit behind the host's resolver; a warning
when a service's backends use different ports. /metrics(optional[metrics]): Prometheus metrics for the daemon,
every service and backend.emfor status: the daemon's state, every service and backend, recent
state changes and flapping backends;--service,--jsonand--watch.SIGHUPreloads the config when it is fully valid;--check-config
checks a config without running.EMFOR_PREFIXat build time sets the default config path
(/etc/emfor/emfor.tomlwithout it).
emfor-check
- Plugins for redis, MySQL / MariaDB, PostgreSQL and LDAP, with TLS.
- The
emfor-plugincrate for writing plugins in Rust.
Clients
- One implementation in Rust (
emfor-client), also built with cargo-c as
the C librarylibemfor. - Go, Python, Perl and PHP clients wrapping
libemfor. - Every client returns
ip:portor one of the error codesUNAVAILABLE,
INCOMPATIBLE,NOT_FOUNDandALL_DOWN, checked against shared
conformance data (tests/conformance/v1). Configured only by
EMFOR_STOREandEMFOR_MAX_STALENESS; safe across threads andfork. - A lookup takes about 1 µs; see docs/performance.md.
Deployment
- Guides for the daemon, systemd, metrics and alerting
(docs/deploy/daemon.md), for forwarding the DNS
zone from dnsmasq, unbound, systemd-resolved and macOS
(docs/deploy/dns-resolver.md), and for
emfor-check(docs/deploy/plugins.md). - Examples: a fully commented config, a systemd unit (run under systemd
257), a FreeBSD rc.d script, and Prometheus alerting rules with promtool
tests.
Known limits
- FreeBSD is a deployment platform but has not been tested on a host yet;
the CI job that covers it is run by hand and has not passed yet. - Lookup throughput across threads levels off at about two million a
second on Linux; see docs/performance.md.