Repository navigation
emfor 0.4.0
Breaking: the store format is version 2. The daemon writes
meta/schema_version 2, which adds the backend field backup. Clients
and emfor status from this release read versions 1 and 2; older ones
fail every lookup on a version 2 store with INCOMPATIBLE. Upgrade the
clients on a host before the daemon (see
Store format version).
Peer messages are encrypted by default. Members running 0.3.0 cannot
read them and drop them as malformed. To upgrade a peer group, set
encrypt = false in [peers] on each host as it is upgraded; once every
member runs this release, remove the line and reload each daemon.
Breaking, on FreeBSD only: examples/rc.d/emfor is gone, and the
example port keeps the store in /var/db/emfor (see Deployment).
The main additions are backup backends, slow start, a preference for
backends whose last check passed, sticky lookups that honour weight,
encrypted peer messages, and the smtp, amqp, mongodb, grpc and
quic plugins of emfor-check. The plugin protocol (1) is unchanged.
As with every release, install libemfor from the same release as the
daemon.
Daemon
-
Peer group messages are encrypted (
EMP2: ChaCha20-Poly1305 with a key
derived fromsecret_fileby HKDF-SHA256 and a random nonce per
message), so backend names, addresses and reasons no longer travel in
the clear. New[peers] encrypt(defaulttrue, applied on reload):
falsesends the signed plaintextEMP1of 0.3.0. Both formats are
accepted either way; withencrypt = truethe daemon warns once a
minute per member that still sendsEMP1. A message that does not
decrypt counts asauthinemfor_peer_messages_rejected_total.
Secret rotation works as before. There is no forward secrecy: whoever
later gets the secret can read recorded messages, so rotate it
regularly. See[peers]. -
Backup backends: a backend table takes
backup = true. Backups are
used only while no other backend passes its checks: clients and the DNS
server take the primaries whose last check passed, else the backups
whose last check passed, else the healthy primaries, else the healthy
backups, andbalanceworks within that set;fastestdecides which
backends are fast among the primaries and among the backups
separately. When no backend is
healthy,on_all_downdecides as before and backups count like the
others. SRV answers list only the chosen set. At least one backend of a
service must not be a backup, andbackupis applied on reload. See
Backup backends. -
Clients and the DNS server leave out a healthy backend whose last
check failed (itsreasonis set) while another healthy backend passed
its last check, so a backend stops being chosen at its first failed
check instead of afterfallof them. When every healthy backend is
failing, all of them are used as before. The store format is
unchanged. -
slow_start(per service, or in[defaults];"0s", off, by
default, else1sto1h): a backend that turns from unhealthy to
healthy gets a growing share of traffic over that time, in 10 steps,
instead of its full share at once. It applies to client lookups and
DNS A/AAAA answers for every method butfailoverandstickywith a
key, and to SRV weights. The store gainsslow_start_mson services
andrecovered_aton backends; clients that do not know them (as
emfor-client 0.3.0 reading a version 1 store) give a recovered backend
its full share at once. See
Slow start. -
With
on_all_down = "open"and no healthy backend, clients and the DNS
server now use theunknownbackends (not decided yet, or whose checks
only give errors) before theunhealthyones, which failedfall
checks in a row: first theunknownones whose last check passed, then
everyunknownone, then every backend as before.closedand a stale
store are unchanged, and so is the store format; older clients still
choose among all backends. -
The
httpcheck takesexpect_body: text the response body must
contain, for health endpoints that answer 200 either way. The body is
read only when the status matches, up to 64 KiB. -
On macOS,
execcheck commands and plugins no longer inherit the
daemon's sockets and pipes. A plugin could keep a check connection
open for as long as it ran, so the backend never saw it closed. -
Peer groups notice a failure that only a member which is not one of the
backend's assigned checkers sees sooner (the known limit of 0.3.0).
Such a member now checks the backend itself at least every third
intervalinstead of every fifth, and when its own check fails while
the assigned checkers see the backend up it starts a fast recheck run
(every 200 ms, up tofallchecks), as after a peer'sdownreport.
With the defaults the backend is down there after at most about 6.4
seconds instead of about 14 (estimated from the schedule; 6 without a
group). In exchange a backend gets about 3 + (N − 3) / 3 checks per
intervalfrom N members instead of 3 + (N − 3) / 5. Measured with
interval = "2s"andfall = 3: 0.8 to 5.8 seconds, 3.3 seconds
median, over 12 failures started at different points of the schedule.
Clients
- Behaviour change:
balance = "sticky"with an affinity key now
honoursweight: a backend of weight w gets w shares of the keys. Its
value is the highest ofMD5(key + id)andMD5(key + id + "#" + i)
for i from 1 to w − 1, integers only, so every host and language still
picks the same backend for a key. With every weight 1 nothing changes.
A sticky service that already sets other weights moves some keys when
its clients are upgraded, and old and new clients pick differently for
those keys meanwhile, so upgrade the clients of such a service
together. Raising a weight only moves keys to that backend; slow start
does not apply, and a weight above 100 in the store counts as 100. A lookup costs one MD5 per unit of weight: 0.8 µs for 10
backends of weight 1, 72 µs for 10 of weight 100 (Apple M4 Max). The
Rust API addstesting::rendezvous_pick_weighted. See
sticky.
emfor-check
role = "replica"formysqlandpgsql, and for all three database
plugins it now means a replica that is replicating: a redis replica
whose link to its master isconnected, a MySQL or MariaDB replica
whose IO and SQL threads both run, a PostgreSQL standby with a WAL
receiver. Otherwise the backend is down with areplication:reason.
A redis replica that was up withrole = "replica"while its link
was down is now down.max_lag: withrole = "replica", the seconds a replica may be behind
before it is down. See Replicas for
what each plugin measures and the rights the checks need.emfor-check smtpchecks SMTP servers: the greeting must be 220 and
EHLOmust answer 250, on a new connection, optionally upgraded with
STARTTLS(starttls = true) or TLS from the start (tls = true,
port 465); then it sendsQUIT. It never logs in or sends mail. It is
down withgreeting:,ehlo:ortls: STARTTLS not offered. See
docs/deploy/plugins.md.emfor-check amqpchecks AMQP 0-9-1 brokers such as RabbitMQ: a
PLAINlogin and Connection.Open ofvhoston a new connection, over
plaintext or TLS, then a normal Connection.Close. It is down with
auth: 403 ACCESS_REFUSED - …for a rejected login and
vhost: 530 NOT_ALLOWED - …for a vhost the user cannot open. See
docs/deploy/plugins.md.emfor-check mongodbchecks MongoDB 4.4 and later:helloon a new
connection, plaintext or TLS, an optional SCRAM-SHA-256 login (user,
password_file,auth_source), androle = "primary"(a replica set
primary, a standalone server or mongos) orrole = "secondary", so
balance = "failover"follows the replica set's primary. See
docs/deploy/plugins.md.emfor-check grpcchecks gRPC services with the standard health
service:grpc.health.v1.Health/Checkover HTTP/2, plaintext or TLS
with ALPNh2, is up when it answersSERVING. It is down with
status: NOT_SERVING,grpc: 12 UNIMPLEMENTED(no health service) or
grpc: 5 NOT_FOUND: …(an unknownservice). See
docs/deploy/plugins.md.emfor-check quicchecks QUIC servers over UDP: a new QUIC version 1
handshake on every check, without 0-RTT, offering the ALPN values of
alpn(default["h3"]) and verifying the certificate (tls_verify,
server_name,ca_file; always TLS 1.3). It is up when the handshake
completes. Withhttp3 = trueit also sends an HTTP/3GETforpath
and judgesexpect_statusandexpect_bodylike the built-inhttp
check. It is down withquic: version negotiation failed,tls: …
(for exampletls: no application protocol (ALPN)),
quic: closed by peer: <code>[: <reason>],quic: stateless reset,
query: HTTP/3: …,unexpected status <n>orexpect_body: got "…".
UDP has no "connection refused", so a closed port or a firewall that
drops UDP is atimeout. The plugin uses quinn 0.11, h3 0.0.8 and
h3-quinn 0.0.10, which make theemfor-checkrelease binary about
1.3 MiB (13%) larger, 11.2 MiB on macOS arm64; the daemon does not
link them. See docs/deploy/plugins.md.ldapreasons no longer end with a NUL byte when the server is Active
Directory, which ends its diagnostic message with one: a wrong password
is nowauth: 49 invalidCredentials: 80090308: LdapErr: …, data 52e, ….- A server that closes or resets the connection during the TLS handshake
is down withtls: connection closed during the TLS handshake (does the server have a certificate?)instead oftls: tls handshake eofor
tls: Connection reset by peer (os error 54). Active Directory does
this on port 636 when it has no certificate. The same reason applies to
the daemon'stcpcheck withtls = true.
Deployment
- The FreeBSD rc.d script starts
daemon(8)with-f. Without it,
daemon(8)and emfor kept the terminal or SSH session that ran
service emfor start, so that session never closed. The rc.d script,
the port and the tests have now run on FreeBSD 15.1-RELEASE (arm64). - The example FreeBSD port puts the store in
/var/db/emfor, where
FreeBSD keeps daemon state, instead of/var/lib/emfor: it creates that
directory, and its sample config sets[store] path = "/var/db/emfor/state.mdb". The daemon's and the clients' default stay
/var/lib/emfor/state.mdb, so applications on the host set
EMFOR_STORE. After updating an installed port, move the store or keep
the old[store] path; the history is all it holds. examples/rc.d/emforis gone. The port's
examples/freebsd/files/emfor.inis the only rc.d script; to use it
without the port, fill in%%PREFIX%%(chapter 2 of the manual shows
the command). Itsemfor_userdefaults toemfor, notroot, so
create that user or setemfor_user.
Documentation
- The operations chapter of the manual says what a host crash loses with
[store] sync = false(the state history only) and how to keep the
store on a tmpfs (Keeping the store in
memory). - docs/deploy/plugins.md says which servers
each plugin was checked against, and what was checked by hand against
Active Directory.
Development
- cargo-fuzz targets for plugin answers, peer messages, metrics, plugin
lines and the config (fuzz/); RELEASING.md runs each
for 10 minutes before a release.
Known limits
- FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM), the port with
make stage check-plist install; not with poudriere, on 14.x or on
amd64. - Peer messages have no forward secrecy: whoever later gets
secret_filecan read recorded messages. Rotate the secret regularly. - The
ldapplugin has been checked against Active Directory only
without a certificate and without a successful login; themysql
plugin only against MariaDB. - Unchanged from 0.1.0: lookup throughput across threads levels off at
about two million a second on Linux
(docs/performance.md). - Unchanged from 0.2.0: with
[store] sync = false(the default), a host
crash can leave the store damaged; the daemon then sets it aside at the
next start and the state history is lost. - Unchanged from 0.3.0: a host announces at most 80 services left out of
the peer group; the other members still count it as a candidate for
the rest.