Repository navigation
emfor 0.5.0
Stricter validation: a config that 0.4.0 accepted can now be refused.
emfor check-config, and so a start or a reload, now refuses options that
could never work and that 0.4.0 let through, to fail later: an interval
above 24h, a timeout below 10ms, an http path with # or \, an
exec or plugin command with a NUL, a user or password with a NUL where
the protocol cannot carry one (mysql, pgsql, amqp), an amqp password too
long for AMQP's first frame, a diameter list too long for one message, a
ca_file with no usable CA, and more (see Daemon and emfor-check). Run
emfor check-config with the new binary before you upgrade a host.
Peer messages have a second encrypted format, EMP3
(XChaCha20-Poly1305, which has no practical limit on the number of
messages per secret). Each member is sent the format it understands, so a group of
0.4.0 and 0.5.0 members works in any upgrade order without a
configuration change (see Daemon).
The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon; its files are now libemfor.so.0.5 and
libemfor.0.5.dylib.
The main additions are EMP3, [peers] accept_plaintext and
previous_secret_file, a DNS server that limits UDP responses per client
network and keeps answering while a commit is slow, plugins that are
restarted when they stop answering, and clients that no longer end the
application with SIGBUS on a truncated store or hang a forked child on
FreeBSD. The rest are fixes, most of them in emfor-check and the DNS
server.
Daemon
- Peer messages have a new encrypted format,
EMP3: XChaCha20-Poly1305
with a 24-byte random nonce and a key of its own derived from the
secret.EMP2's 12-byte random nonce is safe for about 2^32 messages
per secret, which a large group can send within days; a repeated nonce
would let an observer read and forge messages. Withencrypt = truethe
format is picked per member: everyEMP2message now carries
"emp3": truein its encrypted payload, and a member is sentEMP3once
the latest message accepted from it isEMP3or such anEMP2, and
EMP2otherwise (also after it sendsEMP1), so members of this
release useEMP3between each other andEMP2with 0.4.0 members. A
group needs no configuration change to upgrade or to go back; a host
going back to 0.4.0 dropsEMP3messages for about its first second (see
Peer group). - A store file shorter than LMDB records it (changed in place, or its
tail lost in a power loss) is moved to<path>.corruptat start like
other damage, with a warning sayingtruncated. Before, reading it to
check it ended the daemon withSIGBUS, again at every restart.
After every commit the daemon extends the file with zero bytes to the
length LMDB records, which LMDB itself can leave a few pages short, so
clients never take a healthy store for a truncated one. A store an
older emfor left short that way is moved aside on the first start
after the upgrade (only the state history is lost). - A plugin that stops answering is restarted. Until now only a plugin
that exited or closed its stdout was restarted: one that kept running
but answered nothing (stuck in a loop, deadlocked, stopped) left every
check of its backendserrorfor good, and the requests for it queued
up without bound. Now, when the callers of 3 requests in a row give up
waiting and no answer to any request arrives in between, the daemon
logs that the plugin stopped answering, kills it and restarts it with
the usual backoff. A plugin that answers withintimeout_msis never
affected (see Plugin protocol). - A single key or value in the store that is not UTF-8 no longer blocks
the daemon. Undersvc/it made every commit fail from start-up on,
so clients saw a stale store for good; underhist/it kept every
service's history from loading. Such entries are now deleted, or
discarded with a warning, on their own. httphost_headermust be a host with an optional port
(api.internal,api.internal:8443,[2001:db8::1]). Values with
spaces, a path or a scheme passedcheck-configbut made every check
down, or silently checked whatever host the name resolved to instead
of the backend.- State changes already queued when one is written are committed in the
same transaction, so a burst of changes (a network split) costs one
commit, not one per backend. - The queues from checks to the peer task and to a plugin's stdin are
bounded. A peer task that falls more than 4096 messages behind drops new
downandresultmessages instead of holding them (counted in the new
emfor_peer_queue_dropped_total, logged once a minute); checks never
wait for it. A plugin that stops reading its stdin gets at most 1024
waiting requests; further checks areerrorat once with
plugin <name> busy: it is not reading its requestsinstead of each
waiting for its timeout while memory grows. tls_cert_fileandtls_key_file(thediameterplugin) are parsed
again only when they change, likeca_file: an unchanged file costs one
statper check instead of a read and a parse, and a renewed one still
takes effect at the next check. Parsedca_file, certificate and key
files that no check used for 10 minutes are released, so files removed
from the config by reloads no longer stay in memory.- The DNS server limits UDP responses to each client network (IPv4 /24,
IPv6 /56) to[dns] rate_limita second, 100 by default (0turns it
off). An SRV answer can be many times the size of its query, so a server
reachable from other hosts could be used to flood a third party with
forged-source queries. Past the limit every other response is an empty
truncated one, which makes a real client retry over TCP, and the rest
are dropped;emfor_dns_rate_limited_totalcounts them. Loopback
clients and TCP are never limited. At start the daemon now warns about
each[dns] listenaddress that is not a loopback address. - A slow commit no longer freezes DNS answers and
/metrics. While the
state store waited for the disk ([store] sync = true, or the kernel
writing back), the daemon stopped applying check results, so DNS kept
answering with the states from before the commit and the results
queued up in memory. Results are now applied as they arrive; the
changes made meanwhile are written together by the next commit, which
starts as soon as the slow one ends. - A reload that changes
intervalorjitterapplies to the wait
already under way. Before, going from10mto2stook up to 11
minutes to apply. - The 200ms checks while a backend is
unknownstop afterrise + fall
in a row; results that never reach either count no longer check about
5 times a second for good. intervalis at most24handtimeoutat least10ms. Larger
intervals could panic the check task (with a hugefall), and a 1ms
timeout gave plugins 0ms.[store] map_sizeis rounded up to a multiple of 64KiB. A size LMDB
refuses (such as"1048577") passedcheck-configand then stopped
the daemon from starting.- A check task that is restarted after a bug now records the backend's
move tounknownas a transition (reasoncheck task restarted);
before, the stored state changed silently. - The DNS server's view of a service is updated per backend instead of
copying the whole service on every check. - DNS over TCP serves at most 128 connections at once and closes a
connection with no query for 5 seconds, so clients can no longer use
up the daemon's file descriptors. - If a DNS listener stops while the daemon runs, the daemon logs it and
exits with status 70, so the service manager restarts it; before, DNS
went silent. - DNS answers BADVERS to EDNS versions above 0 and REFUSED to classes
other than IN; before, a CH query got IN records. - DNS compares names label by label: a label containing a
.byte no
longer puts a name outside the zone inside it. - Over UDP, a reply whose single record or SOA does not fit (names near
255 bytes) is truncated by emfor itself and keeps its EDNS record. - Plugins run in a process group of their own. When a plugin exits, is
stopped, or is killed because it stopped answering, the processes it
started are killed too; before, they could outlive it. - A plugin's stderr lines and invalid stdout lines are rate limited: up
to 100 at once, then 10 a second each. The log notes how many lines
were dropped, so a plugin writing in a loop can no longer fill the
log. - A
ca_fileis parsed again only when it changes (inode, size or
times), not on every check. A replaced file still takes effect at the
next check. - http clients for checks (one per backend with a domain
host_header)
are built off the async worker threads, because building one reads the
system CA store. The peersecret_fileis read off them too. - Documented: a connection that fails because this host has no route to
the backend, or its own firewall blocks it, isdown, noterror.
That has always been the behaviour. - When a check task is restarted after a bug while its backend is
unknown, the backend's latency is now cleared and its reason set to
check task restarted, in the store and in DNS. - A late result from the check task of a backend removed by a reload can
no longer mark that backend healthy after a later reload adds it back.
Each check task now has its own generation, and a backend keeps its
state exactly while its task keeps running. check-confignow refuses an httppathwith#,\or a control
character. The request URL cut or rewrote these, so another path was
checked.- With
[dns], a service name in the zone must leave room for the SRV
target name of each backend (<ip label>.<service>, at most 253
characters). Before, such a service passedcheck-configand every
SRV query for it got SERVFAIL. - An exec or plugin
commandwith a NUL in any element is refused; it
could never be started. - A tcp
ca_filewith no certificate usable as a CA is refused at
validation instead of failing every check withca_file: no usable certificate. - On start, state history entries with a state this version does not
know (left by a newer one) are dropped instead of discarding that
service's whole history.
Peers
- New
[peers] accept_plaintext(defaulttrue, applied on reload).
falsedrops every plaintextEMP1message without checking it,
counted under the new reasonplaintextof
emfor_peer_messages_rejected_total, so a group whose members all
encrypt no longer has to accept signed plaintext. It needs
encrypt = true;check-configrefuses it otherwise. - A kept result counts once. A member kept another member's latest result
for 3 of the sender's intervals and counted it at every one of its own
check times in that span, so withinterval = "1s"here and"30s"at
the assigned checkers onedownfrom each of two checkers moved
fallup to about 90 times and marked a backend down that the checkers
themselves still saw healthy. Now each result counts at one check time;
a check time decided by votes alone needs a new vote, and an own check
that only repeats counteddownvotes while it passes changes nothing
but is still counted inemfor_backend_checks_total. With equal
intervals most check times get new results, so little changes (see
spec peer-group). - New optional
[peers] previous_secret_file: the secretsecret_file
held before its last change. A daemon restarted within 120 seconds of
that change keeps signing with and accepting the old secret as if it
had not restarted. Before, members whose file had not changed yet
rejected it asauthuntil they changed. See
[peers]. - A second change of
secret_filewithin 60 seconds of the first no
longer signs for a minute with the intermediate secret, which members
whose file had not changed did not know. The daemon keeps signing with
the earlier one, and accepts every secret replaced in the last 120
seconds. - Peer messages that arrive out of order are accepted once, instead of
being dropped asreplay. AnEMP1message whose tag does not match
now counts asautheven when its payload is not JSON: the tag is
checked before the payload is parsed. A message of a kind this version
does not know is accepted (for liveness) even when it reuses a known
field name with another type. - A flood of datagrams from many source addresses no longer suppresses
the warnings about real members' dropped messages. - A
resultordownsent after a reload that changedcheck,
check_optionsortimeoutduring the check now carries the check
definition digest the check ran with, not the new one.
emfor-check
- Reasons longer than 512 characters are cut to 512 followed by
…
before they are sent (the daemon keeps no more). A backend's long error
text, such as a 70 KB redis-ERR, made the answer line longer than the
daemon's 65536-byte limit, so the daemon restarted the plugin, again at
every check of that backend. mongodb: a server asking for more than 1000000 SCRAM iterations is
down withauth: server asks for <n> SCRAM iterations, more than 1000000; one asking for 4294967295 held a worker thread for many
minutes. The key derivation now runs off the async workers, so the
check's timeout ends a slow login.sipover TCP or TLS: a response with a Content-Length near the
largest 64-bit number made the plugin loop forever on the same
provisional response, holding a worker thread at full CPU, with no
timeout to end it. Such a response is nowdownwith
query: response longer than 65536 bytes.- A check that panics is answered
errorwithcheck panicked: <message>(emfor-plugin crate). Until now it sent no answer, so the
daemon waited out the timeout. mysql: a NULL or non-integer answer toSELECT 1or
@@global.read_onlyis down withquery: <query> answered <value>, want an integer. Until now the check panicked and the backend ended
upunknown.mysql: a check cut off by its timeout closes its connection within 1
s. mysql_async's cleanup task waited with no limit for a server that
stalled mid-result, leaking a task and a socket at every check.mysql: a CA certificate that cannot be used as a root is skipped, as
in the other plugins, instead of failing every TLS handshake.redis: a reply over 2 MiB is down withquery: invalid reply: reply too long, and a partly received reply is parsed again only after a
line feed arrives.mongodb: ahelloreply nesting documents more than 100 deep is
down withquery: malformed reply. A reply of about 8000 levels
overflowed the stack and aborted the plugin.mongodb: validation refuses apassword_filewhose password is not
printable ASCII. Until then such a service waserrorat every check.quic: HTTP/3 response headers are limited to 65536 bytes (h3 allowed
any size).- A
ca_filewith no certificate usable as a CA is refused at
validation (no usable certificate), as the checks already did. - diameter: validation builds the TLS client configuration with
tls_cert_fileandtls_key_file, so a key that does not belong to
the certificate is reported ontls_key_fileinstead of failing every
check. - mysql and pgsql
userand pgsqldatabaseand password must not
contain NUL. The amqpuserand password must not contain NUL (PLAIN
separates them with NUL). A pgsql password file that gained a NUL
after validation iserrorat the check instead ofdown. A mysql
password may still contain NUL; the plugin documentation says when
one fails to log in (caching_sha2_passwordwithout TLS). - amqp: a password too long for the login to fit in AMQP's 4096-byte
first frame is refused at validation, and iserrorat a check when
the file grew since. Before, every check sent a frame the
broker may refuse, and the backend could be down at every check. - diameter: options whose Capabilities-Exchange-Request would not fit in
Diameter's 24-bit message length are refused at validation. Before,
the length was cut silently and every check sent a malformed request. - sip
uriandfrommust not contain<or>, which ended the
To/From value early. - sip, diameter, mongodb, amqp and ldap read messages through one shared
framed reader (checked lengths, one size cap, EOF handling); reasons
are unchanged. The ldap BER length check can no longer overflow on
32-bit platforms.
emfor status
- A truncated store makes
emfor statusprintcannot open store <path>: truncated: ...and exit 2 instead of ending withSIGBUS. emfor status | head -1keeps the normal exit status instead of
exiting 2 silently; other output write errors are printed.--watchopens the store again when the file is replaced, instead of
showing the old file as stale forever.- A record from a newer daemon is shown instead of failing: an unknown
state asunrecognized(red), an unknownon_all_downas
on_all_down=unknown.
Clients
- A store file already shorter than LMDB records it when the client opens
it makes lookups fail withUNAVAILABLE(the message saystruncated)
instead of ending the application withSIGBUS. A store changed in
place while it is open can still do that: replace it withmv, never
cpover it ortruncateit. - A running application keeps reading the store after the daemon is
restarted with a larger[store] map_sizeand the store grows past
the old size. Before, every lookup failed withUNAVAILABLE
(MDB_MAP_RESIZED) until the application restarted. - A child forked while other threads of the parent were looking up no
longer hangs in its first lookup. On FreeBSD this did not hold at
first: a child waited for ever, within the first few dozen forks of
the fork test (measured on FreeBSD 15.1-RELEASE, arm64), on a lock of
the threading library that a thread of the parent held inside LMDB at
the fork; 0.4.0 has the same hang there (its tag, built and run with
the same tests: 10 of 10 runs of the Rust test and 5 of 5 of the C
test failed). On FreeBSD the client now makesforkwait until no lookup
is inside the store (pthread_atforkhandlers, registered on FreeBSD
only), andlibemforcan no longer be unloaded there, since the
handlers stay registered. Lookup throughput did not change (measured
with 1, 4 and 8 threads). The lock file and the daemon are the same.
Not guarded: aforkfrom a signal handler that interrupted a lookup
of the same thread (it returns, but the child can still hang), and other
users of process-shared mutexes in the same process (see
Threads, processes and fork). - A store file that exists but cannot be opened is tried again at most
once a second; lookups in between fail with the sameUNAVAILABLE
error. A missing store is still tried on every lookup. emfor.hno longer defines the internal macrosREPLACE_CHECK_MSand
MAX_STICKY_WEIGHT.- Python: the library file named for this release (
libemfor.so.0.5,
libemfor.0.5.dylib) is tried before whatctypes.util.find_library
finds, which may be another release's. - C API:
emfor_lookupcalled from a thread-local destructor during
thread exit returns a status instead of aborting the process. - A backend
stateoron_all_downvalue written by a newer daemon no
longer makes the lookup fail withINCOMPATIBLE: an unknown state
counts asunhealthy, an unknownon_all_downasopen, so a later
release can add such values without raising the store format version.
Deployment
- The example systemd unit sets
LimitNOFILE=65536.
Development
tests/plugins/mysql.sh [version]runs themysqlplugin's live tests
against MySQL from themysqlDocker images (8.4 unless given), with
the layouttests/plugins/run.shuses for MariaDB. They pass on MySQL
8.0 and 8.4, so the 0.4.0 known limit that the plugin was checked only
against MariaDB no longer holds for those versions.emfor-conformancewrites a new store just before each run of the
lookup program, so slow runs earlier in a case cannot make afresh
store stale. A failure in a run longer than the staleness limit says
so. An expected error now needs a non-empty message.- Fuzz builds keep overflow checks and debug assertions on, and run each
input on a 2 MiB stack like a tokio worker's; an integer overflow or a
too-deep recursion is now a crash, as it would be in the daemon. - New
dnsfuzz target for the DNS server's request handling and
response size trimming. peers.rsis split into apeers/module (wire, crypto, stats,
group, targets, task); the peer group's live members have one owner,
and service digest lookups go through one function.
Known limits
- FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests,
the rc.d script, the port withmake stage check-plist install, and
the fork tests. Not tested: poudriere, 13.x, 14.x, amd64 (the optional
CI job covers 14.3 on amd64, and had not been run for this release). - FreeBSD's threading library can hang a child forked while other threads
use a process-shared mutex (LMDB's reader mutex is one), and can
deadlock such a parent.libemforworks around it for its own lookups
(see Clients; 0.4.0 has the problem). Not covered: other users of
process-shared mutexes in the same process, aforkfrom a signal
handler that interrupted a lookup of the same thread, and a
libemfor.alinked statically into a module that is unloaded. A patch
for libthr, tested on 15.1-RELEASE only, is in
docs/freebsd/. - Peer messages have no forward secrecy: whoever later gets
secret_filecan read recorded messages. Rotate the secret regularly. - The
ldapplugin has been checked against Active Directory only
without a certificate and without a successful login. - Unchanged from 0.1.0: lookup throughput across threads levels off at
about two million a second on Linux
(docs/performance.md). - Unchanged from 0.2.0: with
[store] sync = false(the default), a host
crash can leave the store damaged; the daemon then sets it aside at the
next start and the state history is lost. - Unchanged from 0.3.0: a host announces at most 80 services left out of
the peer group; the other members still count it as a candidate for
the rest.