Skip to content

emfor 0.5.0

Choose a tag to compare

@vanillahsu vanillahsu released this 15 Nov 06:41
· 63 commits to main since this release

Stricter validation: a config that 0.4.0 accepted can now be refused.
emfor check-config, and so a start or a reload, now refuses options that
could never work and that 0.4.0 let through, to fail later: an interval
above 24h, a timeout below 10ms, an http path with # or \, an
exec or plugin command with a NUL, a user or password with a NUL where
the protocol cannot carry one (mysql, pgsql, amqp), an amqp password too
long for AMQP's first frame, a diameter list too long for one message, a
ca_file with no usable CA, and more (see Daemon and emfor-check). Run
emfor check-config with the new binary before you upgrade a host.

Peer messages have a second encrypted format, EMP3
(XChaCha20-Poly1305, which has no practical limit on the number of
messages per secret). Each member is sent the format it understands, so a group of
0.4.0 and 0.5.0 members works in any upgrade order without a
configuration change (see Daemon).

The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon; its files are now libemfor.so.0.5 and
libemfor.0.5.dylib.

The main additions are EMP3, [peers] accept_plaintext and
previous_secret_file, a DNS server that limits UDP responses per client
network and keeps answering while a commit is slow, plugins that are
restarted when they stop answering, and clients that no longer end the
application with SIGBUS on a truncated store or hang a forked child on
FreeBSD. The rest are fixes, most of them in emfor-check and the DNS
server.

Daemon

  • Peer messages have a new encrypted format, EMP3: XChaCha20-Poly1305
    with a 24-byte random nonce and a key of its own derived from the
    secret. EMP2's 12-byte random nonce is safe for about 2^32 messages
    per secret, which a large group can send within days; a repeated nonce
    would let an observer read and forge messages. With encrypt = true the
    format is picked per member: every EMP2 message now carries
    "emp3": true in its encrypted payload, and a member is sent EMP3 once
    the latest message accepted from it is EMP3 or such an EMP2, and
    EMP2 otherwise (also after it sends EMP1), so members of this
    release use EMP3 between each other and EMP2 with 0.4.0 members. A
    group needs no configuration change to upgrade or to go back; a host
    going back to 0.4.0 drops EMP3 messages for about its first second (see
    Peer group).
  • A store file shorter than LMDB records it (changed in place, or its
    tail lost in a power loss) is moved to <path>.corrupt at start like
    other damage, with a warning saying truncated. Before, reading it to
    check it ended the daemon with SIGBUS, again at every restart.
    After every commit the daemon extends the file with zero bytes to the
    length LMDB records, which LMDB itself can leave a few pages short, so
    clients never take a healthy store for a truncated one. A store an
    older emfor left short that way is moved aside on the first start
    after the upgrade (only the state history is lost).
  • A plugin that stops answering is restarted. Until now only a plugin
    that exited or closed its stdout was restarted: one that kept running
    but answered nothing (stuck in a loop, deadlocked, stopped) left every
    check of its backends error for good, and the requests for it queued
    up without bound. Now, when the callers of 3 requests in a row give up
    waiting and no answer to any request arrives in between, the daemon
    logs that the plugin stopped answering, kills it and restarts it with
    the usual backoff. A plugin that answers within timeout_ms is never
    affected (see Plugin protocol).
  • A single key or value in the store that is not UTF-8 no longer blocks
    the daemon. Under svc/ it made every commit fail from start-up on,
    so clients saw a stale store for good; under hist/ it kept every
    service's history from loading. Such entries are now deleted, or
    discarded with a warning, on their own.
  • http host_header must be a host with an optional port
    (api.internal, api.internal:8443, [2001:db8::1]). Values with
    spaces, a path or a scheme passed check-config but made every check
    down, or silently checked whatever host the name resolved to instead
    of the backend.
  • State changes already queued when one is written are committed in the
    same transaction, so a burst of changes (a network split) costs one
    commit, not one per backend.
  • The queues from checks to the peer task and to a plugin's stdin are
    bounded. A peer task that falls more than 4096 messages behind drops new
    down and result messages instead of holding them (counted in the new
    emfor_peer_queue_dropped_total, logged once a minute); checks never
    wait for it. A plugin that stops reading its stdin gets at most 1024
    waiting requests; further checks are error at once with
    plugin <name> busy: it is not reading its requests instead of each
    waiting for its timeout while memory grows.
  • tls_cert_file and tls_key_file (the diameter plugin) are parsed
    again only when they change, like ca_file: an unchanged file costs one
    stat per check instead of a read and a parse, and a renewed one still
    takes effect at the next check. Parsed ca_file, certificate and key
    files that no check used for 10 minutes are released, so files removed
    from the config by reloads no longer stay in memory.
  • The DNS server limits UDP responses to each client network (IPv4 /24,
    IPv6 /56) to [dns] rate_limit a second, 100 by default (0 turns it
    off). An SRV answer can be many times the size of its query, so a server
    reachable from other hosts could be used to flood a third party with
    forged-source queries. Past the limit every other response is an empty
    truncated one, which makes a real client retry over TCP, and the rest
    are dropped; emfor_dns_rate_limited_total counts them. Loopback
    clients and TCP are never limited. At start the daemon now warns about
    each [dns] listen address that is not a loopback address.
  • A slow commit no longer freezes DNS answers and /metrics. While the
    state store waited for the disk ([store] sync = true, or the kernel
    writing back), the daemon stopped applying check results, so DNS kept
    answering with the states from before the commit and the results
    queued up in memory. Results are now applied as they arrive; the
    changes made meanwhile are written together by the next commit, which
    starts as soon as the slow one ends.
  • A reload that changes interval or jitter applies to the wait
    already under way. Before, going from 10m to 2s took up to 11
    minutes to apply.
  • The 200ms checks while a backend is unknown stop after rise + fall
    in a row; results that never reach either count no longer check about
    5 times a second for good.
  • interval is at most 24h and timeout at least 10ms. Larger
    intervals could panic the check task (with a huge fall), and a 1ms
    timeout gave plugins 0ms.
  • [store] map_size is rounded up to a multiple of 64KiB. A size LMDB
    refuses (such as "1048577") passed check-config and then stopped
    the daemon from starting.
  • A check task that is restarted after a bug now records the backend's
    move to unknown as a transition (reason check task restarted);
    before, the stored state changed silently.
  • The DNS server's view of a service is updated per backend instead of
    copying the whole service on every check.
  • DNS over TCP serves at most 128 connections at once and closes a
    connection with no query for 5 seconds, so clients can no longer use
    up the daemon's file descriptors.
  • If a DNS listener stops while the daemon runs, the daemon logs it and
    exits with status 70, so the service manager restarts it; before, DNS
    went silent.
  • DNS answers BADVERS to EDNS versions above 0 and REFUSED to classes
    other than IN; before, a CH query got IN records.
  • DNS compares names label by label: a label containing a . byte no
    longer puts a name outside the zone inside it.
  • Over UDP, a reply whose single record or SOA does not fit (names near
    255 bytes) is truncated by emfor itself and keeps its EDNS record.
  • Plugins run in a process group of their own. When a plugin exits, is
    stopped, or is killed because it stopped answering, the processes it
    started are killed too; before, they could outlive it.
  • A plugin's stderr lines and invalid stdout lines are rate limited: up
    to 100 at once, then 10 a second each. The log notes how many lines
    were dropped, so a plugin writing in a loop can no longer fill the
    log.
  • A ca_file is parsed again only when it changes (inode, size or
    times), not on every check. A replaced file still takes effect at the
    next check.
  • http clients for checks (one per backend with a domain host_header)
    are built off the async worker threads, because building one reads the
    system CA store. The peer secret_file is read off them too.
  • Documented: a connection that fails because this host has no route to
    the backend, or its own firewall blocks it, is down, not error.
    That has always been the behaviour.
  • When a check task is restarted after a bug while its backend is
    unknown, the backend's latency is now cleared and its reason set to
    check task restarted, in the store and in DNS.
  • A late result from the check task of a backend removed by a reload can
    no longer mark that backend healthy after a later reload adds it back.
    Each check task now has its own generation, and a backend keeps its
    state exactly while its task keeps running.
  • check-config now refuses an http path with #, \ or a control
    character. The request URL cut or rewrote these, so another path was
    checked.
  • With [dns], a service name in the zone must leave room for the SRV
    target name of each backend (<ip label>.<service>, at most 253
    characters). Before, such a service passed check-config and every
    SRV query for it got SERVFAIL.
  • An exec or plugin command with a NUL in any element is refused; it
    could never be started.
  • A tcp ca_file with no certificate usable as a CA is refused at
    validation instead of failing every check with ca_file: no usable certificate.
  • On start, state history entries with a state this version does not
    know (left by a newer one) are dropped instead of discarding that
    service's whole history.

Peers

  • New [peers] accept_plaintext (default true, applied on reload).
    false drops every plaintext EMP1 message without checking it,
    counted under the new reason plaintext of
    emfor_peer_messages_rejected_total, so a group whose members all
    encrypt no longer has to accept signed plaintext. It needs
    encrypt = true; check-config refuses it otherwise.
  • A kept result counts once. A member kept another member's latest result
    for 3 of the sender's intervals and counted it at every one of its own
    check times in that span, so with interval = "1s" here and "30s" at
    the assigned checkers one down from each of two checkers moved
    fall up to about 90 times and marked a backend down that the checkers
    themselves still saw healthy. Now each result counts at one check time;
    a check time decided by votes alone needs a new vote, and an own check
    that only repeats counted down votes while it passes changes nothing
    but is still counted in emfor_backend_checks_total. With equal
    intervals most check times get new results, so little changes (see
    spec peer-group).
  • New optional [peers] previous_secret_file: the secret secret_file
    held before its last change. A daemon restarted within 120 seconds of
    that change keeps signing with and accepting the old secret as if it
    had not restarted. Before, members whose file had not changed yet
    rejected it as auth until they changed. See
    [peers].
  • A second change of secret_file within 60 seconds of the first no
    longer signs for a minute with the intermediate secret, which members
    whose file had not changed did not know. The daemon keeps signing with
    the earlier one, and accepts every secret replaced in the last 120
    seconds.
  • Peer messages that arrive out of order are accepted once, instead of
    being dropped as replay. An EMP1 message whose tag does not match
    now counts as auth even when its payload is not JSON: the tag is
    checked before the payload is parsed. A message of a kind this version
    does not know is accepted (for liveness) even when it reuses a known
    field name with another type.
  • A flood of datagrams from many source addresses no longer suppresses
    the warnings about real members' dropped messages.
  • A result or down sent after a reload that changed check,
    check_options or timeout during the check now carries the check
    definition digest the check ran with, not the new one.

emfor-check

  • Reasons longer than 512 characters are cut to 512 followed by …
    before they are sent (the daemon keeps no more). A backend's long error
    text, such as a 70 KB redis -ERR, made the answer line longer than the
    daemon's 65536-byte limit, so the daemon restarted the plugin, again at
    every check of that backend.
  • mongodb: a server asking for more than 1000000 SCRAM iterations is
    down with auth: server asks for <n> SCRAM iterations, more than 1000000; one asking for 4294967295 held a worker thread for many
    minutes. The key derivation now runs off the async workers, so the
    check's timeout ends a slow login.
  • sip over TCP or TLS: a response with a Content-Length near the
    largest 64-bit number made the plugin loop forever on the same
    provisional response, holding a worker thread at full CPU, with no
    timeout to end it. Such a response is now down with
    query: response longer than 65536 bytes.
  • A check that panics is answered error with check panicked: <message> (emfor-plugin crate). Until now it sent no answer, so the
    daemon waited out the timeout.
  • mysql: a NULL or non-integer answer to SELECT 1 or
    @@global.read_only is down with query: <query> answered <value>, want an integer. Until now the check panicked and the backend ended
    up unknown.
  • mysql: a check cut off by its timeout closes its connection within 1
    s. mysql_async's cleanup task waited with no limit for a server that
    stalled mid-result, leaking a task and a socket at every check.
  • mysql: a CA certificate that cannot be used as a root is skipped, as
    in the other plugins, instead of failing every TLS handshake.
  • redis: a reply over 2 MiB is down with query: invalid reply: reply too long, and a partly received reply is parsed again only after a
    line feed arrives.
  • mongodb: a hello reply nesting documents more than 100 deep is
    down with query: malformed reply. A reply of about 8000 levels
    overflowed the stack and aborted the plugin.
  • mongodb: validation refuses a password_file whose password is not
    printable ASCII. Until then such a service was error at every check.
  • quic: HTTP/3 response headers are limited to 65536 bytes (h3 allowed
    any size).
  • A ca_file with no certificate usable as a CA is refused at
    validation (no usable certificate), as the checks already did.
  • diameter: validation builds the TLS client configuration with
    tls_cert_file and tls_key_file, so a key that does not belong to
    the certificate is reported on tls_key_file instead of failing every
    check.
  • mysql and pgsql user and pgsql database and password must not
    contain NUL. The amqp user and password must not contain NUL (PLAIN
    separates them with NUL). A pgsql password file that gained a NUL
    after validation is error at the check instead of down. A mysql
    password may still contain NUL; the plugin documentation says when
    one fails to log in (caching_sha2_password without TLS).
  • amqp: a password too long for the login to fit in AMQP's 4096-byte
    first frame is refused at validation, and is error at a check when
    the file grew since. Before, every check sent a frame the
    broker may refuse, and the backend could be down at every check.
  • diameter: options whose Capabilities-Exchange-Request would not fit in
    Diameter's 24-bit message length are refused at validation. Before,
    the length was cut silently and every check sent a malformed request.
  • sip uri and from must not contain < or >, which ended the
    To/From value early.
  • sip, diameter, mongodb, amqp and ldap read messages through one shared
    framed reader (checked lengths, one size cap, EOF handling); reasons
    are unchanged. The ldap BER length check can no longer overflow on
    32-bit platforms.

emfor status

  • A truncated store makes emfor status print cannot open store <path>: truncated: ... and exit 2 instead of ending with SIGBUS.
  • emfor status | head -1 keeps the normal exit status instead of
    exiting 2 silently; other output write errors are printed.
  • --watch opens the store again when the file is replaced, instead of
    showing the old file as stale forever.
  • A record from a newer daemon is shown instead of failing: an unknown
    state as unrecognized (red), an unknown on_all_down as
    on_all_down=unknown.

Clients

  • A store file already shorter than LMDB records it when the client opens
    it makes lookups fail with UNAVAILABLE (the message says truncated)
    instead of ending the application with SIGBUS. A store changed in
    place while it is open can still do that: replace it with mv, never
    cp over it or truncate it.
  • A running application keeps reading the store after the daemon is
    restarted with a larger [store] map_size and the store grows past
    the old size. Before, every lookup failed with UNAVAILABLE
    (MDB_MAP_RESIZED) until the application restarted.
  • A child forked while other threads of the parent were looking up no
    longer hangs in its first lookup. On FreeBSD this did not hold at
    first: a child waited for ever, within the first few dozen forks of
    the fork test (measured on FreeBSD 15.1-RELEASE, arm64), on a lock of
    the threading library that a thread of the parent held inside LMDB at
    the fork; 0.4.0 has the same hang there (its tag, built and run with
    the same tests: 10 of 10 runs of the Rust test and 5 of 5 of the C
    test failed). On FreeBSD the client now makes fork wait until no lookup
    is inside the store (pthread_atfork handlers, registered on FreeBSD
    only), and libemfor can no longer be unloaded there, since the
    handlers stay registered. Lookup throughput did not change (measured
    with 1, 4 and 8 threads). The lock file and the daemon are the same.
    Not guarded: a fork from a signal handler that interrupted a lookup
    of the same thread (it returns, but the child can still hang), and other
    users of process-shared mutexes in the same process (see
    Threads, processes and fork).
  • A store file that exists but cannot be opened is tried again at most
    once a second; lookups in between fail with the same UNAVAILABLE
    error. A missing store is still tried on every lookup.
  • emfor.h no longer defines the internal macros REPLACE_CHECK_MS and
    MAX_STICKY_WEIGHT.
  • Python: the library file named for this release (libemfor.so.0.5,
    libemfor.0.5.dylib) is tried before what ctypes.util.find_library
    finds, which may be another release's.
  • C API: emfor_lookup called from a thread-local destructor during
    thread exit returns a status instead of aborting the process.
  • A backend state or on_all_down value written by a newer daemon no
    longer makes the lookup fail with INCOMPATIBLE: an unknown state
    counts as unhealthy, an unknown on_all_down as open, so a later
    release can add such values without raising the store format version.

Deployment

  • The example systemd unit sets LimitNOFILE=65536.

Development

  • tests/plugins/mysql.sh [version] runs the mysql plugin's live tests
    against MySQL from the mysql Docker images (8.4 unless given), with
    the layout tests/plugins/run.sh uses for MariaDB. They pass on MySQL
    8.0 and 8.4, so the 0.4.0 known limit that the plugin was checked only
    against MariaDB no longer holds for those versions.
  • emfor-conformance writes a new store just before each run of the
    lookup program, so slow runs earlier in a case cannot make a fresh
    store stale. A failure in a run longer than the staleness limit says
    so. An expected error now needs a non-empty message.
  • Fuzz builds keep overflow checks and debug assertions on, and run each
    input on a 2 MiB stack like a tokio worker's; an integer overflow or a
    too-deep recursion is now a crash, as it would be in the daemon.
  • New dns fuzz target for the DNS server's request handling and
    response size trimming.
  • peers.rs is split into a peers/ module (wire, crypto, stats,
    group, targets, task); the peer group's live members have one owner,
    and service digest lookups go through one function.

Known limits

  • FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests,
    the rc.d script, the port with make stage check-plist install, and
    the fork tests. Not tested: poudriere, 13.x, 14.x, amd64 (the optional
    CI job covers 14.3 on amd64, and had not been run for this release).
  • FreeBSD's threading library can hang a child forked while other threads
    use a process-shared mutex (LMDB's reader mutex is one), and can
    deadlock such a parent. libemfor works around it for its own lookups
    (see Clients; 0.4.0 has the problem). Not covered: other users of
    process-shared mutexes in the same process, a fork from a signal
    handler that interrupted a lookup of the same thread, and a
    libemfor.a linked statically into a module that is unloaded. A patch
    for libthr, tested on 15.1-RELEASE only, is in
    docs/freebsd/.
  • Peer messages have no forward secrecy: whoever later gets
    secret_file can read recorded messages. Rotate the secret regularly.
  • The ldap plugin has been checked against Active Directory only
    without a certificate and without a successful login.
  • Unchanged from 0.1.0: lookup throughput across threads levels off at
    about two million a second on Linux
    (docs/performance.md).
  • Unchanged from 0.2.0: with [store] sync = false (the default), a host
    crash can leave the store damaged; the daemon then sets it aside at the
    next start and the state history is lost.
  • Unchanged from 0.3.0: a host announces at most 80 services left out of
    the peer group; the other members still count it as a candidate for
    the rest.