Skip to content

emfor 0.4.0

Choose a tag to compare

@vanillahsu vanillahsu released this 14 Nov 13:07
· 92 commits to main since this release

Breaking: the store format is version 2. The daemon writes
meta/schema_version 2, which adds the backend field backup. Clients
and emfor status from this release read versions 1 and 2; older ones
fail every lookup on a version 2 store with INCOMPATIBLE. Upgrade the
clients on a host before the daemon (see
Store format version).

Peer messages are encrypted by default. Members running 0.3.0 cannot
read them and drop them as malformed. To upgrade a peer group, set
encrypt = false in [peers] on each host as it is upgraded; once every
member runs this release, remove the line and reload each daemon.

Breaking, on FreeBSD only: examples/rc.d/emfor is gone, and the
example port keeps the store in /var/db/emfor (see Deployment).

The main additions are backup backends, slow start, a preference for
backends whose last check passed, sticky lookups that honour weight,
encrypted peer messages, and the smtp, amqp, mongodb, grpc and
quic plugins of emfor-check. The plugin protocol (1) is unchanged.
As with every release, install libemfor from the same release as the
daemon.

Daemon

  • Peer group messages are encrypted (EMP2: ChaCha20-Poly1305 with a key
    derived from secret_file by HKDF-SHA256 and a random nonce per
    message), so backend names, addresses and reasons no longer travel in
    the clear. New [peers] encrypt (default true, applied on reload):
    false sends the signed plaintext EMP1 of 0.3.0. Both formats are
    accepted either way; with encrypt = true the daemon warns once a
    minute per member that still sends EMP1. A message that does not
    decrypt counts as auth in emfor_peer_messages_rejected_total.
    Secret rotation works as before. There is no forward secrecy: whoever
    later gets the secret can read recorded messages, so rotate it
    regularly. See [peers].

  • Backup backends: a backend table takes backup = true. Backups are
    used only while no other backend passes its checks: clients and the DNS
    server take the primaries whose last check passed, else the backups
    whose last check passed, else the healthy primaries, else the healthy
    backups, and balance works within that set; fastest decides which
    backends are fast among the primaries and among the backups
    separately. When no backend is
    healthy, on_all_down decides as before and backups count like the
    others. SRV answers list only the chosen set. At least one backend of a
    service must not be a backup, and backup is applied on reload. See
    Backup backends.

  • Clients and the DNS server leave out a healthy backend whose last
    check failed (its reason is set) while another healthy backend passed
    its last check, so a backend stops being chosen at its first failed
    check instead of after fall of them. When every healthy backend is
    failing, all of them are used as before. The store format is
    unchanged.

  • slow_start (per service, or in [defaults]; "0s", off, by
    default, else 1s to 1h): a backend that turns from unhealthy to
    healthy gets a growing share of traffic over that time, in 10 steps,
    instead of its full share at once. It applies to client lookups and
    DNS A/AAAA answers for every method but failover and sticky with a
    key, and to SRV weights. The store gains slow_start_ms on services
    and recovered_at on backends; clients that do not know them (as
    emfor-client 0.3.0 reading a version 1 store) give a recovered backend
    its full share at once. See
    Slow start.

  • With on_all_down = "open" and no healthy backend, clients and the DNS
    server now use the unknown backends (not decided yet, or whose checks
    only give errors) before the unhealthy ones, which failed fall
    checks in a row: first the unknown ones whose last check passed, then
    every unknown one, then every backend as before. closed and a stale
    store are unchanged, and so is the store format; older clients still
    choose among all backends.

  • The http check takes expect_body: text the response body must
    contain, for health endpoints that answer 200 either way. The body is
    read only when the status matches, up to 64 KiB.

  • On macOS, exec check commands and plugins no longer inherit the
    daemon's sockets and pipes. A plugin could keep a check connection
    open for as long as it ran, so the backend never saw it closed.

  • Peer groups notice a failure that only a member which is not one of the
    backend's assigned checkers sees sooner (the known limit of 0.3.0).
    Such a member now checks the backend itself at least every third
    interval instead of every fifth, and when its own check fails while
    the assigned checkers see the backend up it starts a fast recheck run
    (every 200 ms, up to fall checks), as after a peer's down report.
    With the defaults the backend is down there after at most about 6.4
    seconds instead of about 14 (estimated from the schedule; 6 without a
    group). In exchange a backend gets about 3 + (N − 3) / 3 checks per
    interval from N members instead of 3 + (N − 3) / 5. Measured with
    interval = "2s" and fall = 3: 0.8 to 5.8 seconds, 3.3 seconds
    median, over 12 failures started at different points of the schedule.

Clients

  • Behaviour change: balance = "sticky" with an affinity key now
    honours weight: a backend of weight w gets w shares of the keys. Its
    value is the highest of MD5(key + id) and MD5(key + id + "#" + i)
    for i from 1 to w − 1, integers only, so every host and language still
    picks the same backend for a key. With every weight 1 nothing changes.
    A sticky service that already sets other weights moves some keys when
    its clients are upgraded, and old and new clients pick differently for
    those keys meanwhile, so upgrade the clients of such a service
    together. Raising a weight only moves keys to that backend; slow start
    does not apply, and a weight above 100 in the store counts as 100. A lookup costs one MD5 per unit of weight: 0.8 µs for 10
    backends of weight 1, 72 µs for 10 of weight 100 (Apple M4 Max). The
    Rust API adds testing::rendezvous_pick_weighted. See
    sticky.

emfor-check

  • role = "replica" for mysql and pgsql, and for all three database
    plugins it now means a replica that is replicating: a redis replica
    whose link to its master is connected, a MySQL or MariaDB replica
    whose IO and SQL threads both run, a PostgreSQL standby with a WAL
    receiver. Otherwise the backend is down with a replication: reason.
    A redis replica that was up with role = "replica" while its link
    was down is now down.
  • max_lag: with role = "replica", the seconds a replica may be behind
    before it is down. See Replicas for
    what each plugin measures and the rights the checks need.
  • emfor-check smtp checks SMTP servers: the greeting must be 220 and
    EHLO must answer 250, on a new connection, optionally upgraded with
    STARTTLS (starttls = true) or TLS from the start (tls = true,
    port 465); then it sends QUIT. It never logs in or sends mail. It is
    down with greeting: , ehlo: or tls: STARTTLS not offered. See
    docs/deploy/plugins.md.
  • emfor-check amqp checks AMQP 0-9-1 brokers such as RabbitMQ: a
    PLAIN login and Connection.Open of vhost on a new connection, over
    plaintext or TLS, then a normal Connection.Close. It is down with
    auth: 403 ACCESS_REFUSED - … for a rejected login and
    vhost: 530 NOT_ALLOWED - … for a vhost the user cannot open. See
    docs/deploy/plugins.md.
  • emfor-check mongodb checks MongoDB 4.4 and later: hello on a new
    connection, plaintext or TLS, an optional SCRAM-SHA-256 login (user,
    password_file, auth_source), and role = "primary" (a replica set
    primary, a standalone server or mongos) or role = "secondary", so
    balance = "failover" follows the replica set's primary. See
    docs/deploy/plugins.md.
  • emfor-check grpc checks gRPC services with the standard health
    service: grpc.health.v1.Health/Check over HTTP/2, plaintext or TLS
    with ALPN h2, is up when it answers SERVING. It is down with
    status: NOT_SERVING, grpc: 12 UNIMPLEMENTED (no health service) or
    grpc: 5 NOT_FOUND: … (an unknown service). See
    docs/deploy/plugins.md.
  • emfor-check quic checks QUIC servers over UDP: a new QUIC version 1
    handshake on every check, without 0-RTT, offering the ALPN values of
    alpn (default ["h3"]) and verifying the certificate (tls_verify,
    server_name, ca_file; always TLS 1.3). It is up when the handshake
    completes. With http3 = true it also sends an HTTP/3 GET for path
    and judges expect_status and expect_body like the built-in http
    check. It is down with quic: version negotiation failed, tls: …
    (for example tls: no application protocol (ALPN)),
    quic: closed by peer: <code>[: <reason>], quic: stateless reset,
    query: HTTP/3: …, unexpected status <n> or expect_body: got "…".
    UDP has no "connection refused", so a closed port or a firewall that
    drops UDP is a timeout. The plugin uses quinn 0.11, h3 0.0.8 and
    h3-quinn 0.0.10, which make the emfor-check release binary about
    1.3 MiB (13%) larger, 11.2 MiB on macOS arm64; the daemon does not
    link them. See docs/deploy/plugins.md.
  • ldap reasons no longer end with a NUL byte when the server is Active
    Directory, which ends its diagnostic message with one: a wrong password
    is now auth: 49 invalidCredentials: 80090308: LdapErr: …, data 52e, ….
  • A server that closes or resets the connection during the TLS handshake
    is down with tls: connection closed during the TLS handshake (does the server have a certificate?) instead of tls: tls handshake eof or
    tls: Connection reset by peer (os error 54). Active Directory does
    this on port 636 when it has no certificate. The same reason applies to
    the daemon's tcp check with tls = true.

Deployment

  • The FreeBSD rc.d script starts daemon(8) with -f. Without it,
    daemon(8) and emfor kept the terminal or SSH session that ran
    service emfor start, so that session never closed. The rc.d script,
    the port and the tests have now run on FreeBSD 15.1-RELEASE (arm64).
  • The example FreeBSD port puts the store in /var/db/emfor, where
    FreeBSD keeps daemon state, instead of /var/lib/emfor: it creates that
    directory, and its sample config sets [store] path = "/var/db/emfor/state.mdb". The daemon's and the clients' default stay
    /var/lib/emfor/state.mdb, so applications on the host set
    EMFOR_STORE. After updating an installed port, move the store or keep
    the old [store] path; the history is all it holds.
  • examples/rc.d/emfor is gone. The port's
    examples/freebsd/files/emfor.in is the only rc.d script; to use it
    without the port, fill in %%PREFIX%% (chapter 2 of the manual shows
    the command). Its emfor_user defaults to emfor, not root, so
    create that user or set emfor_user.

Documentation

  • The operations chapter of the manual says what a host crash loses with
    [store] sync = false (the state history only) and how to keep the
    store on a tmpfs (Keeping the store in
    memory
    ).
  • docs/deploy/plugins.md says which servers
    each plugin was checked against, and what was checked by hand against
    Active Directory.

Development

  • cargo-fuzz targets for plugin answers, peer messages, metrics, plugin
    lines and the config (fuzz/); RELEASING.md runs each
    for 10 minutes before a release.

Known limits

  • FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM), the port with
    make stage check-plist install; not with poudriere, on 14.x or on
    amd64.
  • Peer messages have no forward secrecy: whoever later gets
    secret_file can read recorded messages. Rotate the secret regularly.
  • The ldap plugin has been checked against Active Directory only
    without a certificate and without a successful login; the mysql
    plugin only against MariaDB.
  • Unchanged from 0.1.0: lookup throughput across threads levels off at
    about two million a second on Linux
    (docs/performance.md).
  • Unchanged from 0.2.0: with [store] sync = false (the default), a host
    crash can leave the store damaged; the daemon then sets it aside at the
    next start and the state history is lost.
  • Unchanged from 0.3.0: a host announces at most 80 services left out of
    the peer group; the other members still count it as a candidate for
    the rest.