Skip to content

Releases

emfor 0.6.0

Choose a tag to compare

@vanillahsu vanillahsu released this 16 Nov 23:30

The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged: clients and plugins of 0.5.0 keep working with this daemon. Upgrade
libemfor and the clients before you use drain = true, so that an older client
does not choose a drained backend (see below). A config that 0.5.0 accepted is
still accepted.

Manual chapter 7 is split in two (both languages). 07-dns.md keeps the DNS
server, its answers, SRV records, forwarding and caching; the recipes for
nginx, HAProxy, Caddy, Prometheus and pf and the clients that ask for SRV
records themselves moved to 07-dns-recipes.md (chapter 7b), so links to
07-dns.md#using-it-with-nginx and the like now point there. The chapter
numbers do not change.

examples/deploy/emfor-drain.sh is safer. wait-ready no longer returns
at the first healthy: a backend that has just stopped keeps that state for
fall x interval (6 s by default), so a drained backend that was stopped
could be put back dead. It now needs a check newer than the command and
healthy for SETTLE seconds in a row (third argument, default 10). on and
off refuse a backend line they cannot change (it must read
{ addr = "..." }, with a space before the }) without touching the file, and
put the old file back when check-config refuses the new one. New:
EMFOR_CONFIG (passed as --config) and EMFOR_SERVICE (the service, when
two have the same address). tests/deploy has a case with the default
timings, and one for each refusal. Manual chapter 8 (both languages) lists
what to set up first, and the corrections of the review of chapters 7, 8 and
10: emfor status needs write access to the lock file, the drained (checks: …) marker comes last, the upgrade order (clients first, check-config before
the restart), the metrics tables, and in chapters 7 and 10 the SERVFAIL cases,
the 1232-byte UDP cap, include_dir before the first table, rise and fall
of 1 instead of Consul's 0, and weights from 1 to 100.

Using emfor with HAProxy. Manual chapter 7 (both languages) has a section
for HAProxy: a server-template on emfor's SRV answers (resolvers with
nameserver 127.0.0.1:5300) fills its servers, and health and drain reach
HAProxy within a few seconds without a reload. It also says what HAProxy does
not take from the answer: SRV priorities and weights are ignored (so
failover, weighted, locality and fastest do not behave as for nginx),
and that after SERVFAIL HAProxy drops its servers after its hold other
time (30 s) and answers 503, where nginx keeps its last list. The slots of the
template limit the number of backends. tests/haproxy/run.sh [image] checks
all of it, the two ignored features included, and the Integrations workflow
runs it on haproxy:lts and haproxy:2.8. Checked with HAProxy 3.4.7 and 2.8.31.
The section also covers mode tcp (no difference), TLS to the backends (set
sni: without it 3.4.7 refuses the certificate and 2.8.31 accepts any
certificate the CA signed; verifyhost is not reliable) and several name
servers (HAProxy sends each query to all of them and uses the first reply;
with different answers the result varied from run to run, and requests got 503
after a slot came up with port 0, which on HAProxy 3.4.4 (FreeBSD) happened
with one name server as well; use one local nameserver);
tests/haproxy/more.sh checks the agreeing case, run by the same run.sh.

Using emfor with nginx. Manual chapter 7 (both languages) has a section
for nginx: from 1.27.3 the open-source nginx can take its upstream from emfor's
SRV answers (server api.svc.emfor service=http resolve; with resolver 127.0.0.1:5300), so health, drain and weights reach it within a few
seconds without a reload; the section also says what nginx does when emfor
answers SERVFAIL (it keeps its last list) and gives the variable
proxy_pass recipe for older nginx. tests/nginx/run.sh [image] checks it
on a given image, and the Integrations workflow runs it on nginx:stable and
nginx:mainline. That workflow runs every week and when emfor's DNS or
candidate code or these tests change, not on every push.
Checked with nginx 1.30.6, 1.31.8 and (older recipe only) 1.26.3. Not tried:
HTTPS upstreams, nginx Plus, several workers.
The section now also covers the stream module (TCP): the same upstream
works in a stream block, a connection already open to a drained backend is
not cut (it is cut when the backend is stopped), and a new connection to a
stopped backend goes to the next server. And it says that after a lookup has
failed (emfor answered SERVFAIL), nginx asks again only after
resolver_timeout, 30 seconds by default (about 26 seconds to follow emfor's new
answers), so the recipe now sets resolver_timeout 1s (about a second), for
both http and stream, with nginx 1.30.6 and 1.31.8. tests/nginx/run.sh
runs tests/nginx/stream.sh after inside.sh. Not tried for stream: UDP,
ssl_preread, proxy_protocol, nginx Plus. On FreeBSD 15.1 (nginx 1.30.6
package) both scripts passed by hand (about 25 seconds with the default
resolver_timeout, about 1 with resolver_timeout 1s); the package builds stream as a dynamic
module, so nginx.conf needs load_module /usr/local/libexec/nginx/ngx_stream_module.so;.

Rolling deploy with drain (manual chapter 8, both languages).
examples/deploy/emfor-drain.sh drains a backend in a conf.d file (one
backend per line), checks the config, reloads, and waits until emfor shows it
drained (on), waits until the connections to a port are quiet (wait-idle,
Linux, needs ss), waits until the checks say healthy (wait-ready), and
undrains (off). The chapter says what the waiting is for and what was seen:
a request in flight finishes after a drain and is cut after a plain stop;
nginx 1.30 keeps idle connections to its upstream, so "no connection" never
comes. tests/deploy/run.sh [image] checks it on nginx 1.30.6 and 1.31.8
(also in the Integrations workflow). On FreeBSD 15.1 it passed by hand with
wait-idle replaced by a fixed wait (ss is Linux only; the script says so
there). Not tried: HAProxy, HTTPS, clients with their own connection pool,
several emfor hosts.

Clients that ask for SRV records themselves (manual chapter 7, both
languages).
MIT Kerberos (dns_lookup_kdc, the realm is the service name in
capitals), OpenLDAP (ldapsearch -H 'ldap:///dc=ldap%2Cdc=svc%2Cdc=emfor', the
commas must be %2C) and PyMongo (mongodb+srv://, TLS is on by default) find
their servers through emfor's SRV answers and leave a server that fails its checks
or is drained; a _udp name gets NXDOMAIN; with balance = "failover" Kerberos
tries the primary first. tests/srv/run.sh checks them with the Debian
bookworm packages, against plain listeners. Not tried: a real KDC, LDAP or
MongoDB server, LDAPS, sssd, Samba, Active Directory names, other drivers.

Nagios, Icinga and Zabbix (manual chapter 8, both languages).
examples/nagios/check_emfor.sh gives the Nagios status of what emfor status --json reports (CRITICAL for a stale or stopped daemon or a service with fewer
than -c healthy backends, WARNING below -w, UNKNOWN when emfor or the store
cannot be read; drained backends are left out of the counts; perfdata per
service). examples/nagios/emfor-exec.sh runs an existing Nagios plugin as an
exec check, adding -H and -p from EMFOR_IP and EMFOR_PORT; for emfor a
plugin's WARNING is down (--warning up changes that) and UNKNOWN is an error.
tests/nagios/run.sh checks both with a stand-in plugin. Not tried: the real
Monitoring Plugins, a Nagios or Icinga core, NRPE, Zabbix, FreeBSD.

Using it with pf (FreeBSD; manual chapter 7, both languages).
examples/pf/emfor-pf-table.sh asks emfor's DNS server for the SRV records of a
service every second and makes a pf table equal to the targets with the lowest
priority, so that an rdr ... -> <api> round-robin rule spreads connections
over the backends emfor offers, with no proxy. Health and drain reach the
table, a connection that is already open is not cut (pf keeps its state),
priorities give the table its failover, SRV weights are not used, an empty
answer empties the table (pf then drops new connections, so the client waits for
its own timeout; EMPTY puts an address where nothing listens there instead),
and when emfor does not answer the table is left alone. tests/pf/run.sh
checks it on FreeBSD 15.1 (by hand, as root; not in the CI). Not tried: IPv6,
source-hash, daemon(8), many clients.

Using it with Prometheus (manual chapter 7, both languages).
dns_sd_configs of type SRV on emfor's answers gives Prometheus the healthy
backends of a service as scrape targets. The section says what that means: a
backend that fails its checks or is drained is no longer a target, so its up
series ends and up == 0 never fires (alert on emfor's metrics, or keep a
static list for the scrape); a failover backup is scraped too; a
backup = true backend only when no primary is healthy; and the host
resolver must forward the whole zone, since dns_sd_configs cannot be given a
name server or port. tests/prometheus/run.sh checks it on Prometheus 3.16.0
(also in the Integrations workflow); on FreeBSD 15.1 the same script passed by
hand with Prometheus 2.55.1. Not tried: dnsmasq or unbound in front of emfor,
other versions, HTTPS targets, many targets.

Using it with Caddy (manual chapter 7, both languages). reverse_proxy
with dynamic srv takes its upstreams from emfor's SRV answers, so health and
drain reach Caddy without a reload. The section gives the Caddyfile and says
what each part is for: resolvers in both dynamic srv and transport http
(the second dials the names of the targets), grace_period (without it every
request fails when emfor answers SERVFAIL), lb_try_duration (without it a
backend that has just stopped loses requests). Caddy does not follow the SRV
weights, ...

Read more

emfor 0.5.0

Choose a tag to compare

@vanillahsu vanillahsu released this 15 Nov 06:41

Stricter validation: a config that 0.4.0 accepted can now be refused.
emfor check-config, and so a start or a reload, now refuses options that
could never work and that 0.4.0 let through, to fail later: an interval
above 24h, a timeout below 10ms, an http path with # or \, an
exec or plugin command with a NUL, a user or password with a NUL where
the protocol cannot carry one (mysql, pgsql, amqp), an amqp password too
long for AMQP's first frame, a diameter list too long for one message, a
ca_file with no usable CA, and more (see Daemon and emfor-check). Run
emfor check-config with the new binary before you upgrade a host.

Peer messages have a second encrypted format, EMP3
(XChaCha20-Poly1305, which has no practical limit on the number of
messages per secret). Each member is sent the format it understands, so a group of
0.4.0 and 0.5.0 members works in any upgrade order without a
configuration change (see Daemon).

The store format (meta/schema_version 2) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon; its files are now libemfor.so.0.5 and
libemfor.0.5.dylib.

The main additions are EMP3, [peers] accept_plaintext and
previous_secret_file, a DNS server that limits UDP responses per client
network and keeps answering while a commit is slow, plugins that are
restarted when they stop answering, and clients that no longer end the
application with SIGBUS on a truncated store or hang a forked child on
FreeBSD. The rest are fixes, most of them in emfor-check and the DNS
server.

Daemon

  • Peer messages have a new encrypted format, EMP3: XChaCha20-Poly1305
    with a 24-byte random nonce and a key of its own derived from the
    secret. EMP2's 12-byte random nonce is safe for about 2^32 messages
    per secret, which a large group can send within days; a repeated nonce
    would let an observer read and forge messages. With encrypt = true the
    format is picked per member: every EMP2 message now carries
    "emp3": true in its encrypted payload, and a member is sent EMP3 once
    the latest message accepted from it is EMP3 or such an EMP2, and
    EMP2 otherwise (also after it sends EMP1), so members of this
    release use EMP3 between each other and EMP2 with 0.4.0 members. A
    group needs no configuration change to upgrade or to go back; a host
    going back to 0.4.0 drops EMP3 messages for about its first second (see
    Peer group).
  • A store file shorter than LMDB records it (changed in place, or its
    tail lost in a power loss) is moved to <path>.corrupt at start like
    other damage, with a warning saying truncated. Before, reading it to
    check it ended the daemon with SIGBUS, again at every restart.
    After every commit the daemon extends the file with zero bytes to the
    length LMDB records, which LMDB itself can leave a few pages short, so
    clients never take a healthy store for a truncated one. A store an
    older emfor left short that way is moved aside on the first start
    after the upgrade (only the state history is lost).
  • A plugin that stops answering is restarted. Until now only a plugin
    that exited or closed its stdout was restarted: one that kept running
    but answered nothing (stuck in a loop, deadlocked, stopped) left every
    check of its backends error for good, and the requests for it queued
    up without bound. Now, when the callers of 3 requests in a row give up
    waiting and no answer to any request arrives in between, the daemon
    logs that the plugin stopped answering, kills it and restarts it with
    the usual backoff. A plugin that answers within timeout_ms is never
    affected (see Plugin protocol).
  • A single key or value in the store that is not UTF-8 no longer blocks
    the daemon. Under svc/ it made every commit fail from start-up on,
    so clients saw a stale store for good; under hist/ it kept every
    service's history from loading. Such entries are now deleted, or
    discarded with a warning, on their own.
  • http host_header must be a host with an optional port
    (api.internal, api.internal:8443, [2001:db8::1]). Values with
    spaces, a path or a scheme passed check-config but made every check
    down, or silently checked whatever host the name resolved to instead
    of the backend.
  • State changes already queued when one is written are committed in the
    same transaction, so a burst of changes (a network split) costs one
    commit, not one per backend.
  • The queues from checks to the peer task and to a plugin's stdin are
    bounded. A peer task that falls more than 4096 messages behind drops new
    down and result messages instead of holding them (counted in the new
    emfor_peer_queue_dropped_total, logged once a minute); checks never
    wait for it. A plugin that stops reading its stdin gets at most 1024
    waiting requests; further checks are error at once with
    plugin <name> busy: it is not reading its requests instead of each
    waiting for its timeout while memory grows.
  • tls_cert_file and tls_key_file (the diameter plugin) are parsed
    again only when they change, like ca_file: an unchanged file costs one
    stat per check instead of a read and a parse, and a renewed one still
    takes effect at the next check. Parsed ca_file, certificate and key
    files that no check used for 10 minutes are released, so files removed
    from the config by reloads no longer stay in memory.
  • The DNS server limits UDP responses to each client network (IPv4 /24,
    IPv6 /56) to [dns] rate_limit a second, 100 by default (0 turns it
    off). An SRV answer can be many times the size of its query, so a server
    reachable from other hosts could be used to flood a third party with
    forged-source queries. Past the limit every other response is an empty
    truncated one, which makes a real client retry over TCP, and the rest
    are dropped; emfor_dns_rate_limited_total counts them. Loopback
    clients and TCP are never limited. At start the daemon now warns about
    each [dns] listen address that is not a loopback address.
  • A slow commit no longer freezes DNS answers and /metrics. While the
    state store waited for the disk ([store] sync = true, or the kernel
    writing back), the daemon stopped applying check results, so DNS kept
    answering with the states from before the commit and the results
    queued up in memory. Results are now applied as they arrive; the
    changes made meanwhile are written together by the next commit, which
    starts as soon as the slow one ends.
  • A reload that changes interval or jitter applies to the wait
    already under way. Before, going from 10m to 2s took up to 11
    minutes to apply.
  • The 200ms checks while a backend is unknown stop after rise + fall
    in a row; results that never reach either count no longer check about
    5 times a second for good.
  • interval is at most 24h and timeout at least 10ms. Larger
    intervals could panic the check task (with a huge fall), and a 1ms
    timeout gave plugins 0ms.
  • [store] map_size is rounded up to a multiple of 64KiB. A size LMDB
    refuses (such as "1048577") passed check-config and then stopped
    the daemon from starting.
  • A check task that is restarted after a bug now records the backend's
    move to unknown as a transition (reason check task restarted);
    before, the stored state changed silently.
  • The DNS server's view of a service is updated per backend instead of
    copying the whole service on every check.
  • DNS over TCP serves at most 128 connections at once and closes a
    connection with no query for 5 seconds, so clients can no longer use
    up the daemon's file descriptors.
  • If a DNS listener stops while the daemon runs, the daemon logs it and
    exits with status 70, so the service manager restarts it; before, DNS
    went silent.
  • DNS answers BADVERS to EDNS versions above 0 and REFUSED to classes
    other than IN; before, a CH query got IN records.
  • DNS compares names label by label: a label containing a . byte no
    longer puts a name outside the zone inside it.
  • Over UDP, a reply whose single record or SOA does not fit (names near
    255 bytes) is truncated by emfor itself and keeps its EDNS record.
  • Plugins run in a process group of their own. When a plugin exits, is
    stopped, or is killed because it stopped answering, the processes it
    started are killed too; before, they could outlive it.
  • A plugin's stderr lines and invalid stdout lines are rate limited: up
    to 100 at once, then 10 a second each. The log notes how many lines
    were dropped, so a plugin writing in a loop can no longer fill the
    log.
  • A ca_file is parsed again only when it changes (inode, size or
    times), not on every check. A replaced file still takes effect at the
    next check.
  • http clients for checks (one per backend with a domain host_header)
    are built off the async worker threads, because building one reads the
    system CA store. The peer secret_file is read off them too.
  • Documented: a connection that fails because this host has no route to
    the backend, or its own firewall blocks it, is down, not error.
    That has always been the behaviour.
  • When a check task is restarted after a bug while its backend is
    unknown, the backend's latency is now cleared and its reason set to
    check task restarted, in the store and in DNS.
  • A late result from the check task of a backend removed by a reload can
    no longer mark that backend healthy after a later reload adds it back.
    Each check task now has its own generation, and a backend keeps its
    state exactly while its task keeps running.
  • check-config now refuses an http path with #, \ or a control
    character. The request URL cut or rewrote these, so another path was
    c...
Read more

emfor 0.4.0

Choose a tag to compare

@vanillahsu vanillahsu released this 14 Nov 13:07

Breaking: the store format is version 2. The daemon writes
meta/schema_version 2, which adds the backend field backup. Clients
and emfor status from this release read versions 1 and 2; older ones
fail every lookup on a version 2 store with INCOMPATIBLE. Upgrade the
clients on a host before the daemon (see
Store format version).

Peer messages are encrypted by default. Members running 0.3.0 cannot
read them and drop them as malformed. To upgrade a peer group, set
encrypt = false in [peers] on each host as it is upgraded; once every
member runs this release, remove the line and reload each daemon.

Breaking, on FreeBSD only: examples/rc.d/emfor is gone, and the
example port keeps the store in /var/db/emfor (see Deployment).

The main additions are backup backends, slow start, a preference for
backends whose last check passed, sticky lookups that honour weight,
encrypted peer messages, and the smtp, amqp, mongodb, grpc and
quic plugins of emfor-check. The plugin protocol (1) is unchanged.
As with every release, install libemfor from the same release as the
daemon.

Daemon

  • Peer group messages are encrypted (EMP2: ChaCha20-Poly1305 with a key
    derived from secret_file by HKDF-SHA256 and a random nonce per
    message), so backend names, addresses and reasons no longer travel in
    the clear. New [peers] encrypt (default true, applied on reload):
    false sends the signed plaintext EMP1 of 0.3.0. Both formats are
    accepted either way; with encrypt = true the daemon warns once a
    minute per member that still sends EMP1. A message that does not
    decrypt counts as auth in emfor_peer_messages_rejected_total.
    Secret rotation works as before. There is no forward secrecy: whoever
    later gets the secret can read recorded messages, so rotate it
    regularly. See [peers].

  • Backup backends: a backend table takes backup = true. Backups are
    used only while no other backend passes its checks: clients and the DNS
    server take the primaries whose last check passed, else the backups
    whose last check passed, else the healthy primaries, else the healthy
    backups, and balance works within that set; fastest decides which
    backends are fast among the primaries and among the backups
    separately. When no backend is
    healthy, on_all_down decides as before and backups count like the
    others. SRV answers list only the chosen set. At least one backend of a
    service must not be a backup, and backup is applied on reload. See
    Backup backends.

  • Clients and the DNS server leave out a healthy backend whose last
    check failed (its reason is set) while another healthy backend passed
    its last check, so a backend stops being chosen at its first failed
    check instead of after fall of them. When every healthy backend is
    failing, all of them are used as before. The store format is
    unchanged.

  • slow_start (per service, or in [defaults]; "0s", off, by
    default, else 1s to 1h): a backend that turns from unhealthy to
    healthy gets a growing share of traffic over that time, in 10 steps,
    instead of its full share at once. It applies to client lookups and
    DNS A/AAAA answers for every method but failover and sticky with a
    key, and to SRV weights. The store gains slow_start_ms on services
    and recovered_at on backends; clients that do not know them (as
    emfor-client 0.3.0 reading a version 1 store) give a recovered backend
    its full share at once. See
    Slow start.

  • With on_all_down = "open" and no healthy backend, clients and the DNS
    server now use the unknown backends (not decided yet, or whose checks
    only give errors) before the unhealthy ones, which failed fall
    checks in a row: first the unknown ones whose last check passed, then
    every unknown one, then every backend as before. closed and a stale
    store are unchanged, and so is the store format; older clients still
    choose among all backends.

  • The http check takes expect_body: text the response body must
    contain, for health endpoints that answer 200 either way. The body is
    read only when the status matches, up to 64 KiB.

  • On macOS, exec check commands and plugins no longer inherit the
    daemon's sockets and pipes. A plugin could keep a check connection
    open for as long as it ran, so the backend never saw it closed.

  • Peer groups notice a failure that only a member which is not one of the
    backend's assigned checkers sees sooner (the known limit of 0.3.0).
    Such a member now checks the backend itself at least every third
    interval instead of every fifth, and when its own check fails while
    the assigned checkers see the backend up it starts a fast recheck run
    (every 200 ms, up to fall checks), as after a peer's down report.
    With the defaults the backend is down there after at most about 6.4
    seconds instead of about 14 (estimated from the schedule; 6 without a
    group). In exchange a backend gets about 3 + (N − 3) / 3 checks per
    interval from N members instead of 3 + (N − 3) / 5. Measured with
    interval = "2s" and fall = 3: 0.8 to 5.8 seconds, 3.3 seconds
    median, over 12 failures started at different points of the schedule.

Clients

  • Behaviour change: balance = "sticky" with an affinity key now
    honours weight: a backend of weight w gets w shares of the keys. Its
    value is the highest of MD5(key + id) and MD5(key + id + "#" + i)
    for i from 1 to w − 1, integers only, so every host and language still
    picks the same backend for a key. With every weight 1 nothing changes.
    A sticky service that already sets other weights moves some keys when
    its clients are upgraded, and old and new clients pick differently for
    those keys meanwhile, so upgrade the clients of such a service
    together. Raising a weight only moves keys to that backend; slow start
    does not apply, and a weight above 100 in the store counts as 100. A lookup costs one MD5 per unit of weight: 0.8 µs for 10
    backends of weight 1, 72 µs for 10 of weight 100 (Apple M4 Max). The
    Rust API adds testing::rendezvous_pick_weighted. See
    sticky.

emfor-check

  • role = "replica" for mysql and pgsql, and for all three database
    plugins it now means a replica that is replicating: a redis replica
    whose link to its master is connected, a MySQL or MariaDB replica
    whose IO and SQL threads both run, a PostgreSQL standby with a WAL
    receiver. Otherwise the backend is down with a replication: reason.
    A redis replica that was up with role = "replica" while its link
    was down is now down.
  • max_lag: with role = "replica", the seconds a replica may be behind
    before it is down. See Replicas for
    what each plugin measures and the rights the checks need.
  • emfor-check smtp checks SMTP servers: the greeting must be 220 and
    EHLO must answer 250, on a new connection, optionally upgraded with
    STARTTLS (starttls = true) or TLS from the start (tls = true,
    port 465); then it sends QUIT. It never logs in or sends mail. It is
    down with greeting: , ehlo: or tls: STARTTLS not offered. See
    docs/deploy/plugins.md.
  • emfor-check amqp checks AMQP 0-9-1 brokers such as RabbitMQ: a
    PLAIN login and Connection.Open of vhost on a new connection, over
    plaintext or TLS, then a normal Connection.Close. It is down with
    auth: 403 ACCESS_REFUSED - … for a rejected login and
    vhost: 530 NOT_ALLOWED - … for a vhost the user cannot open. See
    docs/deploy/plugins.md.
  • emfor-check mongodb checks MongoDB 4.4 and later: hello on a new
    connection, plaintext or TLS, an optional SCRAM-SHA-256 login (user,
    password_file, auth_source), and role = "primary" (a replica set
    primary, a standalone server or mongos) or role = "secondary", so
    balance = "failover" follows the replica set's primary. See
    docs/deploy/plugins.md.
  • emfor-check grpc checks gRPC services with the standard health
    service: grpc.health.v1.Health/Check over HTTP/2, plaintext or TLS
    with ALPN h2, is up when it answers SERVING. It is down with
    status: NOT_SERVING, grpc: 12 UNIMPLEMENTED (no health service) or
    grpc: 5 NOT_FOUND: … (an unknown service). See
    docs/deploy/plugins.md.
  • emfor-check quic checks QUIC servers over UDP: a new QUIC version 1
    handshake on every check, without 0-RTT, offering the ALPN values of
    alpn (default ["h3"]) and verifying the certificate (tls_verify,
    server_name, ca_file; always TLS 1.3). It is up when the handshake
    completes. With http3 = true it also sends an HTTP/3 GET for path
    and judges expect_status and expect_body like the built-in http
    check. It is down with quic: version negotiation failed, tls: …
    (for example tls: no application protocol (ALPN)),
    quic: closed by peer: <code>[: <reason>], quic: stateless reset,
    query: HTTP/3: …, unexpected status <n> or expect_body: got "…".
    UDP has no "connection refused", so a closed port or a firewall that
    drops UDP is a timeout. The ...
Read more

emfor 0.3.0

Choose a tag to compare

@vanillahsu vanillahsu released this 13 Nov 16:14

Breaking, on FreeBSD only: the rc.d script keeps its pid files in
/var/run/emfor/.
Scripts that read /var/run/emfor.child.pid must read
/var/run/emfor/emfor.child.pid, and installed copies of
examples/rc.d/emfor must be replaced (see
Deployment). Nothing else needs a change: configs, commands and the
--json output of 0.2.0 work as before.

The main addition is peer groups: the daemons of one site can pass
backend failures on, vote on a backend's state and split the checks
between them. Without a [peers] table the daemon behaves as in 0.2.0.
The store format (meta/schema_version 1) and the plugin protocol (1) are
unchanged; the new store key meta/peers is read only by emfor status.
As with every release, install libemfor from the same release as the
daemon.

Daemon

  • New optional [peers] table: the emfor daemons of one site that list
    each other form a peer group. Members send each other a heartbeat every
    second over UDP, signed with HMAC-SHA256 using the secret in
    secret_file. When a member sees a backend go from healthy to unhealthy
    it tells the others, which recheck that backend at once and every 200 ms
    (up to fall checks) instead of waiting for their next interval. A
    report never changes a backend's state by itself. Reports are shared
    only for services whose check, check_options and timeout match.
    New metrics emfor_peer_up, emfor_peer_messages_rejected_total and
    emfor_peer_notifications_total. Without [peers] nothing changes.
    See
    docs/manual/03-configuration.md
    and docs/deploy/daemon.md.
  • Majority vote: with three or more live members, each backend has three
    assigned checkers, chosen by rendezvous hashing over the live members.
    They send their own check results to the others, and two down votes
    mark the backend down on every member, also where its own check passes;
    the reason then reads peers 2/3: …, and a failure only this host sees
    reads local: …. With fewer members or no recent vote each daemon
    decides from its own checks as before. emfor status shows a peers:
    line, members whose check definition differs, and the marker local-up;
    --json has peers, from the new store key meta/peers (format
    version stays 1, clients ignore it). New metrics
    emfor_peer_results_total and emfor_peer_disagreements.
  • Split checks: a member that is not one of a backend's assigned checkers
    no longer checks it every interval. While two assigned checkers agree
    their results decide (up, or down with peers 2/3: …); it still
    checks the backend itself, with the service's own check, at least every
    fifth interval, every interval while its own check fails, and whenever
    the votes do not agree. A failure only such a member sees is noticed up
    to about five intervals later. New metric
    emfor_peer_checks_skipped_total.
  • New setting peers (default true), per service or in [defaults]:
    peers = false leaves a service out of the peer group on this host. It
    sends no reports or results for it, ignores those it receives, and
    decides from its own checks only; a reload applies it and keeps the
    backends' state. meta/peers lists such services in services_off.
    The host announces those services in its heartbeats (off: the first
    4 bytes of SHA-256 of each name, as 8 hex digits, at most 80; with more
    it logs a warning), and the other members leave it out when choosing
    those services' assigned checkers, so each host can set it on its own.
    Each entry of members in meta/peers has services_off with the
    services that member leaves out, and emfor status uses it for voted
    and assigned_checkers.

emfor status

  • emfor status marks backends this daemon is not an assigned checker of
    with voted, adds checks N/M backends here to the peers: line when
    there are any, and --json has assigned_checkers (each backend's
    assigned checkers, worked out from meta/peers).
  • emfor status shows, after the peers: line, one line per member that
    leaves shown services out of the peer group:
    peers: 10.0.0.6:5302 leaves out api.svc.emfor db.svc.emfor. Only the
    services shown are listed (so --service narrows it); --json is
    unchanged.
  • emfor status --watch scrolls when the services and recent changes do
    not fit in the terminal: the store: and daemon: lines stay at the
    top, PgUp/PgDn (or Space) move one screen, Up/Down one line, Home/End
    to the start or end, and the last row shows which lines are on screen.
    Lines wider than the terminal are cut instead of wrapped. q or Ctrl-C
    ends the watch, and the terminal is restored also after a crash. When
    it cannot install its signal handlers it now exits 2 instead of 0.

emfor-check

  • emfor-check dns checks DNS servers: it sends one query (name,
    type, over UDP or TCP) and is up when the response code is one of
    expect_rcode (default NOERROR) and, with expect_answer = true, the
    answer has a record of the asked type. Over UDP the query is sent again
    once at half of the timeout. Reasons start with rcode: , answer: ,
    query: , connect: or timeout. See
    docs/deploy/plugins.md.
  • emfor-check radius checks RADIUS servers: it sends a Status-Server
    (RFC 5997) by default, or an Access-Request with PAP for a test
    account, with Message-Authenticator first, and is up when the server
    answers with an accept signed with the shared secret from
    secret_file. Reasons start with auth: , query: , connect: or
    timeout; servers that check Message-Authenticator, such as
    FreeRADIUS, do not answer a request made with the wrong secret, which
    then shows as timeout. See
    docs/deploy/plugins.md.
  • emfor-check sip checks SIP servers with an OPTIONS request over UDP,
    TCP or TLS, and is up when the final status code is in
    expect_status, by default 200 to 499: a proxy that answers 404 or
    407 is working, while 503 (overload, maintenance) and other 5xx and
    6xx codes are down with status: <code> <reason>. See
    docs/deploy/plugins.md.
  • emfor-check diameter checks Diameter nodes (HSS, PCRF, OCS, DRA)
    over TCP or TLS: a capabilities exchange and a device watchdog on a
    new connection, then a Disconnect-Peer-Request. It is down with the
    Result-Code and its RFC 6733 name, such as
    cea: 3010 DIAMETER_UNKNOWN_PEER or
    cea: 5010 DIAMETER_NO_COMMON_APPLICATION. TLS can present a client
    certificate (tls_cert_file, tls_key_file), which nodes such as
    freeDiameter require. See
    docs/deploy/plugins.md.

Deployment

  • The FreeBSD rc.d script failed to start emfor with any emfor_user
    other than root, such as the port's default emfor: daemon: pidfile ``/var/run/emfor.child.pid'': Permission denied. rc.subr runs the whole
    command as ${name}_user, the daemon(8) supervisor included, and that
    user cannot write /var/run. The pid files are now in /var/run/emfor/,
    which service emfor start creates owned by emfor_user, and daemon(8)
    no longer gets -u (it needs root). Replace installed copies of
    examples/rc.d/emfor; scripts that read
    /var/run/emfor.child.pid read /var/run/emfor/emfor.child.pid now.

Documentation

Known limits

  • Unchanged from 0.2.0: FreeBSD has not been tested on a host yet, peer
    groups included (tests/resolver/freebsd.sh
    checks the daemon, the rc.d script and the port example there).
  • Unchanged from 0.1.0: lookup throughput across threads levels off at
    about two million a second on Linux
    (docs/performance.md).
  • Unchanged from 0.2.0: with [store] sync = false (the default), a host
    crash can leave the store damaged; the daemon then sets it aside at the
    next start and the state history is lost.
  • In a peer group, a failure that only a member which is not one of the
    backend's assigned checkers sees is noticed there up to about five
    intervals later than without a group.
  • A host announces at most 80 services left out of the peer group; the
    other members still count it as a candidate for the rest.

emfor 0.2.0

Choose a tag to compare

@vanillahsu vanillahsu released this 13 Nov 06:02

Breaking: the daemon is started with emfor server. emfor without a
subcommand now only prints its usage and exits 2, so running it to see the
options no longer starts a second daemon. When upgrading from 0.1.x, change
the service definition before restarting:

0.1.x 0.2.0
emfor --config X emfor server --config X
emfor --check-config --config X emfor check-config --config X

--check-config now only prints --check-config was replaced by "emfor check-config" and exits 2. emfor status is unchanged.
examples/rc.d/emfor,
examples/systemd/emfor.service and the
FreeBSD port example use the new commands; copies of the 0.1.x versions
must be replaced. See docs/deploy/daemon.md.

The store format (meta/schema_version 1) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon: the daemon and the clients must also agree on the lock
file's reader slots (below).

Daemon

  • The store is no longer flushed to disk on every commit. The new
    [store] sync setting defaults to false (LMDB's MDB_NOSYNC): a
    stalled disk could make the heartbeat late and every client on the host
    treat the store as stale. The store is rewritten on every start, so
    nothing in it needs to survive a host crash. Set sync = true to keep the
    old behavior. Commits now run on a separate thread.
  • When the daemon starts and finds the store damaged (MDB_CORRUPTED,
    MDB_PAGE_NOTFOUND or MDB_INVALID, as after a power loss without
    flushing), it moves the file to <path>.corrupt, deletes the lock file,
    and starts with a new store instead of exiting. The state history of
    emfor status --history is lost with it. See
    docs/deploy/daemon.md.
  • An application process killed in the middle of a lookup (SIGKILL, OOM,
    a crash) left its LMDB reader slot taken. Each such slot kept every page
    freed after it from being reused, so the store grew until commits failed
    with MDB_MAP_FULL, every client then saw a stale store, and restarting
    the daemon did not help while applications kept the store open. The
    daemon now frees these slots when it opens the store, about once a minute
    after that, and right after a failed commit, and logs how many it freed.
  • The store's lock file has 1024 reader slots instead of LMDB's default
    of 126. The daemon and every client ask for the same number, since the
    process that opens the store first sizes the lock file.
  • While a backend's state is unknown (after a start, a reload that adds
    it, or a run of errors) and a check returns up or down, the next check
    runs after min(interval, 200ms) instead of interval, and the first
    check starts within that time too. With interval = "2s" and rise = 2,
    backends are known within about half a second of a start instead of 2 to
    4 seconds, during which services with on_all_down = "closed" answered
    ALL_DOWN.
  • Check reasons are escaped and bounded before they reach the store, the
    logs and emfor status: tab, CR and LF become \t, \r, \n, other
    control characters \u{1b} and the like, and a reason longer than 512
    characters is cut and ends in …. A backend or plugin could send
    terminal escape sequences, or reasons large enough to slow every lookup.
  • A plugin line longer than 65536 bytes no longer grows the daemon's memory
    without bound: on stdout the plugin is stopped and restarted as if it had
    exited; on stderr the rest of the line is discarded.
  • Signals: emfor server installs its signal handlers before it loads the
    config. Loading starts plugins and can take seconds; a SIGHUP in that
    time (a service emfor reload or config management reloading right after
    a start) used to kill the daemon, and now reloads once it has started. A
    SIGTERM or SIGINT while loading at start, or during a reload, ends the
    loading at once (a reload is then not applied) and the daemon exits 0,
    instead of waiting up to about 10 seconds for slow plugins.
  • If the state writer stopped while the daemon ran, the daemon kept running
    without a heartbeat and every client saw a stale store; it now exits with
    status 70 (EX_SOFTWARE), which the systemd unit restarts. A backend
    check task that stopped left its backend in its last state; it is now
    started again from unknown, at once and then after 1s, 2s, 4s and so on
    up to 60s if it keeps stopping. Both only happen through a bug (a panic).
  • A [dns] zone longer than 242 characters made the daemon panic on every
    query in the zone: the SOA name hostmaster.<zone> went past the
    253-character limit. Such a zone is now a config error.
  • /metrics reports the store and the plugins:
    emfor_store_last_commit_timestamp_seconds, emfor_store_used_bytes
    and emfor_store_map_size_bytes (commits fail with MDB_MAP_FULL when
    the used bytes reach the map size), and emfor_plugin_up and
    emfor_plugin_restarts_total per plugin. Before, a crashing plugin showed
    only as error checks.
  • The http checker shares one HTTP client among a service's backends
    unless host_header is a domain name (which needs one client per
    backend to connect to it), and no longer builds clients while holding a
    lock.
  • emfor status --watch restored the terminal only on Ctrl-C. Killed, or
    ended by a closed ssh session (SIGHUP), it left the terminal on the
    alternate screen with the cursor hidden.

Clients

  • Applications kept reading the old store file after it was replaced (the
    daemon moving a damaged store aside, or the file deleted and created
    again), so they saw a stale store until they were restarted. A lookup
    that finds the store stale, or fails with an error other than ALL_DOWN
    from a fresh store, now checks whether the store path holds another file,
    at most once every 100 ms per process, and switches to it. When the new
    file cannot be opened yet, lookups keep using the old one. This covers
    every client library.
  • Lookups take read locks only, so lookups on many threads no longer queue
    on a mutex, and a round-robin lookup no longer allocates the service name
    each time. Failing and stale lookups run about as fast as successful
    ones.
  • The Python client looked for libemfor.so.0, a name libemfor is never
    installed under, so with only the runtime library installed (no
    libemfor.so symlink, not in the ldconfig cache) import emfor
    failed. It now looks for the versioned names derived from
    emfor.__version__: libemfor.so.0.2 and libemfor.0.2.dylib for 0.2.x.

Deployment

  • The FreeBSD rc.d script has daemon(8) start emfor again 2 seconds
    after it exits, like Restart=on-failure in the systemd unit. The
    supervisor's pid is in /var/run/emfor.pid and emfor's in
    /var/run/emfor.child.pid. service emfor reload checks the config first
    and fails without touching the daemon when it is invalid.
  • service emfor configtest, which the rc.d script also runs before
    starting, checked the config as root and so started the config's plugins
    as root even with emfor_user set. It now runs as emfor_user.
    Replace copies of examples/rc.d/emfor.
  • The example FreeBSD port installs the rc.d script with USE_RC_SUBR
    from files/emfor.in; a test keeps it in step with
    examples/rc.d/emfor.
  • The systemd unit and docs/deploy/daemon.md use
    /usr/local/bin/emfor, where cargo install and the FreeBSD port put
    it, instead of /usr/local/sbin/emfor. Change the unit's paths if you
    installed to sbin.
  • The docs spell out what the default [store] lock_file_mode = "0666"
    allows: any local user can write the lock file and so stop the daemon's
    commits or disturb lookups. See "Who can use the store" in
    docs/deploy/daemon.md for
    limiting the store to a group. The default is unchanged.

Known limits

  • FreeBSD has not been tested on a host yet: neither the daemon nor the
    rc.d script and the port example have run there
    (tests/resolver/freebsd.sh checks them).
  • Unchanged from 0.1.0: lookup throughput across threads levels off at
    about two million a second on Linux
    (docs/performance.md).
  • With [store] sync = false (the default), a host crash can leave the
    store damaged; the daemon then sets it aside at the next start and the
    state history is lost.

emfor 0.1.1

Choose a tag to compare

@vanillahsu vanillahsu released this 13 Nov 00:00

The repository moved to github.com/embark-systems/emfor. The daemon,
emfor-check, the store format and the clients behave as in 0.1.0. As
with every release, install libemfor from the same release as the
daemon.

Packaging

  • cargo install --path crates/emfor-daemon installs only emfor. In
    0.1.0 it also installed emfor-test-plugin, a plugin used only by
    emfor's own tests; it is now built only with the test-plugin feature.
  • docs/deploy/daemon.md gives the cargo install
    commands for crates/emfor-daemon and crates/emfor-check, including
    for packaging systems that run cargo install per crate. The repository
    root is a workspace, so cargo install --path . fails.

Clients

  • The Go module path is github.com/embark-systems/emfor/clients/go
    (was github.com/vanillahsu/emfor/clients/go). Its versions are tags
    clients/go/vX.Y.Z:
    go get github.com/embark-systems/emfor/clients/go@v0.1.1.

Development

  • RELEASING.md lists the steps of a release; a test checks
    that the Python, Perl, PHP and Go clients carry the workspace version.
  • tests/clients/Dockerfile describes the Debian image the local checks
    run in.

Known limits

  • Unchanged from 0.1.0: FreeBSD has not been tested on a host yet, and
    lookup throughput across threads levels off at about two million a
    second on Linux (docs/performance.md).

emfor 0.1.0

Choose a tag to compare

@vanillahsu vanillahsu released this 12 Nov 23:35

The first release. The daemon, emfor-check and every client come from
this one version; libemfor and the clients built on it must be upgraded
together with the daemon (they share its LMDB build).

Daemon

  • Health checks: tcp with optional send / expect and TLS,
    http / https, exec for Nagios-style scripts, and long-running
    plugins speaking protocol v1. Per-service
    interval, timeout, rise, fall and jitter, with [defaults].
  • State store: a local LMDB file, record format version 1, with a
    heartbeat so readers notice a daemon that stopped. Backend state,
    latency, the last failure reason and the last 20 state changes per
    backend.
  • Balance methods: round-robin, sticky (rendezvous hashing on an
    affinity key), failover, weighted, locality (same site first) and
    fastest (backends within a latency margin of the quickest).
    on_all_down = "open" or "closed" decides what an all-down service
    returns.
  • DNS server (optional [dns]): A, AAAA and SRV answers for its own zone
    (default svc.emfor), meant to sit behind the host's resolver; a warning
    when a service's backends use different ports.
  • /metrics (optional [metrics]): Prometheus metrics for the daemon,
    every service and backend.
  • emfor status: the daemon's state, every service and backend, recent
    state changes and flapping backends; --service, --json and --watch.
  • SIGHUP reloads the config when it is fully valid; --check-config
    checks a config without running.
  • EMFOR_PREFIX at build time sets the default config path
    (/etc/emfor/emfor.toml without it).

emfor-check

  • Plugins for redis, MySQL / MariaDB, PostgreSQL and LDAP, with TLS.
  • The emfor-plugin crate for writing plugins in Rust.

Clients

  • One implementation in Rust (emfor-client), also built with cargo-c as
    the C library libemfor.
  • Go, Python, Perl and PHP clients wrapping libemfor.
  • Every client returns ip:port or one of the error codes UNAVAILABLE,
    INCOMPATIBLE, NOT_FOUND and ALL_DOWN, checked against shared
    conformance data (tests/conformance/v1). Configured only by
    EMFOR_STORE and EMFOR_MAX_STALENESS; safe across threads and fork.
  • A lookup takes about 1 µs; see docs/performance.md.

Deployment

  • Guides for the daemon, systemd, metrics and alerting
    (docs/deploy/daemon.md), for forwarding the DNS
    zone from dnsmasq, unbound, systemd-resolved and macOS
    (docs/deploy/dns-resolver.md), and for
    emfor-check (docs/deploy/plugins.md).
  • Examples: a fully commented config, a systemd unit (run under systemd
    257), a FreeBSD rc.d script, and Prometheus alerting rules with promtool
    tests.

Known limits

  • FreeBSD is a deployment platform but has not been tested on a host yet;
    the CI job that covers it is run by hand and has not passed yet.
  • Lookup throughput across threads levels off at about two million a
    second on Linux; see docs/performance.md.