Repository navigation
emfor 0.2.0
Breaking: the daemon is started with emfor server. emfor without a
subcommand now only prints its usage and exits 2, so running it to see the
options no longer starts a second daemon. When upgrading from 0.1.x, change
the service definition before restarting:
| 0.1.x | 0.2.0 |
|---|---|
emfor --config X |
emfor server --config X |
emfor --check-config --config X |
emfor check-config --config X |
--check-config now only prints --check-config was replaced by "emfor check-config" and exits 2. emfor status is unchanged.
examples/rc.d/emfor,
examples/systemd/emfor.service and the
FreeBSD port example use the new commands; copies of the 0.1.x versions
must be replaced. See docs/deploy/daemon.md.
The store format (meta/schema_version 1) and the plugin protocol (1) are
unchanged. As with every release, install libemfor from the same release
as the daemon: the daemon and the clients must also agree on the lock
file's reader slots (below).
Daemon
- The store is no longer flushed to disk on every commit. The new
[store] syncsetting defaults tofalse(LMDB'sMDB_NOSYNC): a
stalled disk could make the heartbeat late and every client on the host
treat the store as stale. The store is rewritten on every start, so
nothing in it needs to survive a host crash. Setsync = trueto keep the
old behavior. Commits now run on a separate thread. - When the daemon starts and finds the store damaged (
MDB_CORRUPTED,
MDB_PAGE_NOTFOUNDorMDB_INVALID, as after a power loss without
flushing), it moves the file to<path>.corrupt, deletes the lock file,
and starts with a new store instead of exiting. The state history of
emfor status --historyis lost with it. See
docs/deploy/daemon.md. - An application process killed in the middle of a lookup (SIGKILL, OOM,
a crash) left its LMDB reader slot taken. Each such slot kept every page
freed after it from being reused, so the store grew until commits failed
withMDB_MAP_FULL, every client then saw a stale store, and restarting
the daemon did not help while applications kept the store open. The
daemon now frees these slots when it opens the store, about once a minute
after that, and right after a failed commit, and logs how many it freed. - The store's lock file has 1024 reader slots instead of LMDB's default
of 126. The daemon and every client ask for the same number, since the
process that opens the store first sizes the lock file. - While a backend's state is
unknown(after a start, a reload that adds
it, or a run of errors) and a check returnsupordown, the next check
runs aftermin(interval, 200ms)instead ofinterval, and the first
check starts within that time too. Withinterval = "2s"andrise = 2,
backends are known within about half a second of a start instead of 2 to
4 seconds, during which services withon_all_down = "closed"answered
ALL_DOWN. - Check reasons are escaped and bounded before they reach the store, the
logs andemfor status: tab, CR and LF become\t,\r,\n, other
control characters\u{1b}and the like, and a reason longer than 512
characters is cut and ends in…. A backend or plugin could send
terminal escape sequences, or reasons large enough to slow every lookup. - A plugin line longer than 65536 bytes no longer grows the daemon's memory
without bound: on stdout the plugin is stopped and restarted as if it had
exited; on stderr the rest of the line is discarded. - Signals:
emfor serverinstalls its signal handlers before it loads the
config. Loading starts plugins and can take seconds; a SIGHUP in that
time (aservice emfor reloador config management reloading right after
a start) used to kill the daemon, and now reloads once it has started. A
SIGTERM or SIGINT while loading at start, or during a reload, ends the
loading at once (a reload is then not applied) and the daemon exits 0,
instead of waiting up to about 10 seconds for slow plugins. - If the state writer stopped while the daemon ran, the daemon kept running
without a heartbeat and every client saw a stale store; it now exits with
status 70 (EX_SOFTWARE), which the systemd unit restarts. A backend
check task that stopped left its backend in its last state; it is now
started again fromunknown, at once and then after 1s, 2s, 4s and so on
up to 60s if it keeps stopping. Both only happen through a bug (a panic). - A
[dns] zonelonger than 242 characters made the daemon panic on every
query in the zone: the SOA namehostmaster.<zone>went past the
253-character limit. Such a zone is now a config error. /metricsreports the store and the plugins:
emfor_store_last_commit_timestamp_seconds,emfor_store_used_bytes
andemfor_store_map_size_bytes(commits fail withMDB_MAP_FULLwhen
the used bytes reach the map size), andemfor_plugin_upand
emfor_plugin_restarts_totalper plugin. Before, a crashing plugin showed
only aserrorchecks.- The
httpchecker shares one HTTP client among a service's backends
unlesshost_headeris a domain name (which needs one client per
backend to connect to it), and no longer builds clients while holding a
lock. emfor status --watchrestored the terminal only on Ctrl-C. Killed, or
ended by a closed ssh session (SIGHUP), it left the terminal on the
alternate screen with the cursor hidden.
Clients
- Applications kept reading the old store file after it was replaced (the
daemon moving a damaged store aside, or the file deleted and created
again), so they saw a stale store until they were restarted. A lookup
that finds the store stale, or fails with an error other thanALL_DOWN
from a fresh store, now checks whether the store path holds another file,
at most once every 100 ms per process, and switches to it. When the new
file cannot be opened yet, lookups keep using the old one. This covers
every client library. - Lookups take read locks only, so lookups on many threads no longer queue
on a mutex, and a round-robin lookup no longer allocates the service name
each time. Failing and stale lookups run about as fast as successful
ones. - The Python client looked for
libemfor.so.0, a name libemfor is never
installed under, so with only the runtime library installed (no
libemfor.sosymlink, not in theldconfigcache)import emfor
failed. It now looks for the versioned names derived from
emfor.__version__:libemfor.so.0.2andlibemfor.0.2.dylibfor 0.2.x.
Deployment
- The FreeBSD rc.d script has
daemon(8)startemforagain 2 seconds
after it exits, likeRestart=on-failurein the systemd unit. The
supervisor's pid is in/var/run/emfor.pidand emfor's in
/var/run/emfor.child.pid.service emfor reloadchecks the config first
and fails without touching the daemon when it is invalid. service emfor configtest, which the rc.d script also runs before
starting, checked the config as root and so started the config's plugins
as root even withemfor_userset. It now runs asemfor_user.
Replace copies of examples/rc.d/emfor.- The example FreeBSD port installs the rc.d script with
USE_RC_SUBR
fromfiles/emfor.in; a test keeps it in step with
examples/rc.d/emfor. - The systemd unit and docs/deploy/daemon.md use
/usr/local/bin/emfor, wherecargo installand the FreeBSD port put
it, instead of/usr/local/sbin/emfor. Change the unit's paths if you
installed tosbin. - The docs spell out what the default
[store] lock_file_mode = "0666"
allows: any local user can write the lock file and so stop the daemon's
commits or disturb lookups. See "Who can use the store" in
docs/deploy/daemon.md for
limiting the store to a group. The default is unchanged.
Known limits
- FreeBSD has not been tested on a host yet: neither the daemon nor the
rc.d script and the port example have run there
(tests/resolver/freebsd.sh checks them). - Unchanged from 0.1.0: lookup throughput across threads levels off at
about two million a second on Linux
(docs/performance.md). - With
[store] sync = false(the default), a host crash can leave the
store damaged; the daemon then sets it aside at the next start and the
state history is lost.