Repository navigation
emfor 0.3.0
Breaking, on FreeBSD only: the rc.d script keeps its pid files in
/var/run/emfor/. Scripts that read /var/run/emfor.child.pid must read
/var/run/emfor/emfor.child.pid, and installed copies of
examples/rc.d/emfor must be replaced (see
Deployment). Nothing else needs a change: configs, commands and the
--json output of 0.2.0 work as before.
The main addition is peer groups: the daemons of one site can pass
backend failures on, vote on a backend's state and split the checks
between them. Without a [peers] table the daemon behaves as in 0.2.0.
The store format (meta/schema_version 1) and the plugin protocol (1) are
unchanged; the new store key meta/peers is read only by emfor status.
As with every release, install libemfor from the same release as the
daemon.
Daemon
- New optional
[peers]table: the emfor daemons of onesitethat list
each other form a peer group. Members send each other a heartbeat every
second over UDP, signed with HMAC-SHA256 using the secret in
secret_file. When a member sees a backend go from healthy to unhealthy
it tells the others, which recheck that backend at once and every 200 ms
(up tofallchecks) instead of waiting for their next interval. A
report never changes a backend's state by itself. Reports are shared
only for services whosecheck,check_optionsandtimeoutmatch.
New metricsemfor_peer_up,emfor_peer_messages_rejected_totaland
emfor_peer_notifications_total. Without[peers]nothing changes.
See
docs/manual/03-configuration.md
and docs/deploy/daemon.md. - Majority vote: with three or more live members, each backend has three
assigned checkers, chosen by rendezvous hashing over the live members.
They send their own check results to the others, and twodownvotes
mark the backend down on every member, also where its own check passes;
the reason then readspeers 2/3: …, and a failure only this host sees
readslocal: …. With fewer members or no recent vote each daemon
decides from its own checks as before.emfor statusshows apeers:
line, members whose check definition differs, and the markerlocal-up;
--jsonhaspeers, from the new store keymeta/peers(format
version stays 1, clients ignore it). New metrics
emfor_peer_results_totalandemfor_peer_disagreements. - Split checks: a member that is not one of a backend's assigned checkers
no longer checks it every interval. While two assigned checkers agree
their results decide (up, ordownwithpeers 2/3: …); it still
checks the backend itself, with the service's own check, at least every
fifth interval, every interval while its own check fails, and whenever
the votes do not agree. A failure only such a member sees is noticed up
to about five intervals later. New metric
emfor_peer_checks_skipped_total. - New setting
peers(defaulttrue), per service or in[defaults]:
peers = falseleaves a service out of the peer group on this host. It
sends no reports or results for it, ignores those it receives, and
decides from its own checks only; a reload applies it and keeps the
backends' state.meta/peerslists such services inservices_off.
The host announces those services in its heartbeats (off: the first
4 bytes of SHA-256 of each name, as 8 hex digits, at most 80; with more
it logs a warning), and the other members leave it out when choosing
those services' assigned checkers, so each host can set it on its own.
Each entry ofmembersinmeta/peershasservices_offwith the
services that member leaves out, andemfor statususes it forvoted
andassigned_checkers.
emfor status
emfor statusmarks backends this daemon is not an assigned checker of
withvoted, addschecks N/M backends hereto thepeers:line when
there are any, and--jsonhasassigned_checkers(each backend's
assigned checkers, worked out frommeta/peers).emfor statusshows, after thepeers:line, one line per member that
leaves shown services out of the peer group:
peers: 10.0.0.6:5302 leaves out api.svc.emfor db.svc.emfor. Only the
services shown are listed (so--servicenarrows it);--jsonis
unchanged.emfor status --watchscrolls when the services and recent changes do
not fit in the terminal: thestore:anddaemon:lines stay at the
top, PgUp/PgDn (or Space) move one screen, Up/Down one line, Home/End
to the start or end, and the last row shows which lines are on screen.
Lines wider than the terminal are cut instead of wrapped.qor Ctrl-C
ends the watch, and the terminal is restored also after a crash. When
it cannot install its signal handlers it now exits 2 instead of 0.
emfor-check
emfor-check dnschecks DNS servers: it sends one query (name,
type, over UDP or TCP) and is up when the response code is one of
expect_rcode(defaultNOERROR) and, withexpect_answer = true, the
answer has a record of the asked type. Over UDP the query is sent again
once at half of thetimeout. Reasons start withrcode:,answer:,
query:,connect:ortimeout. See
docs/deploy/plugins.md.emfor-check radiuschecks RADIUS servers: it sends a Status-Server
(RFC 5997) by default, or an Access-Request with PAP for a test
account, with Message-Authenticator first, and is up when the server
answers with an accept signed with the shared secret from
secret_file. Reasons start withauth:,query:,connect:or
timeout; servers that check Message-Authenticator, such as
FreeRADIUS, do not answer a request made with the wrong secret, which
then shows astimeout. See
docs/deploy/plugins.md.emfor-check sipchecks SIP servers with an OPTIONS request over UDP,
TCP or TLS, and is up when the final status code is in
expect_status, by default 200 to 499: a proxy that answers 404 or
407 is working, while 503 (overload, maintenance) and other 5xx and
6xx codes are down withstatus: <code> <reason>. See
docs/deploy/plugins.md.emfor-check diameterchecks Diameter nodes (HSS, PCRF, OCS, DRA)
over TCP or TLS: a capabilities exchange and a device watchdog on a
new connection, then a Disconnect-Peer-Request. It is down with the
Result-Code and its RFC 6733 name, such as
cea: 3010 DIAMETER_UNKNOWN_PEERor
cea: 5010 DIAMETER_NO_COMMON_APPLICATION. TLS can present a client
certificate (tls_cert_file,tls_key_file), which nodes such as
freeDiameter require. See
docs/deploy/plugins.md.
Deployment
- The FreeBSD rc.d script failed to start emfor with any
emfor_user
other thanroot, such as the port's defaultemfor:daemon: pidfile ``/var/run/emfor.child.pid'': Permission denied. rc.subr runs the whole
command as${name}_user, the daemon(8) supervisor included, and that
user cannot write/var/run. The pid files are now in/var/run/emfor/,
whichservice emfor startcreates owned byemfor_user, and daemon(8)
no longer gets-u(it needs root). Replace installed copies of
examples/rc.d/emfor; scripts that read
/var/run/emfor.child.pidread/var/run/emfor/emfor.child.pidnow.
Documentation
- A user manual in English (docs/manual) and
Traditional Chinese (docs/manual.zh_TW):
concepts, installation, configuration, health checks, backend
selection, clients, DNS, operations and troubleshooting. - SECURITY.md (how to report a vulnerability),
CONTRIBUTING.md and
docs/compatibility.md, which says which
interfaces stay stable across releases.
Known limits
- Unchanged from 0.2.0: FreeBSD has not been tested on a host yet, peer
groups included (tests/resolver/freebsd.sh
checks the daemon, the rc.d script and the port example there). - Unchanged from 0.1.0: lookup throughput across threads levels off at
about two million a second on Linux
(docs/performance.md). - Unchanged from 0.2.0: with
[store] sync = false(the default), a host
crash can leave the store damaged; the daemon then sets it aside at the
next start and the state history is lost. - In a peer group, a failure that only a member which is not one of the
backend's assigned checkers sees is noticed there up to about five
intervals later than without a group. - A host announces at most 80 services left out of the peer group; the
other members still count it as a candidate for the rest.