Repository navigation
Shepherd 0.1.0
Added
- Packaging plan (roadmap step 24). docs/PACKAGING.md specifies easy installs
for non-technical owners: the first-run wizard's questions, every target platform with its installer
and service manager, the port 53 and signing constraints, and the order of work. Documentation only;
no installer is built yet. - Self-update with rollback.
[update](on by default) checks GitHub's releases once a day on
the stable or beta channel, raises aself-updatealarm when a newer Shepherd exists, and the
Overview's new Software card offers "Update now" where the binary can be replaced in place.
Applying downloads the release for this platform, checks it againstSHA256SUMS, verifies that
file's Ed25519 signature against the key built into the binary (refusing an unsigned release or
a build with no key), swaps the binary in with the previous one kept asshepherd.prev, and
restarts into it. An updated binary that does not stay up for two minutes across three starts is
put back automatically on the next start;shepherd -rollbackdoes the same by hand. A read-only
container cannot be updated in place and the card says so. Both CI workflows sign the checksums
when theSHEPHERD_SIGNING_KEYsecret exists (scripts/sign-release,cmd/shepherd-sign); the
public half isupdate.ReleasePublicKeyB64, empty until the maintainer generates the pair. The
card also shows the data directory's size by part. See docs/UPDATE.md. - One retention budget.
[retention] max_mbcaps the data directory as a whole: it is split
into caps for query history, bandwidth history and network-health history (each still enforcing
its own), the Software card shows use against it, and a housekeeping check every six hours raises
a lowstorage-over-budgetalarm, naming the largest part, if the directory outgrows it anyway.
0, the default, leaves each store to its own cap (docs/UPDATE.md "Retention"). - Delivery settings for alarms.
GET/PUT /api/alarms/preferencesread and write the alarm
preferences, and the Notifications tab's new Delivery card sets the severity sent at once,
whether lower alarms wait for the digest, the digest's interval or time of day, and the time zone. - Threat feeds: which device reached for a known bad name or address. Four curated feeds
(HaGeZi's Threat Intelligence Feeds mini edition, IPsum level 3, abuse.ch's Feodo Tracker and
Spamhaus DROP; URLhaus and ThreatFox went behind an abuse.ch Auth-Key the same day and were
swapped out, and abuse.ch's SSL Blacklist has been empty since its deprecation in January 2025)
are downloaded every six
hours, cached under the data directory, and matched against every recorded DNS query's name and
answer, blocked or not: a name matches itself or a parent, an address matches exactly or by
prefix. A hit raises athreat-matchalarm attributed to the device, keyed by device and
indicator so a beacon is one alarm with a rising count, with the name, the indicator, the feed and
what the engine did with the query as evidence. The Alarms screen gains a Threat feeds tab
with each feed's state, a refresh button and the recent matches.[threats]is on by default;
the cost is a few map lookups per query and a few hundred kilobytes per feed. Local AI is set
aside by decision. See docs/THREATS.md. - Web Push delivery, end to end.
internal/webpushhad the sender, VAPID keys and subscription
store for months, waiting on HTTPS; the portal has it now through[tls], socmd/shepherdwires
it in: a VAPID key pair generated once under the data directory, subscriptions saved beside it, and
aPusheron the alarm Router next to ntfy. The Alarms screen gains a Notifications tab that
says whether this browser can subscribe (HTTPS, a home-screen web app on iOS, permission),
registers the service worker and subscribes with one button, lists every subscribed browser with
rename and remove, and sends a test notification through every channel.[alarms.push]is on by
default with an optionalcontact. New endpoints under/api/alarms/push. Amanifest.webmanifest
lets iOS add the portal to the Home Screen, which Safari requires for push. Roadmap step 9 is done
apart from a real-browser trial. - Reverse proxy: per-host access lists, a permanent HTTP-to-HTTPS redirect and HSTS. Each
proxied host can name who may reach it: addresses, prefixes like192.0.2.0/24, andlocal,
which stands for every network this machine is on plus loopback and Tailscale's ranges. Anyone
else gets a 403 from Shepherd itself, logged with the client address at most once per client per
minute.redirect_listennow answers with a 301 (308 for anything but GET and HEAD, so the method
and body survive) instead of a 302, and a per-hosthstsswitch adds a 30-day
Strict-Transport-Securityheader over HTTPS, off by default. The Proxy tab has both controls and
shows "Local only", "Restricted" and "HSTS" badges. See docs/REVERSEPROXY.md. - The passive ARP monitor's findings are alarms. Every event type in docs/ARP.md now maps to
an alarm (arp-gateway-changed,arp-spoofing,arp-ip-moved,arp-duplicate-ip,
arp-many-ips,arp-flood,arp-reply-storm) or deliberately to none: a new MAC is the
registry's ownnew-devicealarm, a private-address rotation is nothing, and the monitor's
baseline period raises nothing except a gateway change or suspected spoofing. The monitor's own
severity is kept per event, the alarm is keyed by the event's stable key so repeats fold, and
it is attributed to the registry's device for the suspect MAC. Roadmap step 10 is done.
Fixed
-
Flaky watch-mode tests. Several
internal/watchengine tests advanced the fake clock once and then
waited, so a tick fired before the engine had re-armed its timer was lost and the test timed out under the
race detector (and failedmain's QA). They now keep ticking until the engine catches up, and no longer
read a retry record that the previous round left behind. Test-only; no behaviour change. -
Release notes over GitHub's size limit. The Release workflow now trims notes longer than 120,000
characters and links the full CHANGELOG.md, which the first stable release exceeded (HTTP 422). -
Flaky daemon tests on temp-directory cleanup. Tests that run the whole daemon failed with
"TempDir RemoveAll cleanup: directory not empty" when a background worker was still writing into the data
directory as the test ended, which also failed the Release and Beta gates. Their config helper now retries
the removal until the workers have stopped. Test-only. -
Flaky wandering and excluded-device e2e tests. On a slow runner the list-load polls were answered
by the upstream and cached before the list loaded, so the later query was logged ascached, not
allowed, and the tests either matched an early un-annotated entry or timed out. They now accept both
actions and wait for the entry carrying the expected reason. Test-only. -
Code-scanning hardening. The trusted-device, session and lock cookies are now
Securewhenever the
request arrived over HTTPS (direct TLS orX-Forwarded-Proto: https); plain-HTTP LAN access keeps working. The
DNS-over-HTTPS client re-checks the upstream scheme right before it sends, the IPv6 route metric is parsed with
bounds that cannot overflowint, and the portfolio-notify workflow runs with no token permissions. -
The loop brake was stopping uploads, and Watch mode is reinstated as opt-in. The three
storms that retired Watch mode earlier on 2026-11-01 were read from sampled frames that shared
one IP identification field and one TTL, which the sampler's verdict called "the same frame
circulating". They were not the same frame. macOS, iOS and Linux (for an unconnected socket) put
identification 0 on every don't-fragment packet, so a Mac uploading to iCloud over QUIC at
1,392 bytes a packet reads as one frame going round to a verdict that counts identification
fields, and the duplicate brake "catching 3% of a runaway" was the brake correctly finding almost
nothing to catch. The 05:00 trip that day was a 270 Mbit/s QUIC flow to Apple from a MacBook,
overnight; the two instrumented "runaways" were 87,516 and 347,963 distinct frames of the same
shape. Reread by the whole frame, the live digest on that NAS now printsid 0 on every one: a don't-fragment sender, not a loopfor exactly that Mac and exactly those destinations.Three changes follow. The sampler's verdict compares the whole frame -- identification, total
length and the first sixteen bytes of the transport header -- and names a flow whose
identification is 0 throughout. The loop brake no longer stops steering on mirrored counters
alone, because a relay carrying a device's upload is mirrored by definition: a trip now needs the
duplicate brake to have refused at least a quarter of what the relay saw during the mirrored
seconds, or Shepherd's own output to have come back, or the sampled frames to be the same frame
over and over, or 300,000 packets a second, which no home device sends; mirrored counters with
none of those are logged once an hour as "a device sending through the relay, not a loop". And
the userspace relay gets the same duplicate brake as the kernel one (intercept.Dedup, same
fingerprint and window), so a loop is contained under either relay andduplicates_suppressed
means one thing. What is real -- the segment handing a steered device's frame back once, tens
of microseconds to a millisecond later, at about 60 to 80% of relayed frames when traffic is
light and 2 to 3% under load -- is dropped as before, and its mechanism is still not identified.
Watch mode stays off by default; the maintainer's NAS runs it again. Written up in
docs/WATCH.md, with the earlier readings kept and corrected rather than removed.
Changed
- A dual-homed forwarding host runs Watch mode by default.
watch.allow_routing_hostnow
defaults to true: the refusal rested on a correlation the measurements took apart (forwarding
moved one datagram during a storm, a single-homed phone echoed as the dual-homed Mac did, and the
storms were uploads), and what the shape actually risks is what the duplicate brake drops. The
engine notes the shape in the log and steers; set it to false to refuse as before. - Ethernet is preferred and Wi-Fi is only a fallback. With
watch.interfaceempty, a default
route through a wireless interface is overridden by a wired interface that is up on the
gateway's network, and the log says so. A relay on Wi-Fi shares the air with the devices whose
frames it carries, so every relayed frame costs airtime twice. - "Steered and healthy with almost nothing relayed" has a likely explanation. macOS caches the
gateway's link-layer address per connection when the connection opens and does not revisit it
when the neighbour entry changes, so connections opened before steering keep going to the real
router and only new ones reach Shepherd. Stated in docs/WATCH.md as the explanation that fits
the 100 MB transfer measurement; not yet verified by a capture on the device.
Changed
-
Network health screen redesigned. The old page spent most of its width on a near-empty status
card, a lonely "Range" row and three small unlabeled charts. It is now one header row (title, a
Healthy / Degraded / Outage pill, the range picker and "Probe now"), four metric tiles that each pair
the current reading with an area chart of the chosen range (latency, jitter, packet loss, and a new
reachability chart that shows an outage as a dip in reachable targets), then the bufferbloat card
(grade, idle / loaded / added, compact history) beside a short "how this is measured" card listing
the probe targets. Charts have labelled min/max gridlines and start/end times, scale strokes
correctly at any width, and the loss chart auto-scales above a 5% floor instead of using a 0–100%
axis that hid a 1% blip. Demo mode now has synthetic network-health data, so the page works there
too. -
Watch mode is retired and off by default. It was on, under the protect-by-default decision in
DESIGN.md; measurement retired it, not taste, and the evidence stays in
WATCH.md because it is the argument. Outbound-only by construction, so a device's
real traffic volume is always understated. It loops on real hardware, the router returning relayed
frames off the wire within tens of microseconds, with four attempted preventions each refuted by
measurement and only the loop brake reliably stopping it. And a device the portal reported as
steered and healthy had zero of a 100 MB transfer relayed, which makes its per-device numbers
unrepresentative rather than merely understated.Every place that advertised it as working now says otherwise: the README's "what works today"
entry, the profile table in DESIGN.md, and roadmap item 11. DESIGN.md also no longer says Watch
mode's relay should stay reusable by a future forwarding path -- GATEWAY.md
deliberately does not inherit it, because Gateway puts no Shepherd code on the per-packet path at
all. The[watch]configuration is kept so an install that deliberately enables it is not
broken by an upgrade, and so the retirement can be read next to the thing it retired. Removing the
roughly 14,000 lines ofinternal/watch,internal/interceptandcmd/intercept-trialis a
separate piece of work on purpose: it is large and mechanical and should not ride along with a
design decision.
Added
-
Bandwidth monitoring: how much each device, group and the whole network moves, live and over
time, including downloads, at no cost to the network. Watch mode already relays every
steered device's outbound frames and counts them, which is an exact upload figure. Downloads
never pass through Shepherd, so they are now estimated from the one thing in those frames that
describes them: every TCP segment a device sends carries an acknowledgement number, the count of
bytes it has received on that connection, and summing the advances per connection gives the
download volume with nothing new on the wire. The meter lives in both relays (intercept.AckMeter
in userspace,meter_ackin the XDP program, held to agree frame for frame), credits only
genuine advances (a first sighting is a baseline, a duplicate ACK is nothing, a jump of a
gigabyte is a reused port and re-baselines), and is blind by construction to QUIC, other UDP and
VPN traffic, which the portal says everywhere it shows the number. A newinternal/trafficstore
keeps one series per device per minute for two days, per hour for 90, per day forever under an
8 MiB cap, with exact rollups since bytes add; groups and the network are sums over devices
computed from current membership. A Bandwidth screen (Monitor) shows the live rates and
totals per device, a chart of download and upload for the network, a group or a device over the
last hour to 90 days, and who used the most;GET /api/traffic,/api/traffic/seriesand
/api/traffic/topare the API. The Watch screen and each device's drawer show the live download
estimate too. See docs/TRAFFIC.md. -
The vulnerability scanner now updates itself from a signature feed. Until now
internal/vulnscanmatched against a fixed table of twenty signatures compiled into the binary, so a
bug exploited this week could not be recognized before a release. Every Shepherd now fetches
feeds/vulnscan.jsonfrom this repository — rebuilt daily by the newvulnfeedworkflow
(cmd/vulnfeed,internal/vulnfeed) from the NVD CVE API and CISA's Known Exploited Vulnerabilities
catalog — every six hours, conditionally, and re-scans every device the moment the feed changes.
The feed is declarative data, never code: a signature names a product and version ranges, or a banner
substring, over what the existing probes already collect, and it cannot make Shepherd probe anything
new. It covers the fourteen products a banner or HTTPServer:header states a version for (OpenSSH,
Dropbear, vsftpd, ProFTPD, Exim, Apache httpd, nginx, lighttpd, OpenSSL, PHP, Boa, mini_httpd,
GoAhead, IIS); CISA-listed CVEs get a critical signature each, and the NVD's network-exploitable,
no-credentials critical and high CVEs are aggregated per product with the matched CVE identifiers in
the evidence. The last good feed is cached under the data directory, a bad download never replaces it,
[vulnscan].feed_url = ""turns it off, and the Vulnerabilities page shows the feed's state with an
"Update feed" button. VULNSCAN.md has the selection rules, the format and the
trust argument; its former "no external CVE database, and there is not going to be one" promise is
withdrawn in favor of this, deliberately. -
A design and staged migration for Gateway mode: GATEWAY.md. Written
because the night's Watch-mode measurements settled that per-device bandwidth, the thing actually
wanted, cannot be had from one-sided ARP steering at all: it is outbound-only by construction, its
counters are per device rather than per destination, and on the maintainer's own network a device
reported as steered and healthy had zero of a 100 MB transfer relayed.The document eliminates the cheaper placements one at a time -- steering beside the router, a
one-armed router on the flat/24, bridging on the LAN side, bridging on the WAN side -- to show
that per-device bidirectional accounting requires Shepherd to be the routing and translating
point, which is Gateway. Its central decision is that Shepherd should not write a datapath:
the kernel forwards and translates,nftablesis configured by Shepherd, and per-device and
per-destination bytes come fromnf_conntrackaccounting in both directions, joined with the DNS
engine's log for names. Verified against the live host, because both change what must be built:
conntrack accounting is off by default there and/proc/net/nf_conntrackis absent on
that kernel, so it must be enabled deliberately and read over ctnetlink. It also notes that naive
polling undercounts, since a flow can begin and end between two polls, so destroy events are
needed as well as periodic dumps.It records what the hardware cannot do yet (the NAS has one 10 Gb NIC; Gateway needs two), stages
the migration so the risky step is last and short and so each stage proves what the next depends
on, and is explicit that two of IPV6.md's hard rules forbid what a router must
do -- never run a Router Advertisement server, never change a client's IPv6 default gateway --
so they have to be rescoped to the Guard and Watch profiles rather than quietly broken, which is
how a Thread mesh gets broken. Five decisions were left to the maintainer, of which the first is
now answered: the WAN is DHCP, not PPPoE, removing the PPP session, the credentials and the
1492-byte MTU with its MSS clamping from the build. It adds one cutover risk worth having in
writing, since many ISPs tie the lease to the MAC that holds it: read the eero's WAN MAC first and
clone it onto Shepherd's WAN interface rather than hoping, which also makes the rollback
symmetric. Alongside it, a scope reduction nobody had noticed -- this LAN has no global IPv6
route today, being unique-local only, so the IPv6 requirement through stage 3 is "do not break
unique-local, Thread, Matter and mDNS" rather than "provide IPv6 internet", and the rescope of
hard rules 1 and 5 only truly bites once global IPv6 is wanted. That needs one fact nobody has
checked: whether the ISP delegates a prefix at all. The second is answered too: the eero keeps DHCP
until cutover, which leaves the live network alone until the day it has to change and moves a
risk rather than removing one. Shepherd must still serve leases from the moment of cutover, since
a bridged eero stops, so DHCP now gets its only real test on the parallel segment and then meets
the whole household at once. The staging is rewritten around that: the parallel stage must include
the difficult devices on purpose (a Thread or Matter device, a printer, a TV) rather than a
convenient laptop, and three things must happen before cutover day -- Shepherd takes the eero's
LAN address so every existing lease stays correct, the eero's lease time is shortened in advance so
the network converges in minutes rather than days, and the eero's WAN MAC is cloned. The third is answered as well:
fail closed, a considered departure from Guard mode's fail-open precedent. That turns out to be
the item least like what it sounds like, because the kernel forwards and Shepherd does not: if
Shepherd exits, the rules are still loaded and traffic keeps flowing, so doing nothing fails
open. A shutdown hook does not fix it either -- it covers a clean stop and missesSIGKILL, the
OOM killer, a panic, a destroyed container and a wedged process, which are the cases fail-closed
exists for. So the design inverts it into a dead man's switch: forwarding is permitted only
while an nftables set element with a short timeout is present, Shepherd refreshes it while
healthy, and the kernel expires it on its own schedule if Shepherd stops for any reason. One
exception is carved out, because failing closed otherwise takes out the means of repair along with
everything else: the switch governs forwarded traffic only, so devices are cut off as intended
while Shepherd's own outbound traffic, and therefore Tailscale and remote access, survives. The
fourth is answered too: Watch mode is retired. The remaining decision is whether the
eero stays as an access point. -
The relay loop has a mechanism, a measurement and a fix, and Watch mode runs on the
maintainer's NAS again. Three times the relay carried traffic round in a circle at line rate
and each time the cause was argued rather than measured, because the one thing nobody could see
was what the storm was made of: a frame the in-kernel relay bounces back out withXDP_TXnever
reaches anAF_PACKETsocket, so no packet capture on the host can show it. The relay now
samples every 256th frame it relays into a small ring the engine reads once a second, and
logs a digest of the busiest flows with each one's TTL range and how many distinct IP
identification fields it carried. That digest answered the question on the first run:74 02:00:5e:00:00:37 -> 02:00:5e:00:00:20 tcp 192.0.2.37->203.0.113.9 :50027->:443 ttl 64 1 ids, SAME IDs, SAME TTL: circulating at layer 2 avg 832 BOne IP identification field across thousands of frames, a TTL that never decremented, and the
device's own source MAC with Shepherd's destination MAC -- the same frame, going round, never
through a router. Repeated storms all looked like that.The fix is a duplicate brake in the relay itself. The program fingerprints each frame it
relays (the identification field, the lengths, both addresses, the protocol and the first
sixteen bytes of the transport header) and drops a frame it has already relayed within
100 ms. Dropped rather than passed, because IPv4 forwarding is on on many hosts and passing it
would hand the kernel a frame it would route straight back out of the same interface. The window
is deliberately shorter than any TCP retransmission timeout (Linux's minimum is 200 ms) and
vastly longer than a loop's round trip, which is microseconds, so a genuine retransmission is
never suppressed and a frame going round is caught the second time it is seen. A frame that
really is a network-level duplicate is dropped, which is what should happen to it: the first copy
already reached the gateway.Live result on the host that had never managed a minute: Watch mode steering 13 devices,
including the machine behind every storm, with no loop and no brake trip. The engine reports
the suppressed count (duplicates_suppressedonGET /api/watch), and on that host it runs at
about 80% of relayed frames -- so whatever re-delivers the frames is still doing it, and is now
contained instead of catastrophic. -
The loop brake records what it measured where a restart cannot lose it. Its log lines are
deliberately the evidence the first unexplained incident lacked -- what the interface was doing,
which devices were relaying most, and what the sampled frames looked like -- and twice now a
docker compose up --buildhas discarded the container and taken them with it, the second time
during this very investigation. The kill switch already survived a restart; its reason now does
too. A trip is written towatch.jsonaslast_loop_stop(when, the interface's receive and
transmit counts and how far apart they were, what the relay itself reported, how many frames
the duplicate brake had already suppressed, the busiest relaying devices and the sampled-flow
digest), and reported onGET /api/watch. The suppressed count is the one to read first: a trip
with a high figure is a loop the duplicate brake could not break, while a trip with zero means
the mirrored traffic was not duplicate frames at all and the cause is somewhere else entirely. -
What the investigation established, by counting rather than arguing. Three candidate
mechanisms are now excluded by direct measurement, and two of them had been asserted here
before: kernel IPv4 forwarding (one datagram during a 24,500-packet-a-second storm),
same-segment hairpin routing (zero packets dropped by a rule that counted them), and local
duplication inside the kernel -- the program's ownexaminedcount came to 0.972 of the
interface'srx_packetsover 30 seconds, so every duplicate frame genuinely crosses the wire
rather than being duplicated on the way to the hook. New counters record the two shapes of
Shepherd's own output returning to it (a frame with its source MAC, a frame addressed to the
gateway's); both read zero throughout every storm, so the relay's own transmitted frames are not
what comes back. -
"Steered and healthy" can mean almost none of a device's traffic, measured. A device the portal
reported as steered, healthy and with tens of thousands of relayed frames was given a 100 MB
download over the steered interface. The device's own per-interface counters confirm it went that
way (115 MB in, 26 MB out on the steered interface, under 12 KB on the other). On Shepherd's side
over the same download: 20,010 frames examined, 0 relayed, against the interface's own
rx_packetsof 18,454 -- so the frames never reached this host at all, and the device sent them
to the real router despite its neighbour table showing the gateway at Shepherd's MAC.Two consequences. A device's relayed byte count can be an arbitrary fraction of its traffic,
not merely outbound-only and understated: here the relayed frames were background chatter while
the bulk went past, and a number like that is worse than none because it looks like a measurement.
And "healthy" means "frames were relayed for this device", not "this device is steered" -- the
health loop answers correctly by its own definition, since traffic did move, but it cannot tell a
device whose traffic flows through the relay from one whose trickle does while the rest bypasses
it. Both written up in WATCH.md. -
A unicast ARP claim can still reach an interface Shepherd is not steering. Observed live, and
a limit of one-sided steering rather than a bug in it. Shepherd replies only to the steered
device's own MAC; a host with two interfaces on one segment may apply the answer to both. On a Mac
with a steered wired interface and an unsteered wireless one, the gateway's mapping changed to
Shepherd's MAC on both, although Shepherd never addressed the wireless one. Nothing is
relayed that should not be -- rule 3 only relays frames whose source MAC is actually steered -- but
the frames do arrive, and on a forwarding host they then meet the kernel's routing table instead
of the relay. So a per-device Watch switch is not a guarantee about a multi-homed device:
turning it off for one interface does not stop that device's other interface from being told the
same thing by its own operating system. Written up in WATCH.md. -
The echo is measured leaving the machine and coming back, which says it is the router. The
duplicate table already held the moment each frame first arrived, so the delay before its copy
returns was free to compute, and it is the discriminator nothing else supplied. First reading on
the live host:<10us: 9 <100us: 17,093 <1ms: 4,229 <10ms: 6 >10ms: 021,331 echoed frames, of which nine -- 0.04% -- came back fast enough to have stayed inside the
machine. A frame duplicated inside the host, in the kernel or the driver or on the way to the
XDP hook, returns in under ten microseconds. Eighty per cent of these returned in tens of
microseconds and the rest within a millisecond, which is a wired round trip and nothing else. So the frames genuinely leave the interface and the
segment hands the device's frame back to Shepherd's port, which agrees withexaminedbeing 0.972
of the interface'srx_packetsand with both return-path counters reading zero: what comes back is
the device's original frame, not Shepherd's rewritten one.
That is not something Shepherd can fix from this host, and
DESIGN.md anticipated it: "Firewalla documents ARP compatibility problems with
some mesh routers and does not list eero 7, so this must be tested per router before relying on
it." This is that test, and the answer on this router is that one-sided ARP steering can be
contained here but not made clean.EchoDelays.Verdict()states which shape points where, and
stays silent below fifty samples rather than guessing from a handful. -
The duplicate brake does not contain a runaway, and the live trip proves it. The brake was
measured holding the steady echo at about 80% of relayed frames, and that reading did not
generalise. A loop brake trip on the live host, with the evidence now persisted rather than lost
to a rebuild, recorded this:rx 27,532 tx 27,148 (1.39% apart) relay reported 26,251 pkt/s = 95.3% of received duplicates suppressed during the burst: about 2,400 of 87,516 frames relayed -- 3% 127 of 128 sampled frames: one flow, one IP ID, one TTL, "circulating at layer 2"So it was a genuine single-frame runaway, and the duplicate brake caught 3% of it rather than the
80% it manages at rest. Its effectiveness collapses as the rate rises, which is the opposite
of what a safety mechanism should do. The explanation offered was a race in the brake's
own shape:is_duplicatereads a slot, compares and writes it back without atomicity, so at tens
of thousands of frames a second across twelve cores two copies might both read "not seen" and both
be relayed. The table was duly made per CPU, removing that race outright, and it changed
nothing: the next trip suppressed 8,259 of 347,963 relayed frames, 2.4% against 2.8% before, and
again sampled a single flow with one IP identification field and an undecremented TTL. So why one
frame can be relayed hundreds of thousands of times while a fingerprint table holding it catches
two per cent of the passes is not known, and that is the fourth explanation offered here and
refuted by measurement, after the polluted registry record, kernel forwarding and the dual-homed
device. The duplicate brake helps at rest and does not work when it matters; the loop brake is
what stands between this feature and a melted network.Watch mode is off on the maintainer's NAS as a result, with
allow_routing_hostback to false
so the dual-homed refusal applies on its own, and the config records why. DNS filtering, the
device registry and the passive ARP monitor are untouched and verified working.What did work is the layer underneath: the loop brake stopped it in three seconds and, for the
first time, the reason survived the redeploy -- the interface counts, the relay's share, the
duplicate count at that moment, the devices and the sampled flows, all inwatch.json. Also for
the first time the trip carried the echo histogram, which still said a wired round trip. -
The dual-homed-device lead is refuted too, by the per-device counts. The machine behind the
worst storms is itself dual-homed on the segment, which looked like the answer; it is not.
Measured live: that machine runs 16,196 duplicates against 18,361 relayed (0.88), a
single-homed phone 419 against 692 (0.61), and a barely-active wired device 0 against 7. So
the echo is not peculiar to the dual-homed machine, nor to wired or wireless, and what it tracks
is how much a device relays rather than how it is attached. That is the third explanation
offered and refuted by measurement, after the polluted registry record and kernel forwarding, and
it points at the relay and the segment under load rather than at any device. docs/WATCH.md says
so where it previously recorded the dual-homing as the next thing to test. -
A relay self-test, before a single device is steered. On attach, Shepherd relays one frame of
its own -- a 60-byte UDP datagram inside 198.51.100.0/24 (TEST-NET-2) to the discard port, from a
locally administered MAC no real device can hold -- and then checks whether it comes back. The
program only ever sees frames the interface receives, and a frame this host transmits is not
one of them, so a healthy relay relays the probe zero times: it goes out, the switch declines to
send it back out the port it arrived on, and that is the end of it. A probe the program does
see is a frame that came back and was relayed again, which is a loop, and the relay is refused
(intercept.ErrRelayLoops) with nothing steered. A relay found to loop is recorded per
interface inwatch.json(looping_relays) and never attached there again, the way flapping
drivers already are, because running the test costs a brief burst of looping frames and the
answer does not change. The probe's frame is built in portable code and checked on every
platform -- 60 bytes so nothing pads it, a header that checksums to zero, both addresses inside
TEST-NET-2 so it goes nowhere if it escapes, and a deterministic result, since the verdict is
whether this exact frame was seen coming back; only the part that needs a loaded program and a
live interface is Linux-only. -
watch.allow_routing_hostruns Watch mode on a host that forwards IPv4 and has a second
interface on the relay's own segment, which Shepherd otherwise refuses. The refusal was only ever
correlation -- it is this host's shape, and this host is the only one that ever looped -- and the
mechanism it stood in for is now excluded by measurement. It stays the default because it is
cheap and the loop is severe, but it is no longer a dead end: the knob is how a dual-homed host
runs Watch mode, with the duplicate brake and the loop brake both armed.watch.diagnostics
logs the relayed-frame digest every few seconds while traffic is moving; addresses, ports, TTL
and identification fields only, never payload. -
A loop brake for Watch mode.
Twice on real hardware the relay ended up carrying its own output back to itself at line rate --
about 10 Gbit/s and over a million packets a second on a 10 GbE card -- and both times it ran for
minutes because a person had to notice. The first had a cause and it is fixed (it steered one of
the host's own interfaces); the second is still not explained, which is what the brake is for.
The brake reads the interface's kernel counters every second and looks for two things: it receives
and transmits the same amount, within 15%, at 15,000 packets a second or more. Three consecutive
seconds of that and it panic-stops, unsteering everything with its corrective reply, and latches
like the kill switch so a restart cannot resume the loop. It logs what it measured, the relay's
own share of it, and the devices that were relaying most, because that per-device list is
exactly the evidence the second incident lacked.
Symmetry is what separates looping from busy, and both thresholds come from measurement. The live
host's own background, nothing steered, runs 89 to 92% apart with max(rx, tx) between 12,000 and
40,000 a second; the loops measured 0.03% apart at 1.0 to 1.3M, 9.3% apart at about 27,000, and
10.1% apart at 517,000. So 15% leaves wide margin either way. The floor is emphatically not the
discriminator -- the host's ordinary background is above it and is excluded by symmetry alone --
its only job is to stop a quiet network tripping, so it sits between that host's transmit
background (about 3,000) and the smallest loop measured (about 27,000).
The relay's own share is deliberately not part of the test, and an earlier version of this brake
required it. That version asked the relay to report at least 80% of the interface's receive rate
and at least the packet floor itself. The reason for dropping it is not that the relay carries
little of a loop -- it carries most of one -- but that its counter lags the storm by seconds,
and those are the seconds that matter. Sampled every two seconds from the moment steering began,
one device gave examined/relayed of 17,952/0, then 62,756/27,587, then 89,336/45,516, then
102,516/73,704 at 90 MB/s. The share is 0% at two seconds and passes 80% only after the card has
been at 90 MB/s for several seconds; a separate run had the interface already mirrored at 27,000
against 24,500 at 3.5 seconds while the relay still reported 8 pkt/s. Gating on the share buys
nothing and costs the only seconds in which stopping is cheap. It is still logged as evidence, and
a low figure now means the brake caught the storm early rather than that the relay is innocent.
A limit no version of this brake escapes, recorded so it is not rediscovered as a bug: mirrored
receive and transmit is inherent to a relay that bounces frames back out the same interface, since
each relayed frame is counted once arriving and once leaving. WithRrelayed packets a second the
asymmetry is|bg_rx - bg_tx| / (max(bg_rx, bg_tx) + R), which on the measured host is
27,000 / (30,000 + R)and falls under 15% onceRreaches about 150,000 a second (~1.8 Gbit/s).
A single fast wired client pulling that much through the relay would trip the brake while working
correctly; the old share test would have been satisfied there too (83%), so this is a property of
judging a relay by interface counters rather than a regression. Separating "working hard" from
"looping" robustly needs duplicate detection, a TTL check, or relayed bytes against what the device
could actually receive. The symmetry threshold moved from 10% to
15% in the same change, because the 10.1%-apart loop was excluded by six hundredths of a percent,
and a threshold that only just misses a real incident is the wrong threshold. Accepted cost: any
genuinely symmetric load above the floor sustained for three seconds now stops steering.
The floor came down from 50,000 to 15,000 in the same change, because 50,000 was above the loop
it was meant to catch. That number came from a sampling mistake worth recording: theatlantic
driver publishes/sys/class/net/<if>/statisticscounters in one lump roughly once a second, so
sampling every 100 ms reads0 0 0 0 0 24073 0 0 0 0, and a half-second bucket that catches a
lump holds a whole second of packets. Doubling such a bucket overstates the rate twofold, which is
how the instrumented loop was first written up as 54,000 packets a second instead of 27,000, and
how the floor ended up above it. The brake itself samples once a second and was never affected;
only the threshold chosen from the bad figure was.Ip:InReceivesin/proc/net/snmpupdates
continuously and can be sampled freely; the interface counters cannot. The regression test for
this uses the measured rates against the production floor and fails if the floor rises again.
Fixed
smb1-enabledfired on servers that have SMBv1 disabled. The probe treated any SMBv1-framed reply
to its Negotiate Protocol request as proof of SMBv1, but a server with SMBv1 turned off (Samba with
server min protocol = SMB2, which is what most NAS appliances ship) still answers in SMBv1 framing —
with a DialectIndex of0xFFFF, the protocol's own way of saying "none of your dialects". A NAS whose
SMB settings offered only SMB2/2.1/3 was listed as critical on every address it had. The probe now reads
the reply's status and DialectIndex and only reports SMBv1 when the server actually selected the
dialect; the refusal, a bare header, and an error status all stay unflagged.- Shepherd will not steer a device record that carries one of this host's addresses, or the
gateway's. The registry on the live host had merged the gateway, both LAN interfaces and all
nine container bridge addresses into a single device record -- the ARP monitor learning
Shepherd's own steering frames as though they were a device's -- and that record was eligible to
steer. The policy now refuses any record carrying a host or gateway address by MAC and by
address, across every address on the record rather than its primary one, so a merged record, a
randomised MAC or an address learned from Shepherd's own frames is caught even when the MAC looks
like an ordinary device's. - Traffic from a steered device to any of this host's other addresses reached the gateway instead
of this host. The relay's "not for Shepherd itself" rule was given only the relay interface's
own addresses, so on a host with a second interface on the same segment, or with container
bridges, a frame addressed to one of those was relayed out to the router rather than passed up to
the stack. The maintainer's NAS answers DNS on such an address, so this is the ordinary case. The
rule now covers every address of the host, for IPv4 and IPv6; the gateway's own address stays out
of it, because that traffic belongs to the gateway and passing it up would strand it.
Changed
-
The refusal to relay on a host that routes the relay's own segment stays, but on correlation
only: forwarding has been excluded as the mechanism. It went in as a precaution, was then
explained twice, and both explanations have now been refuted by measurement on the live host.
What is established: steering one ordinary phone, named by explicit MAC with the registry and
the gateway out of the picture, loops every time. An instrumented run reached about 27,000
packets a second in and 24,500 out, mirrored, 3.5 seconds after steering began, and counters
sampled in that same second excluded three candidates outright —ForwDatagramsmoved by
1 datagram (not kernel IPv4 forwarding), a rule counting hairpin forwarding in and out the
relay interface dropped 0 packets (not same-segment routing), and the relay's own counter
reported 8 pkt/s. The explanation previously recorded here — the router learning the client at
Shepherd's MAC and the kernel routing its downloads back onto the segment — is withdrawn rather
than softened; one forwarded datagram during that storm is not a routing problem.
The relay reading was mis-read as a third exclusion, and it is not one. A later run let the
same storm develop instead of stopping it at 3.5 seconds, and the relay's own counter went 0, then
27,587, then 45,516, then 73,704 pkt/s at 90 MB/s by eight seconds in — 72% of what the program
examined, still rising when stopped. The 8 pkt/s was the first instant of a ramp. So the relay is
the leading candidate, not an excluded one, and the same run says why: with the XDP program off
the device was steered for the same duration withexaminedholding at 16,000 to 28,000 pkt/s and
no storm at all -- but that is weaker than it looked, because-xdp offdoes not mean "no relay":
the trial then relays in userspace across four workers and managed only 88 pkt/s and 107 kB/s, so
"no storm without XDP" may mean only "a userspace relay is too slow to make one". The fast path is
a suspect, not a conclusion. Two mechanisms are excluded, not three. One apparent clue is not one: examined 102,516 against 73,704 relayed leaves ~29,000 frames
a second seen and not relayed with both rule-4 counters at zero, but the program bumpsexamined
for every frame it sees and bumps nothing on theXDP_PASSpaths for "not Shepherd's MAC" and
"not from a steered device", so that gap is the rest of the segment's traffic -- and ~29,000/s is
this host's measured background.
What survives is the other half of the condition, and it is the one structural feature common to
every incident: the host has two interfaces on one segment, and the steered device is
REACHABLEthrough both at once. That makes the IPv4-forwarding half of the test look like it is
doing no work; dropping it would make the refusal narrower and more honest, but not before a
mechanism is actually identified, because the last two named were both wrong.
Two earlier claims on this branch, for the record. An interim commit blamed the merged registry
record and relaxed this refusal to a warning; a single-device trial disproved it. Two five-second
samples were read as a steady 3,900 to 4,900 forwarded datagrams a second; both had landed inside
bursts, and a minute of one-second samples gives a median of 34, a p90 of 5,759, a peak of 8,607
and 80,521 in all. That burstiness also explains the first trial's unexplained pre-steering
baseline of 12,000 packets a second inbound. Those bursts are the relay interface's inbound
traffic routed to a container bridge for a service the owner runs on purpose, so clearing
net.ipv4.ip_forwardis not available as a remedy and would not help if it were.
Two earlier five-second samples both landed inside bursts and were written up as a steady 3,900 to
4,900 a second; the distribution above supersedes that. The same burstiness also explains the
first trial's unexplained pre-steering baseline of 12,000 packets a second inbound. -
Watch mode no longer tries to steer devices that are not on the relay interface's subnet. The
host's own Docker containers arrive in the registry looking like ordinary LAN devices: a172.x
address learned from the host's neighbour table and several MACs each. An ARP reply on the relay's
interface cannot reach them, so steering them was meaningless; five of them were in the steerable
set on the live host. -
DNS keeps serving when one listen address goes away, and can always be reached over loopback.
Both halves of a real outage. Shepherd was configured to listen on three addresses; the wired
interface holding one of them lost carrier, and becausednsengine.Serverequired every bind to
succeed, Shepherd refused to start — so a network whose resolver was fine on two other addresses
had none at all, in a restart loop.Servenow takesTolerateMissingAddresses, which
cmd/shepherdpasses: an address this host does not have is a loud warning naming it, startup
carries on with the rest, and binding nothing at all is still fatal. The tolerance is deliberately
narrow — an address another process already holds stays a hard failure, because then something
else is answering on it and starting anyway would leave the owner believing Shepherd was handling
their DNS when a different resolver was;TestRunFailsFastOnBusyPortsholds that line, and it
caught exactly this when a first, broader version of the change made it hang. An address is never
half-served (if its TCP half fails the UDP half is closed too), and without the option the strict
behaviour is unchanged. Second half: new
dns.listen_loopback(on by default) also serves 127.0.0.1 and[::1]on the port of the
first configured address, because when Shepherd is its own host's resolver, loopback is the one
address that cannot vanish with an interface — in the outage the host lost name resolution
entirely, which wedgedsshd(it blocks on reverse lookups) and left the machine unreachable to
fix the very interface that had failed. Addresses already covered are not duplicated and a
wildcard listener is left alone. Because Shepherd adds these itself rather than the owner asking
for them, they are best-effort in a way the configured addresses are not (dnsengine.Optional):
any bind failure on one is a logged warning, the port already being taken included, which on a
NAS whose system resolver owns127.0.0.1:53is the normal case and not a fault. Without that
distinction this option would have refused to start on exactly the hosts it was written for, and
on a default install, since it is on by default. An address indns.listenis still strict: one
another process holds stays fatal there. It only helps if the host's own resolver lists 127.0.0.1,
which the example config now says. -
Watch mode is wired in and on by default (
[watch],internal/watch, docs/WATCH.md) — ROADMAP.md
step 11, after the throughput ceiling that held it back for months was root-caused and fixed and the relay was
measured at full gigabit on real hardware, with the maintainer's explicit go-ahead. Until now
internal/interceptwas a library nothing called except a hand-run CLI. Nowcmd/shepherdruns one engine for the
life of the process: it finds the interface from the default route, the gateway's MAC from the kernel's neighbour
table, and Shepherd's own MAC and addresses from the interface; attaches the relay once (in-kernel XDP in the
NIC driver, else XDP generic, else the parallel userspace relay — a native attach rebuilds the driver's rings for
about eight seconds, so attaching per device would stall traffic every time a phone woke up) and from then on only
toggles the steered set. Every eligible device is steered: never the gateway, never Shepherd's own interfaces,
never an excluded device, never network or smart-home infrastructure (router/mesh/AP/switch/hub types, anything
advertising Thread or Matter over mDNS, vendors that only make network gear), and never a device not seen for 15
minutes. Wandering and quarantined devices are steered like any other, since Watch mode is visibility and blocks
nothing. Every device gets a three-way Watch switch (automatic/always/never) in the registry, the portal, bulk
actions and search (watch:off), and the engine re-steers by MAC on an address change. Safety: the engine owns a
kill switch that panic-stops every device at once and persists across restarts until re-armed
(<data_dir>/watch.json, the only thing it saves; a Shepherd with Watch mode switched off reports
itself armed, since nothing is holding steering down — reporting otherwise would read as a kill
switch somebody pressed); the interception library's health loop still fails a device
open on silence, now with a widening retry backoff so a failing device is not flapped; a relay that dies unsteers
everything and the engine restarts on a backoff; the passive ARP monitor'sgateway-mac-changedevent makes the
engine re-read the kernel's neighbour table (the event's own MAC is never trusted) and restart with the new MAC.
GRO is turned off on the interface for the userspace relay's lifetime and restored on shutdown, done with the
ETHTOOL_SGROioctl rather than by shelling out toethtool, which the container image does not have. A host that
cannot run any of this — not Linux, missingCAP_NET_RAW/CAP_NET_ADMIN/CAP_BPF, no usable interface — logs one
clear reason, reports itself unavailable in the portal, retries quietly and changes nothing else; DNS filtering is
never affected. New Watch tab (js/watchview.js, pure logic injs/watch.jswith its own tests) with the
relay in effect, live relayed bits and packets per second overall and per device, every unwatched device's reason,
the recent "stopped watching" log and the kill switch; a Watch column and per-device panel on the Devices
screen;GET /api/watch,POST /api/watch/kill,POST /api/watch/rearm,watch/watch_stateon every device,
and aset_watchbulk op. Demo mode synthesizes the whole thing, policy and live rates included. New config:
watch.enabled(true),interface,mode(auto/xdp/userspace/off),workers,
steer_infrastructure,active_window_secondsand the health/grace knobs.NET_ADMINandBPFadded alongside
NET_RAWin both compose files. -
IPv6 Watch-mode steering (
watch.ipv6,internal/ndp, docs/IPV6-WATCH.md), off by default. The IPv6 mirror of
the IPv4 mechanism, and the narrowest shape that could work: unsolicited unicast Neighbor Advertisements sent
only to a device being steered, claiming the router's link-local address for Shepherd's MAC, with a solicited reply
to that device's own revalidation probe and a corrective advertisement on unsteer and panic-stop. Shepherd never
sends Router Advertisements under any setting — docs/IPV6.md hard rule 1 stands unchanged, and rule 6 now records
this unicast-only, per-device exception to it. The relay (userspace and XDP both) relays IPv6 only for a device
explicitly steered for IPv6, and the Controller keeps a master switch, a required gateway link-local address and a
separate, stricter IPv6 excluder so a smart-home hub, Thread border router or Matter device can never be
IPv6-steered — a refusal the per-device "always watch" switch cannot override, unlike for IPv4. The engine engages
it only where the network can carry it: an IPv6 default route whose next hop is link-local, a global (2000::/3)
address on the interface, and an IPv6 address on the device (its link-local preferred). With unique-local addresses
only — the maintainer's own network, and a common case — there is no off-link IPv6 traffic to steer, so it reports
itself idle with that reason instead of pretending. A device merged from several MACs stays IPv4-only, because
which of its MACs holds its IPv6 address cannot be known from the IPv4 neighbour table. An IPv6 refusal never costs
a device its IPv4 steering. Never run against a live global-IPv6 network: the design's own recommendation was
to wait for one, and the maintainer does not have one. -
Secrets managed in the portal (
internal/secrets): the two credentials Shepherd cannot generate for
itself — the DNS provider token ACME DNS-01 needs, and the accountability SMTP relay — no longer mean
hand-editingshepherd.tomland restarting. Both live in<data_dir>/secrets.json, the same JSON, atomic
save (temp file, fsync, rename), mode 0600, nothing-written-until-there-is-something shape as
reverseproxy.jsonanddevices.json, and the same honest security model as every other secret this
project persists: 0600 is the whole protection, no encryption at rest. Both are write-only through the
API (only "set or not", where it came from and when it changed ever come back; the types also redact
themselves under%vso a stray log line cannot leak one), and both are checked before they are saved
with a real, side-effect-free call, so a bad credential is refused now (422, nothing stored) instead of
surfacing hours later as a failed renewal or a digest that never went out: the Cloudflare token with the
read-only zone lookup issuance already makes (tlscert.CloudflareProvider.Verify, never a record write),
for the names[tls]and[reverseproxy]will actually issue for; the relay with a real connect,
TLS/STARTTLS and AUTH that quits beforeMAIL FROM(accountability.SMTPSender.Verify, nothing is ever
sent). One DNS provider token, shared:[tls]and[reverseproxy]each had their own Cloudflare token
fields for what is one credential on one zone; now both read the same portal-managed entry, set from the
Proxy tab, and the record carries aproviderfield (onlycloudflareexists) so another provider can
be added without a format change. The SMTP relay (host, port, username, password, From, TLS mode) is set
from the Accountability tab. Neither needs a restart: certificate issuance reads the token fresh on
every attempt, and a save kicks each certificate manager to check at once (tlscert.Manager.Kick), so a
first certificate that was waiting on a token arrives within seconds; the accountability engine now sends
through anaccountability.RelaySendera newly saved relay is swapped into live. The engine is also now
built whenever[accountability]is enabled (the default), relay or not, so the Accountability tab no
longer 404s on every call when no relay is set — the confusing "looks broken" state it showed before —
and a saved relay's password is only ever reused for the same host and username (a changed host must have
its password typed again, so a saved one can never be redirected to another server). A default install
still pays nothing on the DNS hot path for it: a newdnsengine.Engine.OnAccessWantedgate skips even the
item lookup while no relay is set, and no access is queued or persisted without one.
Upgrading keeps whatshepherd.tomlalready has:[tls]/[reverseproxy]'s
cloudflare_api_token/cloudflare_api_token_fileand the whole[accountability.smtp]section
(password_fileincluded) become read-once seeds, the[[lists]]precedent applied per secret — imported
the first time Shepherd starts without a portal-managed value, and ignored from then on (a startup line says
so when the file's value differs). If both sections hold different tokens at import,[tls]'s wins and the
conflict is logged by setting name; a seed that cannot be read yet (a missing token file) is logged and
retried at the next start rather than lost; clearing a secret in the portal is remembered, so the file never
brings it back. Neither token field is required by[tls]/[reverseproxy]any more; enabling either with
no token anywhere starts normally and says in the log (and the Proxy tab) that a token is needed. New API:
GET/PUT/DELETE /api/dns-providerandGET/PUT/DELETE /api/accountability/smtp. Not done: encryption at
rest, other DNS providers, a "send a test email" button, and turning[accountability],[tls]or
[reverseproxy]themselves on or off from the portal (those switches, and hostnames, are still config). -
Passkey unlock (
internal/passkeys): the portal can now be unlocked with a passkey — Face ID, Touch
ID, Windows Hello, a phone, or a hardware security key — alongside the token and the PIN, never instead
of them. The step docs/LOCKING.md had designed and deliberately deferred until the portal had a real
HTTPS name, which[tls]now gives it. The WebAuthn protocol isgithub.com/go-webauthn/webauthn
v0.18.2, not hand-rolled — the one deliberate new dependency of tonight's work, because challenge,
origin and RP ID binding, attestation and COSE key parsing and signature checks are exactly what a
well-vetted library is for; it brings in CBOR, TPM-attestation, JWT and msgpack modules
(docs/THIRD-PARTY.md). The relying party is[tls].hostname, its originhttps://<hostname>:<port of tls.listen>(plus the same name on 443, for a container publishing 443 onto the listener); a passkey
registered there works only there, never at the LAN IP, another name, or plain HTTP; each passkey records
the hostname it was made for, so after a[tls].hostnamechange the old ones stay listed (flagged, to be
removed) but are never offered at sign-in. Every ceremony is
refused before it starts when it could not work: with[tls]off the Locking screen says passkeys need
it, and from any page that is not the HTTPS listener at that name (the plain-HTTP LAN address, or the
HTTPS listener reached by IP) it names both where you are and where to go, instead of a WebAuthn error in
the console. Passkeys can only be added from an already-unlocked session (token or PIN), never used to
gain first access; registration and login require user verification (a biometric or device PIN, since a
passkey stands in for a PIN plus a trusted browser), ask for no attestation, prefer but do not require
discoverable credentials so hardware keys without resident-key storage still work, and refuse a
signature counter that goes backwards (a possible clone) without saving it (and never lower the stored counter
when two sign-ins race). Ceremonies are single use
(a finish attempt consumes one, right or wrong), expire after 5 minutes, are tied to the browser that
began them by anHttpOnly/Secure/SameSite=Strictcookie holding only a random id (the challenge
stays on the server), and are capped at 32 pending. A successful passkey login opens exactly the session
a token or PIN login does — the sameshepherd_session, idle lock and 30-day cap — and revoking that
passkey ends the sessions it opened. Up to 20 passkeys, each named, renamable and revocable from a new
Passkeys card on the Locking screen; the lock screen gains Sign in with a passkey, shown only on
the HTTPS listener and only once one is registered (the unauthenticated check reveals neither the
hostname nor how many exist). Stored in<data_dir>/passkeys.json(0600, atomic; public keys only — a
passkey's private key never leaves its authenticator), wiped when the portal token rotates like the PIN
and trusted browsers; a file that cannot be read disables passkeys alone (503, file untouched) while the
token and PIN keep working. New API:GET /api/login/passkey,POST /api/login/passkey/begin|finish,
GET /api/passkeys,POST /api/passkeys/register/begin|finish,PATCH/DELETE /api/passkeys/{id}.
Tested against a software authenticator, not a real one:internal/passkeys/passkeystestbuilds and
signs real ES256 WebAuthn payloads the way the library tests itself, and every tampered response (wrong
origin, localhost, wrong RP ID, wrong challenge or ceremony type, no user verification, someone else's
user handle, a wrong key, a replay, a revoked or unknown credential, a backwards counter) is checked to
fail the specific check it targets, through the real HTTP API over a real TLS listener. No real browser,
platform authenticator or security key has been tried yet; see docs/PASSKEYS.md, including its "Before
you rely on this" checklist. -
2D map fallback for the HUD: browsers without WebGL, notably Safari in Lockdown Mode, now get a
flat canvas map (js/flatmap.js) with the same arcs, comets, pulses, picking, Set home and panning and
zooming, plus day/night shading, instead of a "needs WebGL" notice.?nogl=1forces it. -
Reverse proxy (
[reverseproxy],internal/reverseproxy): the main remaining piece of
ROADMAP.md step 18 ("Access") — Shepherd can now do what a separate reverse-proxy
manager does: own an HTTPS listener, terminate TLS for every configured hostname, and forward each
request by itsHostheader to that host'shttp://orhttps://backend, path, query, body and
headers untouched, withX-Forwarded-For/X-Real-IPset to the client's own address (set, never
appended to a client-supplied chain: this is the edge),X-Forwarded-Proto/X-Forwarded-Host, and the
publicHostpassed through the way such managers do. A name with no enabled host gets a plain 404 from
the proxy itself, never the portal; a host pointed back at the proxy is answered 508 instead of looping.
Forwarding ishttputil.ReverseProxy, and WebSocket upgrades are proven end to end, not assumed:
internal/reverseproxy/websocket_test.gohand-rolls RFC 6455 on both sides with only the standard
library (zero new module dependencies) — a hijacking backend that pushes a message first and echoes, and
a raw TLS client negotiatinghttp/1.1like a browser — and checks the 101 handshake, the unprompted
backend-to-client frame, echoes, a 70 KB frame each way and the close handshake, for bothhttpand
httpsbackends;cmd/shepherd's tests repeat it through the real listener started byrun().
One wildcard certificate (*.<domain>) covers every host, issued over the same Cloudflare DNS-01
flow as[tls](now shared asstartCertManager;[tls]behaves exactly as before) but kept in its
own<data_dir>/reverseproxy/tls/;tlscert.Obtainerwas already wildcard-capable and a test now
proves the CSR and challenge naming. The same order also asks for the bare base domain itself
(Obtainer.ExtraHostnames, a general "more names on the same certificate" seam): a pure wildcard never
coversexample.comon its own, only*.example.com, which the first live cutover hit directly (a
host at the bare apex domain). A plain-HTTP listener (redirect_listen, off by default likelisten
itself never defaulting to:443— binding a port below 1024 needs a capability a fresh install does
not have) answers every request with a 302 to the same host and path over https when set (e.g.:80),
so a bookmarked or hardcodedhttp://link still gets there instead of failing to connect once
nothing else owns port 80.
Backend certificates are verified by default, with an explicit
per-host "accept any backend certificate" switch for self-signed backends (a typical manager never
verified them at all). Hosts live in<data_dir>/reverseproxy.json(0600), managed only from the
portal — never seeded fromshepherd.toml, nothing written until the first edit, editable while the
listener is off so everything can be lined up first — overGET/POST /api/reverseproxy/hosts,
PUT/DELETE /api/reverseproxy/hosts/{id}andGET /api/reverseproxy(listener and certificate state),
and a new Proxy tab (js/reverseproxyview.js, pure logic injs/reverseproxy.jswith its own
tests) that badges any name the wildcard cannot cover and any backend left unverified. New config:
[reverseproxy]'senabled(false by default),listen(:8443by default, never 443: moving
the standard port over is the owner's deliberate cutover),domain,emailand
cloudflare_api_token/cloudflare_api_token_file(same rules as[tls]), andredirect_listen
(empty/off by default). A taken listen port (either one) stops startup with a clear error; obtaining the
certificate never blocks it. Not yet: per-host access control of any kind, custom per-host config, an
access log, or import from an existing manager — a straight migration can lose per-host features a
manager had. Every automated test runs against fakes and loopback, but this has also now been run for
real: a real wildcard certificate issued against a live Cloudflare zone, a dozen real services imported
from an existing reverse-proxy manager and verified one by one (including a real WebSocket handshake
against a live backend), and a real cutover of port 443 itself. See docs/REVERSEPROXY.md, including its
"Before you point real traffic at this" checklist. -
Watch mode: in-kernel XDP relay (
internal/intercept.Controller.AttachXDP, Linux). The same
one-sided ARP relay as an eBPF/XDP program on the interface: a steered device's IPv4 frame to
Shepherd's MAC gets both MACs rewritten and is bounced straight back out (XDP_TX) without ever
reaching the kernel stack or userspace; everything else, including the device's traffic for
Shepherd itself, is passed through untouched. The Controller stays the only authority on what is
steered; the program only consults a map kept in step with it. Native (driver) mode with a
generic-mode fallback.cmd/intercept-trial -xdp autouses it in place ofParallelRelay. The
program is compiled bygo generate(needs a clang with the BPF backend); the generated objects
are committed sogo buildneeds nothing new. Verified against a real kernel's verifier and
BPF_PROG_TEST_RUNand end to end on a veth pair (make test-xdp), then live on the NAS in native
mode: 180k frames/s and ~1.77 Gbit/s relayed during a phone's speed test, healthy throughout. See
docs/INTERCEPT.md, "Beyond userspace". -
Marketing site (
site/): a public, vanilla-HTML/CSS/JS landing page — hero, the real shipped feature
set (DNS filtering and the 95-list catalog, devices, the globe HUD, alarms, port and vulnerability
scanning, geo-block, portal locking, self-healing upstreams and more), an honest "what's coming" section
that calls out Watch mode's current trial-only state by name, and a Raspberry Pi section. The
centerpiece issite/demo/: the portal's real compiled frontend (internal/portal/web/), forced
permanently into its own built-in demo mode for a static host with zero backend — the real globe HUD and
Devices/Blocking/Alarms screens, populated with synthetic data generated in the browser, not screenshots
or a video.site/demo/js/demomgmt.jsadditionally gives the Alarms screen synthetic data (the shipped
demo fake never covered/api/alarms; alarmsview.js already shows a clean "not available" card for that
case, so this is additive, local to the site build, and does not touch
internal/portal/web/js/demomgmt.js, the filecmd/shepherdactually embeds). Mirrored to
blaineam.github.io/apps/shepherd/by the portfolio repo'smirror-app-docs.yml.
New.github/workflows/notify-portfolio.yml, modeled on Haven's, dispatches that mirror on every push to
site/**(needs aPORTFOLIO_DISPATCH_TOKENrepo secret to actually fire; skips cleanly without one, same
as Haven's). -
Alarm archiving: the Alarms view had no way to dismiss a handled alarm from the default list short
of resolving it (which means "this is fixed," a different claim) or waiting for retention's age/count
pruning to remove it outright. A newArchived bool/ArchivedAt time.Timeonalarms.Alarm,
independent ofState(an alarm in any state, including New, can be archived without claiming anything
was fixed), plumbed through as one moreFilterdimension (IncludeArchived, excluded by default the
same way storm-bundle members needIncludeChildren) and two new batch verbs,Engine.Archive/
Unarchive, following the exactselectLocked/mutate/emitBatch/undo-handle pattern every other batch
verb already uses. An escalating repeat (a dedupe fold with a higher severity) clearsArchived, since a
bigger version of a dismissed problem deserves to resurface.GET /api/alarmsgetsinclude_archived
(off by default — the default view is actually quieter now, not just capable of being asked to be);
POST /api/alarms/bulkgetsop: "archive"|"unarchive"(a single alarm archives through this too,
{"ids":["id"],"op":"archive"}, rather than a dedicated route);GET /api/alarms/summaryexcludes
archived alarms from every count. The Alarms tab gets a "Show archived" toggle, a bulk Archive/Unarchive
button, and a per-row archive action next to acknowledge/resolve/mute/delete. See docs/ALARMS.md
("Archiving") for the Option A vs. Option B design reasoning. -
Real TLS for the portal (
[tls],internal/tlscert): the first, additive step of
ROADMAP.md step 18 ("Access") — the portal can now get a real Let's Encrypt
certificate for its own hostname, issued over ACME's DNS-01 challenge against Cloudflare DNS, the
same mechanism the maintainer's existing reverse-proxy manager already uses for other services on
the same Cloudflare zone. The ACME protocol itself (nonces, account keys, order/challenge polling,
certificate parsing) is handled bygolang.org/x/crypto/acme— already a transitive part of this
project's existinggolang.org/x/cryptodependency, so this adds zero new module dependencies
(see docs/TLS.md for why that was chosen overgo-acme/lego).internal/tlscertsupplies the one
piece that library does not (creating/removing the DNS-01 challenge's TXT record through
Cloudflare's API,CloudflareProvider) plus the scheduling, persistence and hot-swap wiring:
Managerrenews 30 days before the certificate's real 90-day expiry, checking every 12 hours
without ever touching the network while the current certificate is still comfortably valid, and
never blockscmd/shepherd's startup — the same "never block on something outside Shepherd's
control" conventionstartARPandstartGeoBlockalready follow. Wired intocmd/shepherd
(cmd/shepherd/tls.go'sstartTLS) as a second, additive HTTPS listener (tls.listen) alongside
the existing plain-HTTP one onportal.listen, which is completely unchanged;[tls].enabled = false(the default) leaves Shepherd behaving exactly as it did before this landed. New config:
[tls]'senabled,hostname,listen,email, andcloudflare_api_token/
cloudflare_api_token_file(mutually exclusive, following[accountability.smtp]'s own
password/password_fileconvention exactly). The certificate, its key, and the ACME account key
are persisted under<data_dir>/tls/, mode 0600 like every other secret this project persists.
Every test (internal/tlscert's own, andcmd/shepherd's wiring tests, including an end-to-end
test that makes a real HTTPS request against the running portal) runs against fakes — a fake
Cloudflare API and a fake ACME client implementing the exact interface*acme.Clientdoes — never
a real network call to Cloudflare or Let's Encrypt, staging or otherwise. See docs/TLS.md,
including the honest note on what a real live first issuance (a real Cloudflare account and
token, done separately and deliberately) would still need to confirm. -
docs/WATCH-MODE-COMPARISON.md: a row-by-row, honest comparison of
what the DNS-only baseline already covers versus what Watch mode (one-sided ARP interception, still
a hand-run trial library, not wired intocmd/shepherdor the portal) would add once it ships —
written for the maintainer to point to when deciding how to present the two, not aspirational copy. -
Portal UX overhaul: a dedicated Vulnerabilities page, mobile reflow, and several maintainer-reported
rough edges.internal/vulnscan's full API had no page of its own, only a panel inside each device's
drawer; a new Vulnerabilities tab (internal/portal/web/js/vulnscanview.js) lists every finding across
every scanned device, worst severity first, filterable, with a "Scan all" action, and clicking a row
opens that device's drawer rather than duplicating the finding detail a second time — the drawer panel
stays as the quick per-device summary. The Devices screen gained a "Needs attention" panel above the
main list (internal/portal/web/js/attention.js): newly quarantined, wandering and excluded devices
pulled to the top instead of mixed into 91 otherwise near-identical rows. Mobile: the Alarms table (a
fixed 7-column grid with a >700px minimum width) now reflows into stacked cards below 760px, the same
technique Devices and History already used; the HUD's Legend panel no longer force-displays on mobile,
only behind a small "i" toggle; the top nav collapses behind a hamburger button into a vertical dropdown
once its tabs stop fitting in one row; pinch-to-zoom is pinned off and every text input kept at an
effective 16px so iOS Safari never force-zooms on focus; a duplicate "Wandering" badge next to a
device's name in the Devices list was removed (the Mode column's own badge, which carries the
countdown, already said it). The HUD inspector now shows "Blocked by" for a blocked query
(dnsengine.Entry.Reason), and a blocked History row offers a one-click, domain-specific "Allow"
(POST /api/rules, the same mechanism the HUD inspector's own block/allow buttons use). The Details
view now states plainly that Watch mode is a planned feature, not available in this build yet, instead
of leaving its absence to read as a bug (confirmed unwired:cmd/shepherdnever imports
internal/intercept; see docs/ROADMAP.md step 11). See docs/PORTAL.md, docs/VULNSCAN.md,
docs/DEVICES.md and docs/HISTORY.md. -
WAN exposure for open ports (
internal/upnpigd):internal/portscannow tells you not just that a
port is open on the LAN, but whether your router also forwards it from the internet — the far more
dangerous case for the same open port. A new, read-only UPnP IGD client discovers the router over SSDP,
finds its WANIPConnection/WANPPPConnection control URL, and enumerates its NAT port-mapping table (never
calling AddPortMapping or DeletePortMapping), cached for 10 minutes by default so a slower, more fragile
router round trip never runs on every scan. Every openPortRecordgainsExposure:"lan_only",
"wan_forwarded"or"unknown"—"unknown", never a guess, whenever the router does not cooperate
(UPnP IGD disabled, or not implemented the same way by some consumer mesh systems), and the field is left
out entirely when the new[portscan.wan_exposure]config is disabled. Enabled by default, alongside
[portscan]itself (see docs/PORTSCAN.md's "Should this be on by default?" for the reasoning). The
portal'sGET /api/portscan*endpoints and each device's Ports panel now surface it, with a small "WAN"
badge on a forwarded port's chip. See docs/PORTSCAN.md's new "WAN exposure" section for the full
limits — in particular, that this is best-effort and will readunknownfor every port on any router
that does not answer UPnP IGD discovery at all. -
Network health history (
internal/nethealth): latency, jitter, packet loss and reachability,
probed every 2 minutes once enabled against the configured DNS upstreams plus1.1.1.1/8.8.8.8, and an
occasional, tightly bounded bufferbloat check (idle vs. loaded latency against a bounded download
burst). Both the probe and bufferbloat default off — unlike[portscan]/[vulnscan], which only
ever contact devices already on your own LAN, this probes fixed public internet addresses on a timer,
which is a real behavioral difference worth an owner's explicit opt-in rather than inheriting
portscan/vulnscan's default-on reasoning (see docs/NETHEALTH.md for the full reasoning, including a
concrete hermetic-testing problem the default-on version of this caused in this project's own test
suite before the default was flipped). Unlikeinternal/portscan/internal/vulnscan's
gentle, mostly-stateless checks, this keeps real history: raw samples for 3 days, hourly rollups for 60
days, daily rollups indefinitely (capped at 8 MiB), so months of trend data stay available without disk
usage growing without bound. Latency is timed as a plain, unprivileged TCP connect to port 53 rather
than an ICMP raw socket (noCAP_NET_RAW, fully unit-testable with an injected dial function, the same
seaminternal/portscan.Scanneruses) — see docs/NETHEALTH.md for the full reasoning. A new
alarms.TypeNetworkDegradedalarm raises after 3 consecutive bad probe cycles (not a single blip — the
same "don't cry wolf" disciplinewireUpstreamAlarmsalready applies to upstream DNS health) and
resolves on the next healthy sample. Wired intocmd/shepherd([nethealth], both off by default) and
the portal (GET /api/nethealth,GET /api/nethealth/series,
GET /api/nethealth/bufferbloat,POST /api/nethealth/probe,POST /api/nethealth/bufferbloat/run),
with a new top-level Network tab (current status plus latency/jitter/loss time-series charts). The
default gateway is added to the probe targets automatically when it can be determined (reusing
internal/arp's/proc/net/routehelper). Traceroute is deferred, for the same reason
internal/portscandeferred router port-forwarding discovery: it needs real router/path verification
this project's test environment cannot provide. See docs/NETHEALTH.md's "What is verified, and what is
not" section for exactly what was and was not checked against a real network. -
Vulnerability scanning (
internal/vulnscan): a real vulnerability-detection layer on top of
internal/portscan's open-port findings — not "port 23 is open" but "Telnet is open" or "this OpenSSH
build is missing a specific named fix." A small, curated, built-in signature table (no external CVE API,
matching this project's no-paid-API convention) covers SSH protocol 1 and outdated OpenSSH, unencrypted/
anonymous FTP and the vsftpd 2.3.4 backdoor, outdated Apache/nginx/lighttpd banners, an unauthenticated
Docker API and SMBv1 still being enabled. Every signature needs a gentle, protocol-appropriate probe to
actually confirm exposure (a banner read, one HTTP request, one well-known anonymous FTP login, one raw
SMBv1 negotiate packet) through the same injectable-dial-function seaminternal/portscan.Scanner
already uses, so tests touch zero real sockets. Wired intocmd/shepherd([vulnscan], enabled by
default, needs[portscan]enabled too — see docs/VULNSCAN.md for why), a newalarms.TypeVulnerabilityFound
alarm type (raised once per newly-found vulnerability per device, never suppressed by learning mode —
this feature's whole point is telling the owner about exposure found on day one; a finding that clears
resolves the alarm rather than leaving it to linger), and the portal (GET /api/vulnscan,GET /api/vulnscan/devices[/{id}],POST /api/vulnscan/scan), with a Vulnerabilities panel in each device's
drawer (reusing the Ports panel's layout and a scan-now button) showing active findings with a severity
badge, description and evidence, plus a collapsed "N resolved" history. See docs/VULNSCAN.md. -
internal/vulnscan: TLS probing and a UPnP-exposure signature, six new signatures in all. A real
TLS handshake (over an injectableTLSConnectFuncseam, the TLS analog ofDialFunc, so tests still
touch zero real sockets or certificates) now runs against 443, 8443, 465, 993, 995 and 2376 — offering
every protocol version down to TLS 1.0 and every cipher suite Go's owncrypto/tlsimplements,
including the ones its standard library marks insecure, so a weak server reveals itself by what it
actually negotiates. New signatures:tls-weak-protocol(TLS 1.0/1.1 negotiated),tls-weak-cipher- suite(RC4, 3DES, or a padding-oracle-prone CBC suite negotiated),tls-cert-expiredandtls-cert- not-yet-valid(the certificate's own dates, nothing else), anddocker-tls-unauthenticated-api(2375's
own check, now also over Docker's TLS port). Certificate verification itself is skipped on purpose
(InsecureSkipVerify): a self-signed or hostname-mismatched certificate is routine on a home LAN and
deliberately never a finding — only a certificate's dates are judged. 443 and 8443 additionally get the
same plaintext-shaped HTTP probe as the other HTTP ports, now over the decrypted channel, so the
existing Apache/nginx/lighttpd signatures (and the newupnp-exposedone, below) fire over HTTPS too
with no new signature code. Also new:upnp-exposed(medium) flags aServer:header naming UPnP on
any HTTP port already probed — a deliberately narrow, banner-only check, not SSDP/IGD discovery (a
separate, lower-level feature). Along the way, 465/993/995 were reclassified frombannerPorts(reading
a plaintext greeting) to the new TLS path, fixing a latent, harmless-but-wrong read of raw TLS handshake
bytes as a banner on those three implicit-TLS ports. Threat feeds (roadmap step 23) stay explicitly out
of scope — a different, larger feature that needs to hook into traffic observation, not a per-port
probe. See docs/VULNSCAN.md for the full signature table and the self-signed/hostname-mismatch judgment
call. -
Import wired into the portal: the Devices screen's new "Import" modal drives
internal/importers
end to end — CSV upload or eero login (phone/email, then the code eero sends, then a network), a
dry-run preview of what would be created or changed, and an apply step, matching the Devices tab's
look. New endpoints under/api/import/eero/{start,verify,preview,apply,cancel}and
/api/import/csv/{preview,apply}. The multi-step eero login needs a session between requests; it is
kept server-side, in memory only, under a randomsession_idthe browser holds instead of eero's own
token, and is discarded on apply, on cancel, or automatically 15 minutes after it was last touched —
it is never written to disk. See docs/IMPORT.md. -
ARP monitoring wired into
cmd/shepherd: new[arp]config section (enabled,interface, auto-detected from the default route when empty);cmd/shepherd/arp.go'sstartARPopens the real Linux socket, runs the monitor for the life of the process, and feeds the device registry, the same lifecycle as the classifier and device sources.arp.Events go through a plainfunc(arp.Event)callback that, for now, just logs each one — nothing downstream consumes them yet (an alarm engine wiring will supply a real callback later). ARP monitoring never blocks startup: off by default, on the wrong platform, or withoutCAP_NET_RAW, it logs a clear warning and Shepherd starts normally.GET /api/arp/statsexposes the monitor's counters ({"enabled":false}when it is not running). See docs/ARP.md. -
docs/RASPBERRY-PI.md: Shepherd already cross-compiles to
linux/arm64in CI, so a Raspberry Pi 3 or newer (or similar cheap ARM64 hardware) is a real, documented place to run DNS-only protection and the passive ARP monitor today, with an honest note that Watch mode's software relay needs more headroom than the baseline does. -
Alarms wired into
cmd/shepherdand the portal: the alarm engine (previously a library only) now
runs for real.[alarms]config (shepherd.example.toml) turns it on and configures ntfy, the one
delivery channel wired in ([alarms.ntfy]: server, topic, token or basic auth). A new, minimal
dnsengine.Pool.OnHealthChangehook, in the same plain-callback style asEngine.Observe, raises
upstream-downwhen every configured upstream has gone unhealthy (a true total outage, not one flaky
resolver) andupstream-recoveredper resolver as it comes back. Portal API under/api/alarms: list
with filters, get one, acknowledge/resolve/delete (single and batch via aSelector), mute (single
snooze or batch by alarm/type/device/domain/key), undo, a summary and mute-rule CRUD. A new Alarms tab
matches the Devices tab's structure: severity/state filters, search, single and bulk actions with a
bulkbar and undo toasts, and a Rules sub-view. Web Push is explicitly deferred, not forgotten: it
needs a secure context (HTTPS), which the portal does not have yet; wiring it up today would be
unreachable code. See docs/ALARMS.md ("Wiring intocmd/shepherdand the portal", "Not built yet"). -
Ports and exposure (
internal/portscan): scheduled, gentle TCP connect scans of every known device's
common ports (roughly 40, chosen for home-network relevance, not completeness), with per-port history
(open/closed, first/last seen), flags for a newly-opened port and for a short built-in list of
specifically risky ports (Telnet, RDP, SMB, VNC, rpcbind, NetBIOS, unauthenticated Docker, PPTP, FTP)
whenever they are found open at all. Plain connect scans only — no OS/service fingerprinting, no UDP, no
stealth scan types — with bounded concurrency and a short per-connection timeout, fully unit-testable
through an injectable dial function with zero real sockets in tests. Wired intocmd/shepherd
([portscan], enabled by default — see docs/PORTSCAN.md for why) and the portal (GET /api/portscan,
GET /api/portscan/devices[/{id}],POST /api/portscan/scanfor a manual scan of one or every device),
with a Ports panel in each device's drawer showing its open ports, risky and newly-opened flags, last
scan time and a "Scan now" button. Router port-forwarding discovery (what the internet can reach) is
explicitly not built: there is no reliable, generic router API for it. -
Allowlist mode and scheduled access (
internal/dnsengine,internal/portal): two additions to the policy engine, built together since they touch the same evaluation path. Every scope (network, group or device) now has a Mode,"block"(the existing, unchanged default) or"allow"; effective Mode resolves the same most-specific-wins way as everything else (device, else group, else network). In allow mode, a scope's switched-on catalog or custom item is its allowlist instead of its blocklist — the same picker, just inverted — and a name nothing decides is blocked by default instead of allowed; built-in policy switches always block in both modes, and there are deliberately no automatic infrastructure exceptions (NTP, OS checks, captive portal), so switching to allow mode can break basic device functionality until you explicitly allow what it needs. The existing block-vs-allow precedence (an allow at the same or a more specific scope beats a block from a less specific one; a block at a more specific scope beats an allow from a less specific one) is unchanged and reused exactly for allow mode, not reinvented. Separately, any allow or block rule can now carry an optional schedule (days of week plus a start/end minute-of-day, server-local time, no midnight-spanning in v1 — use two schedules) that makes the rule apply only inside that window; outside it the rule is inert, as if unconfigured, and evaluation falls through normally.POST /api/policy/modesets or clears a scope's Mode;POST /api/policy/rulestakes an optionalschedule;GET /api/policyreports both per scope. The Blocking screen gains a Blocklist/Allowlist segmented toggle at the scope selector and a day-of-week/time schedule picker on the rule-add form, with a schedule badge on existing rules; the demo-data mock API supports both. See docs/POLICY.md's "Allowlist mode" and "Scheduled access". -
Portal locking: the portal shows synthetic demo data by default until it is unlocked with the token or a PIN, reusing the existing
?demo=1code path for the locked view itself. Signing in creates a random, in-memory, server-side session (shepherd_session,HttpOnly/SameSite=Strict, never carrying the token); it idle-locks afterportal.idle_lock_minutes(default 10, 0 = never) of real activity — the frontend's own background status/query polling and the live stream deliberately do not count as activity, or idle locking would never trigger while a tab merely sat open — and never outlives 30 days or a server restart regardless. A PIN, set from the new Locking settings screen after the first token unlock, only works paired with a random per-browser trusted-device cookie (up to 20 registered at once); PINs are 6-12 letters/digits, rejecting obvious ones, hashed with Argon2id at the same costinternal/historyalready uses for passphrases; 5 wrong attempts in a row lock a browser out for 15 minutes with growing delays, and attempts during the lockout still count toward an outright revoke at 10. The Locking screen lists and revokes trusted browsers and sets the idle timeout or turns it off; rotating the portal token (detected by its hash no longer matching across a restart) wipes the PIN and every trusted browser. Passkeys (WebAuthn) are designed for but explicitly not built here — they need HTTPS, which the portal does not have yet (a separate roadmap item) — see docs/LOCKING.md. -
Country blocking (geo-block): a Firewalla-style, per-scope (network, group or device) list of ISO country codes whose DNS answers get blocked, built as an additive check layered outside the Filter's own rule evaluation — it runs only after a query would otherwise be allowed, reusing the engine's already-geolocated answer destinations and the existing sinkhole/
block_modemechanism, the same hook shape Apple Private Relay blocking and the classifier'sObservealready use (Engine.CountryBlock). State persists to its owngeoblock.json, separate frompolicy.json. Depends on[map]'s GeoIP database; degrades silently (never blocks, never errors, logs once) when the map is off or not loaded yet. Blocked queries showreason: "geo"andgeo_countryin the live stream and history. Portal API (GET/POST /api/geoblock) and a Countries tab on the Blocking screen's existing scope picker. See docs/GEOBLOCK.md for the honest DNS-answer-only limitation (same caveat the rest of the DNS-based filtering carries). -
cmd/intercept-trial: a standalone, Linux-only, hand-run CLI for a one-off controlled trial of Watch mode's ARP interception engine against exactly one named device, for a mandatory, capped trial duration, with Ctrl-C wired to a panic stop; disables GRO on the interface for the trial and always restores it on exit (a GRO-coalesced frame can be larger than the wire allows, which the health loop correctly fails open on but which is a capture artifact, not a real fault), needingCAP_NET_ADMINalongsideCAP_NET_RAW. Not part ofcmd/shepherd; not run automatically. Run clean against a disposable VM and, with the maintainer's explicit go-ahead, against the maintainer's own phone (confirming the maintainer's eero 7 Max accepts one-sided ARP steering). See docs/INTERCEPT.md. -
Controller.ParallelRelayandNewRelay(internal/intercept): a real, measured throughput problem — a single capture-and-relay goroutine cut a steered device's throughput to roughly a third in a live test — addressed by splitting capture across severalPACKET_FANOUTworkers (OpenManyfor independent per-worker senders,NewRelayfor additionalRelays sharing oneController's state), each on its own goroutine.cmd/intercept-trialuses it by default (-workers). See docs/INTERCEPT.md's "Throughput" section — this alone did not fix the slowdown. -
rawSender's kernel-levelSO_SNDTIMEOand theselfBoundedfast path (internal/intercept): the real packet-rate cost turned out to besendWithTimeout's per-packet goroutine andtime.Aftertimer, there to bound aSender.Sendthat might hang — unnecessary once the real socket enforces its own send timeout at the kernel level.sendWithTimeoutnow skips its defensive wrapper entirely for a Sender that reports itselfselfBounded, taking a direct call (still panic-recovered) instead;FakeSenderdoes not report it, so every existing wedged/panicking-Sender safety test is unchanged. Root cause found by live-testing each hypothesis in turn (parallelism, GRO, then this) rather than assuming the first one. See docs/INTERCEPT.md. -
Accountability-partner email alerts:
internal/accountabilitylets someone invite a trusted contact to be emailed when a device in a group they chose accesses a category they chose — a self-hosted, free equivalent of Covenant Eyes or Accountable2You, built for someone who wants this visibility shared with a person they trust. Summary detail (no domain names) and digest delivery (daily by default) are the defaults; detailed and immediate are opt-in, and even immediate mode coalesces a burst into one email. Matching is decoupled from blocking: an "any" trigger (the default) is disclosure, not enforcement, so an access that was never blocked is reported neutrally, not as a violation. Contacts and subscriptions persist as atomic JSON (mode 0600); email is composed from two reviewabletext/templatestrings and sent through a minimal, standard-library-only SMTP client (STARTTLS/implicit TLS, bring-your-own relay, no paid API) with classified send outcomes. Now wired up end to end:dnsengine.Engine.OnAccessfires for every completed query, blocked or allowed, that matches a watched catalog or custom list item (unlikeObserve, which exists for the automatic classifier and skips blocked queries on purpose);cmd/shepherd'sstartAccountabilityresolves each access's device and groups from the device registry and feedsEngine.Recordthrough a small worker queue so a subscriber's digest write never holds up a DNS response; a new[accountability]/[accountability.smtp]config section (enabled by default, inert until an SMTP relay is actually configured, with apasswordorpassword_filechoice for the SMTP secret); and a portal/api/accountability/...API plus an Accountability screen for managing contacts and subscriptions (scope, watched items, detail, mode, trigger). See docs/ACCOUNTABILITY.md, which no longer lists any of this as future work. -
PACKET_IGNORE_OUTGOINGon every capture socket,Controller.Stats(), and arate:log line incmd/intercept-trial(internal/intercept): a code audit of the capture/relay/spoof path, prompted by a live packet capture showing an unexplained retransmission storm, found no frame duplication anywhere in this package (Relay.HandleFrame's destination-MAC filter is correct in every case checked) but did find a real, avoidable cost: withoutPACKET_IGNORE_OUTGOING, the non-promiscuous capture socket also receives a loopback copy of every frame Shepherd itself just transmitted, which is then parsed and discarded — a second parse-and-discard for every relayed frame, at the same hot path already under suspicion. Capture sockets now set it (Linux 4.20+; ignored, not fatal, on an older kernel).Controller.Stats()and the newSteeredDevice.PacketsRelayedgivecmd/intercept-trialenough to log packets/bytes-per-second and the self-capture rate every 2 seconds, so the next live trial does not need a fresh packet capture to see this. Now measured live (a 2-minute trial, real phone, real eero, with the maintainer's explicit go-ahead): still 300–400 Mbps while steered, the same range as every earlier measurement — this fix, like the two before it, is real but did not close the gap. Therate:line watched live for the first time narrows the search:relayedstayed in the single digits through two ~20-second bursts that spikedexaminedto thousands of packets/sec, which rules out the self-capture loopback (already removed by this fix) and weakens the pure ACK-storm theory, pointing instead at the eero's own bridging behavior during the ARP change as the more likely remaining cause. See docs/INTERCEPT.md's "Throughput" section for the full, updated writeup and what to test live next. -
Device quarantine: a newly discovered device lands in a reserved Quarantine group automatically and stays there until it is moved to a real group (or explicitly released), so nothing new gets the run of the network before you have looked at it. Quarantine is a group like any other for policy, so its rules are set the same way (Blocking → A group → Quarantine). It cannot be deleted;
devices.quarantine_new_devices = falseturns off automatic assignment. Shepherd cannot stop a quarantined device from reaching other LAN devices directly (that needs your router's guest network); see docs/DEVICES.md. -
Shepherd core (Wave 2): one native binary with an embedded DNS filtering resolver and web portal. It cross-compiles, with no CGO, to Linux, macOS and Windows.
-
DNS engine: blocklists in hosts, plain-domain and adblock formats; allow and block rules that match subdomains, with allow winning over every block source; a TTL cache; UDP and TCP serving with truncation handled correctly; both IPv4 and IPv6 answers.
-
Upstream health: continuous probing with uncached names, automatic failover, fastest-healthy-first ordering, and TCP and TLS upstreams. Learned from a real outage where an encrypted upstream was reset on the network and stalled all DNS.
-
List guardrails: a list update is rejected if it is empty, blocks a protected domain, or changes size drastically, and the previous list stays active. Lists are cached so filtering works offline at startup.
-
Apple iCloud Private Relay blocking: a built-in policy answers NXDOMAIN for the relay hostnames, as Apple documents for networks, switchable from the portal. Relay attempts are logged by device.
-
Portal: live query list with per-row allow and block, upstream and list health, rules, policy switches, top devices and top blocked domains. Token authentication, strict cookies, a CSRF header on every change, and a strict content security policy.
-
Configuration: a single TOML file (
shepherd.example.toml), validated at startup with clear errors. -
Public-development safety:
scripts/leakcheckkeeps private addresses, MAC addresses, tailnet names, keys, personal paths and email addresses out of commits and commit messages (git hook and CI), with a private.leakcheck.localfor your own identifiers. -
Documentation travels with code:
scripts/doccheckfails a commit that changes code without changing docs. -
Testing: 97% statement coverage with the race detector, and a coverage floor in CI that only goes up.
-
Releases: rolling beta builds and tagged stable releases through GitHub Actions, SemVer throughout.
-
Wave 1 Docker testbed (AdGuard Home, Tailscale, Uptime Kuma),
shepherd-doctorpreflight (hardened against a real UGOS host) and a guide for migrating an existing Tailscale container. -
Design, roadmap, IPv6/Thread rules, Contributor Covenant code of conduct, security policy, contributing guide and issue templates.
-
Icon: the black sheep, reused from the Lamdi app.
-
Optional Docker packaging in
deploy/docker/(Dockerfile and compose file) that runs the same static binary under a restart policy with a read-only filesystem and minimal capabilities, plus docs for supervising Shepherd and cutting over from another resolver. -
Encrypted history (opt-in): a searchable record of past queries on disk, sealed with AES-256-GCM in immutable, individually keyed segments with an encrypted index, retention by age and size, tamper detection, and a key file or passphrase (Argon2id) unlock. In-memory only remains the default. See docs/HISTORY.md.
-
Live event stream and history API:
GET /api/stream(Server-Sent Events with resume),GET /api/history,POST /api/history/unlock. -
Geolocated destinations: query entries carry the places their answers point to, using a free GeoIP database (DB-IP Lite, CC BY 4.0) downloaded monthly, validated before use and kept if an update fails.
-
Automatic home: the globe's home point is found from the network's public address with a DNS "who am I" query and placed with the local database; only the place is stored. Choose your own home and it always wins, with reset and re-detect. Turn off with
map.auto_home = false. -
Settings API (
GET /api/settings,POST/DELETE /api/settings/home,POST /api/settings/home/detect). -
[history]and[map]configuration sections, including a database mirror URL. -
Roadmap and design for the reverse proxy with wildcard certificates and tunnels, port and exposure management, and inbound monitoring and protection.
-
The globe HUD is the portal's new home view: a dot-matrix 3D globe with traffic arcs from home to every destination, weighted so heavy lanes are visibly thicker and brighter, color coded by allowed, cached, blocked and error, with blocked queries pulsing a shield at home. Click an arc to inspect it and block or allow the domain with undo, filter by device, action or domain, set home by clicking the globe, and replay any retained period with a timeline scrubber and speed control. It runs from the live stream, falls back to polling, works offline (the globe library and land outlines are bundled), respects reduced motion and falls back to the Details view without WebGL.
?demo=1shows synthetic data with no API calls. The earlier dashboard lives on in the Details tab. -
Front-end unit tests (94,
node --test), run in CI. -
docs/PORTAL.mdanddocs/THIRD-PARTY.md. -
Blocking policy with a catalog (steps 4 to 6 wired in): a catalog of 95 lists, apps and categories (Instagram, Discord, YouTube, Steam, Social, Gaming, Adult, Gambling and more) you can switch on for the whole network, a group of devices or one device, plus custom lists from a URL or pasted text. Blocks add up across the scopes that apply to a device; an allow at a more specific scope beats a block from a less specific one. See docs/POLICY.md. Endpoints:
/api/catalog,/api/policyand/api/policy/items|rules|switches|custom-lists|refresh. -
Devices that have names: a registry of the machines on the network, learned passively from DNS clients, the neighbor table (ARP and NDP), mDNS announcements, reverse DNS (asked of your router) and a built-in MAC vendor table. Display names follow alias, then self-announced name, then reverse DNS, then vendor and type, then the IP address, and appear in the live stream, the query ring and the encrypted history (
deviceanddevice_idon every entry). Search with filters (group:,vendor:,type:,ip:,seen:,tag:,excluded:,unnamed:), groups, bulk edit and merge. Saved to<data_dir>/devices.json(0600). New[devices]config section; see docs/DEVICES.md. Endpoints:/api/devices,/api/devices/{id},/api/devices/bulk,/api/devices/merge,/api/groups. -
Configurable upstreams: plain, TCP, DNS over TLS and DNS over HTTPS upstreams (by IP or name), bootstrap resolvers (
dns.bootstrap), verified presets, test before adding, add, remove and reorder in the portal. Runtime edits persist to<data_dir>/upstreams.jsonand win overdns.upstreamsonce it exists. See docs/UPSTREAMS.md. Endpoints under/api/upstreams. -
/api/statusreportsdevice_count,group_count,catalog_count,custom_list_countandnetwork_item_count. -
Excluded devices are answered without blocking (log reason
excluded); in Watch mode they will also never be steered through Shepherd. -
Catalog list licenses and data sources in docs/THIRD-PARTY.md.
-
Wandering mode: a per-device mode that stops all filtering while still watching every request. Optional timer (15 minutes to 30 days, or until turned off) that the DNS engine honours at query time, a
shadowfield recording what filtering would have blocked,wanderingflags in the stream and history, awandering:truesearch filter, and a bulkset_wanderingoperation. It is distinct from Excluded, which leaves a device alone. -
Automatic device classification: devices are guessed as a phone, camera, TV, speaker and more from their DNS lookups (the strongest signal), MAC vendor, and, when
[map]is enabled, the networks they talk to (ASN, from a second free DB-IP database downloaded and guarded the same way as the city one). A type you set yourself always wins and is never overwritten. Guesses appear in the existingtype/type_reasonfields, plus newtype_autoandtype_confidence. New[classify]config section. See docs/CLASSIFY.md. -
Passive ARP monitoring (
internal/arp): a read-only listener that learns real MAC addresses for the device registry and raises events for a new MAC, an IP moving to a different MAC, and a changed gateway MAC (the classic spoofing signature). It never transmits anything. See docs/ARP.md. Deployed incmd/shepherd(above); not yet wired into the alarm engine. -
Watch mode's ARP interception engine (library, not yet wired up):
internal/interceptimplements one-sided ARP steering of a chosen device on top ofinternal/arp's codec — aSpooferthat claims the gateway's MAC and reverts on stop, aRelaythat rewrites and retransmits a steered device's outbound frames, and aControllerthat is the only thing allowed to start or stop steering: it enforces exclusions (the gateway, anExcluder-backed device list) at the call, verifies real relayed traffic on a health loop and fails a device open the instant something looks wrong, supports a trial mode with an automatic revert after a fixed duration, and has aPanicStopthat unsteers everything immediately even if the sender is wedged. IPv4 only; IPv6 is never touched, proven with a fuzz test. Outbound-direction visibility only — the gateway's own ARP table is never touched, so inbound traffic bypasses Shepherd by design. See docs/INTERCEPT.md. Nothing calls this automatically yet; wiring a trial UI or API, and running it against a real network, needs the maintainer's explicit go-ahead. -
Router importers (
internal/importers): a one-time, optional import of device names and groups from eero's unofficial API or a generic CSV, with a dry-run preview before anything changes. See docs/IMPORT.md. Not yet wired into the portal. -
Alarm engine (
internal/alarms) with Web Push (internal/webpush) and ntfy (internal/ntfy) delivery: severities, dedupe, storm control, mute rules and quiet hours, batch actions, and a digest for low-severity alarms. See docs/ALARMS.md. Not yet wired into Shepherd or the portal. -
Design and roadmap entries for device quarantine (a reserved group new devices land in until moved), portal locking (demo data until unlocked, configurable auto-lock, PIN, passkeys), and running Shepherd as a dual-NIC router someday (a stated, unscheduled future direction).
Fixed
- The sidebar's close button no longer shows on desktop, where it did nothing: the generic
icon-button rule came later in the stylesheet and overrode the one hiding it. - Watch mode no longer attaches native XDP on a driver that drops the link doing it, and learns
the ones it is not told about. A native attach makes the driver tear down and rebuild its
receive rings; on the maintainer's 10G Aquantia card (atlantic) that takes the interface down
and up. It cost the network its wired DNS listener, because the address vanished in the seconds
between the attach and the bind, and twice left the interface needing to be brought up by hand.
Shepherd now reads the interface's driver from sysfs and attaches generic XDP instead for any
driver on a known-bad list, whichatlanticseeds: still in the kernel, still no frame copied to
userspace, and it does not reinitialise the interface. For every other driver it samples the
kernel's owncarrier_changeseither side of the attach, and if the link went down and up it
says so plainly, records the driver in<data_dir>/watch.jsonand uses generic from the next
start — so an unlisted bad driver costs one flap ever rather than one per restart.mode = "xdp"
still means XDP only; it just means generic XDP where native would break the link. Also fixed in
passing:KillandRearmwrote a fresh state struct, which would have erased that learned list,
so the engine now keeps the whole persisted state and saves it whole. mode = "userspace"turned out not to be the safe fallback on that card either, which is
worth recording because the previous note recommended it. The software relay has to turn GRO off,
and with GRO off every frame reaches the stack separately instead of coalesced; at a few hundred
megabits theatlanticdriver reset repeatedly, takingcarrier_changesfrom 1 to 9 in about
twelve minutes with no XDP program attached at any point. docs/WATCH.md now says so, and says
that generic XDP is the one path on this hardware that avoids both failure modes.make checkdid not run the portal front-end tests, so a broken JavaScript file passed it.
The target's comment said "Everything CI runs, locally" and it ran gofmt, vet, the Go suite with
the coverage floor and both leak guards — but notnode --test internal/portal/web/test/, which
CI runs as its own step. A merge resolution had leftdemomgmt.test.jsone});short, so the
whole file failed to parse and every test in it was silently not running, on this branch and on
main.make checkreported green twice over it. The missing brace is back, the file's 18 test
blocks balance, all 298 front-end tests run and pass, andchecknow runs them in the same
place CI does so the local command means what its comment claims.- Watch mode steered the host's own other interfaces, which made the relay carry Shepherd's
traffic back to itself at line rate.watch.Engine.ownMACstreated "this host" as the MAC of
the interface the relay was pinned to, plus the gateway's. It never asked the operating system
what else belonged to the machine. On a host whose interfaces share a subnet — a NAS with wired
and wireless both on the LAN, the ordinary case — the other interface looks exactly like a LAN
device: its own MAC, its own address, seen by the passive ARP monitor, typed by the classifier.
Steering it closed a loop, and because the relay runs in the kernel at the driver it closed it at
line rate. Measured on the maintainer's NAS: 9 Gbit/s and about 1.05 million packets per second
in each direction on a 10 GbE link, sustained, with load average above 20. Two things hid it:
Shepherd's own CPU stayed under 4%, since the work was in the kernel, so every resource reading
looked excellent; and the portal showed no meaningful traffic, which was correct and was the real
clue, because the traffic was not the network's. NewPlatform.LocalAddrsenumerates every MAC
and every address on the host, memoised for 15 seconds so an interface appearing mid-session is
excluded promptly rather than at the next restart, and the policy'sownargument became a
watch.Ownholding both. A device is now refused if any of its MACs or any of its addresses
belongs to the host or the gateway — the address check catches the same machine arriving under a
MAC the registry has not tied to it, which is what the live registry actually held, since the ARP
monitor had learned Shepherd's own steering frames and merged the wireless MAC with the gateway's
address into one "smart-home hub" record. If the host's addresses cannot be read at all, nothing
is steered and the log says why: refusing costs visibility, steering the host costs the network.
internal/interceptis untouched. Two of the four new tests fail without the fix. - Watch mode treated an idle device as a failing one, which on a real network meant constant ARP
churn and a ten-minute blind spot. The interception library's health loop cannot tell a device
with nothing to relay from one whose steering never took, so it fails both open — correct, since
silence might mean a device cut off. But the engine counted every one of those fail-opens as a
failure, and the retry backoff doubles from 30 seconds to a 10-minute cap. On the maintainer's
network, 84 of 109 devices are active in any 15-minute window and most are idle at any moment,
so each of them cycled steer → 15 seconds of silence → unsteer → re-steer, flipping its ARP entry
to Shepherd and back for traffic that did not exist, while a phone that genuinely woke up could
go unwatched for up to ten minutes. The mismatch was between two windows that were never
reconciled: the policy keeps steering a device for 15 minutes after it was last seen, while the
health loop drops it after 15 seconds of silence. The engine now tells idleness from breakage
using its own two pieces of evidence, without touchinginternal/intercept: frames really were
relayed for that MAC at least once (steering provably worked, so the silence is the device), or
nothing other than the relay has seen it for a minute (it is asleep). Either way it is not a
failure, the pause is flat rather than widening, and the device is not steered again until the
registry sees it later than it did when it went quiet — which the passive ARP monitor and DNS
supply within seconds of a real wake-up, so the blind spot goes from ten minutes to about
fifteen seconds. A device something has seen lately that never relayed a frame is still a
failure, still widens and still retries on the timer, because that is steering not taking and is
exactly what the backoff is for. The evidence is deliberately cleared only when the unsteer is
reported and never by the sampler: the Controller drops a device fromStatus()before it says
why, and a sample landing in that window would erase the evidence just as it is needed. - The globe's per-frame comet-arrival check scanned all 600 comet slots every frame, regardless of
how many were actually animating — cheap enough with a JIT to go unnoticed, real cost without one
(Safari Lockdown Mode disables it entirely, and the portal felt sluggish there as a direct result).
GlobeRenderernow tracks which comet slots have anything left to do (liveComets) and only ever
iterates those, so the per-frame cost scales with what is actually on screen instead of a fixed cap
that is almost always mostly inert between spawns.
The notes were cut to fit GitHub's limit. The full list is in CHANGELOG.md.