Repository navigation
v0.11.0
A new face and a hardening release. Hora gets its brand (the night watch,
a little owl that dozes while all is well) and a status page redesigned
around it; the findings of a full audit are fixed; and the page stays
instant on huge databases: measured on 575 million checks across 700
monitors, the status page answers in 1.5 ms at p99 (it took 13 s in
0.10.0), weighs 71 KB instead of 6.5 MB, and the daemon peaks at 353 MB of
RAM instead of 887 MB. See UPGRADES.md for what to
check before upgrading.
Added
-
A redesigned status page. The hour and one sentence ("Everything is
running."), count chips, then what needs attention first; groups that are
all well fold away, large groups show compact rows, and each monitor's
daily bar is one image instead of ninety elements. Every state reads as a
colour, a shape and a word, never colour alone. Light and dark follow the
system,?theme=light|darkforces one (handy for embeds and kiosks).
Down monitors say since when ("Down · 38 min"), rows show uptime over the
bar's days, maintenance days are marked, and recent incidents and a
"No incident in N days" chip sit under the groups. -
/monitor/{id}: a page per monitor with its figures, the 24-hour
chart, a day-by-day bar whose details open by keyboard or tap, the
hour-by-day latency heatmap (moved here from/history), what each
vantage point sees, and its incidents. -
/watchers(operator only): the mesh from this node, its peers'
heartbeats, how often outages were confirmed, and each shared service as
seen from each place. -
"Not an outage." A down seen from this node only, while every peer
reaches the target, shows as up with a notice explaining why nobody was
paged. The JSON API is unchanged. -
alerts.notify_unconfirmed: the channels that receive a local-only
down (see Changed), validated like anynotifyroute. The webhook payload
gainslocal_only(bool) ondownandrecoveredevents. -
A certificate that cannot be read alerts (
cert_unreadable: TLS
certificate could not be read, with the error), once per streak and not
while the monitor is down. It used to be a log line, and the expiry of a
server whose STARTTLS dialogue or TLS versions changed silently stopped
being watched. -
The HTTP probe decodes
gzip,br,deflateandzstdbodies
before the assertions run (a CDN that always compresses failed every
keyword check).max_body_kbcaps the decoded size. -
Kiosk refresh is opt-in:
?refresh=<secs>(10 to 3600) reloads the
page; without it the page no longer reloads itself every 30 seconds,
which closed open groups and disturbed screen readers. -
hora compactgives the file's free pages back to the disk: applies
the retention now (--purge-removedskips the 7-day grace), rewrites the
file withVACUUM INTOand swaps it in atomically.--dry-runmeasures
every table and index and the gain. It needs the daemon stopped: the
daemon now holds<db>.lock, and CLI commands hold<db>.writers.lock
so none of them writes into a file being replaced.hora topshows the
database size and suggests a compaction past a fifth of free pages; the
operator/api/summaryreportsstorage { db_bytes, reclaimable_bytes }. -
Add to home screen: a web app manifest and icons, so a phone can keep
the status page as an app (no service worker: always the live page). -
The monthly report gets a day strip per service and shorter headings, and
is cached per month and audience. -
History, timeline, post-mortem, the empty and private-page states and the
plain-text (curl) output share the new design and wording. -
The documentation site wears the brand, with new pages: When Hora fits,
Concepts (how a failed check becomes one alert, or none), The status page,
Brand, Development, and the changelog. -
/api/announcetakesuntilas a time of day (18:00, UTC, the next
occurrence) as well as a duration (4h), likehora announce; the hint
inhora topis now true. -
Silence a watched peer:
POST /api/silenceandhora silenceaccept a
peer'slisten_id, and ad-hoc silences now mute peer-watch alerts too. -
Degraded alerts say why: a push with
status=degraded&msg=...or an
exec plugin's WARNING output is carried into the alert (and into the
webhook'smessagefield, now set ondegradedevents). -
hora_monitor_status{id,name,status}Prometheus gauge, set for every
monitor including the ones whose status is still unknown. -
Permissions-PolicyandCross-Origin-Opener-Policyresponse headers. -
hora event -- <title>andhora announce -- <title>record a title that
starts with a subcommand word (hora announce clear skiesused to clear
every banner). -
just deps: a report-only freshness check across crates, docs packages,
pinned Actions and base images.
Changed
- Local-only downs no longer page. With
confirm_with_peers, a down
that every peer that answered sees up used to be sent to the monitor's
channels anyway, softened. It now goes toalerts.notify_unconfirmed(or
is only recorded: incident, timeline, Not an outage). Only that explicit
contradiction quiets a down: no peer, no answer, or any peer seeing it
down alert as before. The peers are asked again every 5 minutes while it
lasts; once they no longer all see it up, the real down goes out, once.
A recovery goes to whoever received a down, across a daemon restart too
(the routing is kept on the incident, migration0022, instant). The
operator panel Why nobody was woken says what the config does with such
a down. - Probe failures name their cause. connection failed: connection
refused, connection failed: TLS invalid peer certificate: Expired
instead of a bare connection failed; proxy failed: tunnel error: proxy
authorization required (HTTP 407) or proxy failed: SOCKS error:
credentials not accepted instead of blaming the target; domain does not
exist (NXDOMAIN) apart from no A records found, and server answered
Server Failure (rcode 2) instead of hickory's debug text; an assertion
judged on a body cut atmax_body_kbsays (body cut at 256 KiB). A
push to an unknown id, a cron grace of 20 seconds (+ 20s grace, was
+ 0m grace) and an ICMP error (icmp destination unreachable (code 1)
from 192.0.2.1) read as what they are. - An https monitor redirected to plain http is down (redirected to
plain http, public reason of the same words): following it carried the
session in clear text without a word. - TCP and certificate connects try the next address after 250 ms
(happy-eyeballs): a dual-stack name whose IPv6 address blackholes no
longer times out while IPv4 answers. - A custom
dns_resolverfalls back to TCP for answers too large for a
datagram (long TXT, SPF, DKIM records), which used to fail as truncated. - A proxied monitor's certificate is read through its proxy (HTTP
CONNECT,socks5,socks5h), like its probe;hora probedoes the
same. It was read by a direct dial around the proxy. - Certificate warnings count hours under a day (expires in 10 hours;
hora probeprints 10 hours left): with less than a day left they said
has expired. POST /api/push/{id}answers 401 for an unknown id, like a wrong
token (it answered 404, which listed the push ids).- An announcement's
untilis at most a year (365d); a longer one is
refused (400, CLI error). - A confirmation peer that does not watch the target (it answers 404) is no
longer counted as unreachable: the verdict reads no peer watches this
target, unconfirmed, and/watchersno longer counts it under No
answer. - A peer that misses a vantage poll keeps its last view for three rounds:
one lost poll no longer makes its column (and the Not an outage notice)
vanish for a minute. - The status summary is never built on a request. A background task
rebuilds it continuously (about 150 ms on 575M checks, was 13-20 s) and
every page,/api/summary,/metrics, group page and peer exchange is
served from the latest snapshot, at most a few seconds old. It is built
once for the operator and cut in memory for the public and each group;
pages are rendered once per snapshot and audience. Only the first request
after start waits for the first build (logged as "status page ready").
Announcing, clearing banners or recording an event returns once the page
shows it. - 24-hour figures read the hourly roll-ups plus the current hour, not
millions of raw checks, and hourly roll-ups now run every 5 minutes
instead of every 6 hours. p50/p95/p99 come from per-hour latency
histograms: exact below 64 ms, within 1.6% above. - Push heartbeats drive alerting with their own verdict. A push with
status=downmakes the monitor down on the next tick (confirmed after
fail_threshold), with the job'smsgas the reason;status=degraded
counts as degraded, soalert_on_degradedworks for push monitors. It
used to be read as a heartbeat like any other, so the alert came one
interval late, as a missed heartbeat, without the message. - A push monitor or watched peer that never sent a heartbeat alerts
("no heartbeat received yet") once its interval (or its first scheduled
run plus grace) has passed since Hora first watched it - across restarts.
It used to stay grey forever. - Alert state survives edits and restarts. Editing a monitor or peer no
longer re-sends a down for an ongoing outage, nor loses its recovery; a
daemon restart mid-outage does not announce the down again (it is seeded
from the open incident) and still sends the recovery. Monitor tasks that
die are restarted and logged. - Notifications are worded once for every channel. The ten channels
render from one neutral message and only add their own markup and
priority, so wording no longer drifts between them. Visible changes: ntfy,
Gotify and Pushover bodies readAPI is DOWN/API is slow (812ms)/
API recoveredinstead ofDOWN: API/DEGRADED:/RECOVERED:; Free
Mobile SMS use the same text (no emoji), now with fingerprints; email
subjects are[TAG] <headline>([DOWN] API is DOWN); Slack uses Unicode
emoji; a certificate change carries the MITM / renewal hint everywhere. - Oversized notifications are cut to each API's limits (Discord 256/4096,
Telegram 4096, Pushover 1024, ntfy 4 KB...) with an ellipsis, instead of
being refused with a 400 that then counted against the channel. A 413 no
longer counts toward the failing-channel watchdog. - Removed monitors keep their history for 7 days. Taking an id out of
the config (a rename, a monitor commented out for an afternoon) used to
delete its checks, aggregates and incidents at the next 6-hourly prune. A
warning now names the ids and the deletion date; restoring the id within
the week keeps everything. - One duration format everywhere (
2h 5m,3m 10s): the monthly report
used to print2h 05mand incident phrases1h13m. - Page rate limit. Every non-static route (pages, history, timeline,
feed, reports, badges, heatmaps,/metrics,/healthz) is now limited per
client IP, at four times the API burst and refill./report/{month}and
the heatmap SVG are cached for 60 seconds, and reports older than the
12-month aggregate retention answer 400. /healthzanswers 503 when the node is degraded (stalled scheduler,
broken database), body unchanged - so the image'sHEALTHCHECK, and any
orchestrator probing it, now sees an unhealthy node. While the node is
still starting (before the scheduler's first tick) that 503 is logged at
debug, not as an ERROR line per health check.- Less memory per monitor. Monitors share their probe client (one per
proxy, instead of one per monitor at about 45 KB each), and the daily
bars keep numbers instead of a formatted title per day: a node with
10,000 monitors idles at 433 MB instead of 679 MB. - The WAL file (
<db>-wal) is truncated back to 64 MiB after a
checkpoint; a burst of writes used to leave it at its high-water mark. - Prometheus:
hora_monitor_up/hora_monitor_degradedare omitted
while a monitor's status is unknown (they read as down after every
deploy);hora_monitor_latency_msis typedgauge. - Atom feed content is escaped HTML as RFC 4287 requires, entry ids point
at/incident/{id}(they said/incidents/), each entry links to its
post-mortem, and the feed is titled afterpage.title. POST /api/push/{id}answers 400 on an unknownstatus(dwonwas
recorded as up) or a negativeping. Two pushes in the same second: the
last one wins, and a push replaces a recorded miss of the same second.- A second daemon on the same database refuses to start ("locking the
database: ... held by another Hora process") instead of racing the first. - Failure reasons are stored as a code next to their text (migration
0021, instant), so the public page no longer guesses them from the
wording. An exec plugin's failure now always reads "plugin check failed"
publicly, and a push monitor that never pinged reads "no heartbeat
received yet". - Settings that belong to another kind of monitor (
expected_status,
headersormax_body_kbon a non-http monitor,keyword_invert
withoutkeyword,json_expectedwithoutjson_query) were silently
ignored; they still load, now with a warning.hora probe --kindchecks
its target against the kind up front. - Hora logs one warning when requests arrive through a proxy (forwarding
headers) whileserver.client_ip_headeris unset, since every visitor
then shares the proxy's rate-limit bucket. - OpenAPI: status fields are string enums; the JSON is unchanged.
- Config bounds:
interval_secs,timeout_secs,expect_every_secs
andhealth.interval_secsare capped at 30 days,expected_statusmust be
100-599, an IPv6 tcp target must be bracketed ([::1]:80) and
dns_resolvermust be an IP and port. A warning flagstimeout_secs
aboveinterval_secs. hora topnever blocks on the network: requests run in the background,
qand Ctrl-C always answer, and the header colour follows the API's
overallstate.- Multi-vantage confirmation is never attempted for exec monitors; a peer
asked to probe an exec or push monitor answers 400. - Cert checks run eight at a time under a 5-minute deadline and stop on
shutdown; monitor stagger phases shift once (a stable hash replaces
DefaultHasher). - Image: the build context is an allowlist, the cargo registry and target
directory are BuildKit cache mounts, and the published image carries
build provenance and an SBOM. CI Actions are pinned to commit SHAs. - Dependencies: utoipa 6, nix 0.31 (process-group kill), docs on Astro 7.3
and Starlight 0.42. - Fonts: Cal Sans, Fraunces, Borel and Mona Sans Mono, served from
/assets/with a content hash; the old/assets/CalSans-SemiBold.woff2
route is gone. Badges and the heatmap use the brand palette.
Deprecated
?token=on write endpoints (push, alert, announce, silence, event).
It still works, but the response carriesDeprecation: trueand aLink
to the authentication docs,
and Hora logs one warning per endpoint. SendAuthorization: Bearer(or
X-Push-Tokenfor push and alert) instead. Read-only views keep
?token=.
Fixed
- Checks were lost at every 6-hourly prune on large databases. Once
the history reached its retention, the prune deleted six hours of checks
in one statement that held the write lock for 5 to 10 seconds, so the
scheduler's inserts failed with "database is locked". Retention (and the
sweep of a removed monitor's history) now deletes five minutes of checks
at a time, oldest first, pausing in between; a tick spends at most two
minutes deleting and comes back five minutes later for any backlog, so a
much lowered retention never blocks the roll-ups or a shutdown. - Group pages leaked operator data: a group token saw every group's
deploy markers in the sparklines and the names of other groups' private
monitors in "caused by" / "impacts". A group audience now names only
public monitors and its own. - Monthly report and digest under-counted older months: they looked at
the 1000 (500) most recent incidents only; they now query the incidents
that overlap the period, and the report stops at the month's end. - Exec monitors were "confirmed down from N/N vantage points" when
[health].confirm_with_peerswas on: peers answered "not a network probe"
as down. - Certificate pin mismatch accepted for good when first seen during
maintenance, or when the alert failed to deliver. It now alerts after the
window and retries every check, and changingcert_pinto another value
that still does not match alerts again. expected_status = 301(or any 3xx) could never pass: redirects were
always followed.dns_resolver = "[2620:fe::fe]:53"passed validation but failed every
probe; a hostname resolver now fails validation instead of every probe.- Exec timeouts left grandchildren running: the plugin now runs in its
own process group, killed as a whole; a plugin that exits while a
background child holds stdout reports its real result, not "timed out",
and that child is killed instead of piling up, one per interval. - ICMP monitors were up when a router answered "destination
unreachable": surge-ping hands back the ICMP errors that quote the
request as a reply, so the router's round-trip passed for the host's.
Only an echo reply counts now. - A cron push heartbeat that arrived slightly before its run counted as
missed. - A body cut mid-read was reported as a failed assertion ("keyword missing")
instead of an invalid response body. - Sparkline buckets jittered on every refresh.
- Cert checks did not work for HTTP monitors on an IPv6 literal.
- RDAP and release lookups: relative redirects work, an https-to-http hop is
refused, and answers are capped at 1 MiB. - The alert endpoint told unknown ids (404) from private ones (401).
- History and timeline pages showed fewer entries than they should when
private monitors filled the limit. - The plain-text page's columns were not aligned, and
/status/{group}now
serves it to curl too. The empty sparkline text is no longer stretched. hora import kuma: valid TOML for any name, unique ids, bracketed IPv6.- The CLI no longer exits from deep inside a command, so the database pool
closes cleanly; the silence reason is capped at 500 characters like the API.
Security
- Remote text cannot drive the operator's terminal. A probed body, a
pushmsg, a pushed alert or a plugin's output could carry escape
sequences (retitle or clear the terminal, write the clipboard with OSC 52)
intohora incidents,hora postmortemandhora probe. Control
characters other than newline and tab are replaced with U+FFFD where the
text is stored and again where the CLI prints it. /monitor/{id}showed the raw target (https://user:pass@...,
?api_key=...) to group viewers; credentials are now redacted on the
page.- IPv6 clients are rate-limited by their
/64(one subscriber holds a
whole prefix, so a bucket per address was no limit), IPv4-mapped
addresses as IPv4. - Responses to credentialed requests (
Authorization,X-Push-Token,
?token=) carryCache-Control: no-store; the CSP adds
form-action 'self'. /report/{month}accepts digits only (2026-+5was a second spelling,
and cache entry, of2026-05) and rendered report pages are evicted with
their report.- Post-mortems put the remote text of the First failure and timeline
lines in a code span, so markdown in a response body (a tracking image, a
link) stays text in the ticket it is pasted into. - Secrets stay on their origin. Notifier, webhook and mesh requests
follow a redirect only within the same origin (or an http-to-https upgrade
on the same host): reqwest stripsAuthorizationacross hosts but not
X-Push-Token, the Gotify key, or a Pushover body re-sent on 307/308.
A webhook URL that redirects to another host now fails delivery. - Failure snapshots redact credential headers (
Set-Cookie,
Authorization,WWW-Authenticate, API-key headers...) before they reach
incidents,/historyand post-mortems. - Monitor and release-watch
Debugoutput masks URL query values;
peers.witness_urlis a secret. hora backupcreates the file0600atomically instead of tightening it
after the copy.- A malformed inbound
x-request-idis replaced instead of echoed; CSP
img-srcno longer allowsdata:, andstyle-srcno longer needs
'unsafe-inline': pages carry no inline style, and still run no script. - The token in a query string is spliced into each response, never into
the pages cached and shared between viewers. - CI checkouts no longer persist the token, and the Pages write permission
is scoped to the deploy job.