v3.16.0

v3.16.0

47 commits since v3.15.0: 23 fix, 9 feat, 6 docs, 2 test, 2 perf, 2 ci, 1 ui, 1 deps,
1 chore. Nine feats, so minor not patch.

This release finishes the liveness work of v3.15.0. There, the server stopped
counting a relay hop as "heard" when its short prefix is shared with other
repeaters. The ingestor still did that for the stored last-seen time, and now it no
longer does, so some repeaters keep an older Last Heard after the upgrade. The new
features are optional: personal MQTT logins for observers, each observer's latest
neighbour report, and the radio setting a packet was heard on. The rest is mostly
pages that could show an older answer or a wrong number.

Read this before upgrading

Repeaters that are only heard in paths keep an older Last Heard

A node's Last Heard also follows the last-seen time the ingestor stores for it. The
ingestor refreshed that time from relay hops by a weaker rule than the server uses
since v3.15.0, so a repeater could still look heard on traffic that was really
another one. Three changes close that:

• A relay hop only refreshes a node when its prefix belongs to that node alone
  (#2334, closes #2329). Before, when a prefix matched several nodes, the ingestor
  picked the one that neighbours the previous hop.
• Only hops a packet has already passed count, which means flood packets (#2308,
  closes #2307). In a direct packet the hops in the path are still to come, and in a
  trace they are the route that was asked for. Tracing or messaging a switched-off
  repeater kept it looking heard.
• On the server, relay evidence has to come from the path of the same observation,
  not from another observation of the same packet (#2298, closes #2266).

What you will see after the upgrade:

• On a mesh with 1-byte paths, relay hops practically stop refreshing Last
  Heard. In a sample from a production database (the last 200,000 observations,
  about 24 hours, 1,818 nodes), none of the 506,545 one-byte hop instances had a
  prefix of its own. A repeater on such a mesh stays Active on its own packets, such
  as its adverts.
• In the same sample, 2.0% of resolved hops no longer count (12,894 of 652,074).
  7 of the 1,053 nodes that relay hops refreshed lost all of those refreshes in that
  window; 1 of those 7 sent no advert in it either.
• The green "actively relaying" mark on a repeater's Last Relayed row now needs
  certain relay evidence within healthThresholds.relayActiveHours (#2336, closes
  #2331). Forwarding another node's advert counts, sending its own advert does not.
  The yellow "alive (idle)" mark is gone: no mark does not mean the repeater is idle
  or off. The Last Relayed time itself still includes shared prefixes.

Nothing to configure. As in v3.15.0: if a repeater you know is on the air shows
Uncertain or Stale, its prefix is shared and its last advert is older than its
silent window.

With user accounts on: back up users.db first

users.db is migrated from schema version 6 to 8 at the first start (#2306,
#2348), also when you do not turn on observer logins. A v3.15 binary refuses it after
that ("database schema version 8 is newer than this binary supports"). To roll back,
you need the copy from before the upgrade: the daily backups in backups/ next to
users.db, or your own. Without user accounts nothing changes.

Building from source needs Go 1.26, and the image moves to Alpine 3.24

#2297 updates the Go dependencies after a scan of the published image found 19 high
and 20 medium advisories in them, and moves the runtime image from Alpine 3.20, which
is out of support, to Alpine 3.24.

• The go.mod files of the server, the ingestor and internal/users now ask for Go
  1.26.0 (was 1.22). The Docker build and CI already use 1.27. If you build the
  binaries yourself with an older Go, it downloads a newer toolchain, or refuses to
  build with GOTOOLCHAIN=local.
• In Docker, the bundled mosquitto, Caddy and supervisor now come from Alpine 3.24.
  The local build in the PR had mosquitto 2.1.2, Caddy v2.11.4 and supervisor 4.3.0.
  The user and group ids of the caddy and mosquitto accounts did not change, so
  mounted volumes keep their owner.

Observer MQTT logins

All three need user accounts (userManagement.enabled), an EMQX broker and
userManagement.observerLogins.enabled. Off by default. Setup, with a check and a
rollback for every step, is in docs/observer-mqtt-logins.md.

• A personal MQTT login per user (#2306, closes #2319) - observers usually share
  one broker login, and anyone who has it can publish as any observer. With this on,
  a user requests a personal login on the account page, CoreScope generates it and
  mails it, and EMQX asks CoreScope whether that login may connect and publish. A
  personal login can only publish for its own observers. The first publish for an
  observer nobody owns yet binds it to that user, up to 20 per account. Admins get a
  mail when a known observer is claimed, and can reassign it or revoke the login.
  Logins CoreScope does not manage keep working through the broker's own users, so the
  shared login goes on working during the move. The owner also gets "My observers go
  offline" mails once notifications are on. Any account can claim an observer nobody
  owns yet, so the doc suggests assigning known observers to their owners before you
  announce the feature.
• One credentials mail for every uploader (#2306, #2347, #2349) - ready-to-paste
  blocks for meshcoretomqtt, the observer firmware CLI (on MQTT slot 4, which is empty
  on a fresh install), Home Assistant, OpenHop, and the plain values. The firmware and
  OpenHop blocks also turn on the neighbour report described below. Older firmware
  does not know that command, and the mail says to skip the line then.
• Lock the shared login out of moved observers (#2348) - list the broker's shared
  login names in observerLogins.sharedUsernames. The shared login can then no longer
  publish as an observer once its owner has published with a personal login. The
  admin page shows which observers are confirmed this way. An observer that still runs
  a second uploader on the shared login loses that uploader; the denial in the log
  names the observer.

Observers

• Latest neighbour report (#2300, closes #2270, #2271, #2330) - for observers
  that send a /neighbors report, the observers list has a sortable Neighbors column
  with the time of the last report and the number of neighbours. The observer page
  lists every neighbour by name, with SNR, when it was heard and its scopes. A report
  the broker replays after a restart does not make a dead observer look fresh, and a
  purged observer's report is removed with it.
• The radio setting a packet was heard on (#2292, part of #1869) - CoreScope kept
  only an observer's current frequency, bandwidth, SF and CR, so after a retune every
  older packet looked as if it was heard on the new setting. It now keeps each
  observer's radio history and works out the setting per observation, also per radio
  on nodes with more than one. The packet detail shows it, the packet filter takes
  radio, freq, bw, sf and cr (for example sf == 8), and the observer page
  shows the history once the setting has changed. The history starts at the upgrade,
  from each observer's current setting. Packets heard between a retune and the
  observer's next status message, minutes later, count for the old setting.
• The observer page shows its details first (#2339) - name, status and radio
  appear as soon as they arrive, without waiting for the analytics. An analytics error
  no longer hides them.
• The observer page stays on the observer you picked (#2323, closes #2316) - a slow
  answer for an earlier observer or period no longer replaces the one on screen. A new
  visit now asks for the 7 days it shows; before, it could ask for the period of the
  previous visit.
• Compare no longer counts transmissions as receptions (#2346) - an observer's own
  transmitted packets were counted as received, which made the overlap between two
  observers look higher.

Nodes

• Node previews show the same fields as the full page (#2315, closes #2309) - Last
  Relayed, Traffic share and Hash Prefix in the side and tablet previews. The tablet
  preview now shows the node's name.
• "View example packet" (#2296, closes #2291) - the observed-path links now say
  that they open one example packet.
• Battery badges (#2282, closes #2232) - node analytics showed SVG source text and
  oversized icons instead of the battery status icon.
• LoRa settings respect hidden nodes (#2276, closes #2268) - the optional radio
  settings feature (rfParams) no longer returns blacklisted or hidden nodes, also
  from its cache.

Pages and navigation

• Browser tab titles name the page (#2287, closes #2259) - for example
  Node abababababab | Reach | CoreScope, with your own site name kept. Useful with
  many tabs open.
• Maps and charts load from your own server (#2299) - Leaflet, leaflet.heat and
  Chart.js come from vendor/ instead of unpkg.com. Visitors' browsers no longer
  fetch these scripts from a third party, and the map and charts work where unpkg is
  blocked. Map tiles are not affected by this change. The API docs
  page (/api/docs) still loads Swagger UI from unpkg.
• Navigation - the links no longer overlap More when the account controls appear
  late (#2286, closes #2216), the More menu keeps keyboard focus when it rearranges,
  and a late packets answer no longer pulls you back from the page you went to (#2333,
  closes #2332, #2335).
• Packets - All time survives a reload and a shared link (#2279, closes #2231). The
  HB column no longer shows 4, a value that cannot exist (#2351): it read the hash size
  at the wrong place when observers heard the same packet with a different route type.
  In the last 300,000 observations of a production database, about 4,300 had a
  different route type than their packet's first observation.
• Analytics - Nodes counts roles within the selected region or area (#2275, closes
  #2213), and column resize handles stay on their header (#2274, closes #2217).

Coverage and reach

• My coverage and the observer filter on the noise layer (#2326) - on the Mobile RX
  coverage page, the noise layer showed every companion's samples whatever you
  selected.
• English tooltips on the coverage leaderboard (#2280) - the score column now
  explains itself: every cell of about 150 m adds 1, divided by the number of observers
  that heard something there.
• Reach waits for a free slot instead of answering "busy" at once (#2249) - at most
  2 uncached reach reports still run at the same time. The next one now waits up to 3
  seconds for a slot; at most 8 wait, and any further one gets the busy answer (429) at
  once. On a database with 10.5 million observations, uncached reports took from about
  9 to over 60 seconds, so there the wait mostly ends in the same 429, 3 seconds later.
  It helps with two tabs, or switching between 7, 14 and 30 days.
• An interrupted reach report is not cached (#2277, closes #2258) - it could leave an
  empty or partial result in the cache.

Server and ingestor

• Live packets flow during the startup clean-up (#2328) - the ingestor held
  incoming MQTT messages until the whole packet-retention sweep at startup had
  finished. Ingest now opens before that sweep and writes between its batches. No
  timing was measured. This is a partial fix for #2325, which stays open.
• Distance analytics - after a chunked startup the distance index is built on first
  use instead of at once (#2278, closes #2227), and a background fill can no longer
  crash the server or mark an incomplete index as ready (#2284, closes #2281).

Faster

• Bridge and coverage scores of the neighbour graph, on a test graph of 2,000 nodes:
  8.2 to 0.82 seconds and 4.9 to 0.69 seconds, with allocations per run from about
  2.1 GB and 1.5 GB to about 2.5 MB and 2.2 MB (#2293, closes #2226).
• The health check (/api/healthz) no longer walks every packet in memory once startup
  is done: 1,229 to 82 microseconds per probe with 100,000 packets (#2305, closes #2301).

Docker and operations

• Containers are healthy only when startup is done (#2285, closes #2229) - the
  healthchecks of the five compose files and the deploy check of manage.sh use
  /api/healthz. manage.sh update now records the version and build time of what it
  checked out, and handles env files with Windows line endings. Only applies if you
  take the new compose files or manage.sh.
• The backup guide takes a consistent copy (#2345, closes #703) -
  docs/deployment.md now backs up through /api/backup and checks the download,
  instead of copying a running meshcore.db, which can miss data that is still in the
  write-ahead log. It also describes the restore.
• The unused Dockerfile.go and Dockerfile.node are removed (#2314, closes #2310).

Tests, CI and docs

• The API reference now describes the packet-list parameters (#2294), six diagnostics
  endpoints (#2303), and node resolution and battery history (#2338). The list of
  undocumented routes went from 24 to 16.
• Old release notes and planning documents moved to docs/release-notes/ and
  docs/archive/ (#2313, closes #2311; #2321, closes #2318).
• CI runs the JavaScript unit tests before the Go checks (#2322, closes #2317), and runs
  the browser tests on Ubuntu 24.04, because the browser install failed on 26.04
  runners (#2350).
• Two unstable tests are fixed (#2273, closes #2272; #2289, closes #2288).

Issues closed

33 issues, from the 47 pull requests in this range. Collected from each pull
request's closing references rather than from commit text.

• #703 feat: SQLite database replication, live backup copies
• #2213 Analytics Nodes shows global role totals when a region is selected
• #2216 bug(nav): primary link floor and late account controls can overlap More
• #2217 bug(analytics): column resize handles attach to page instead of static headers
• #2226 perf: graph score warmups delay startup and allocate gigabytes
• #2227 perf: chunked startup still eagerly builds the lazy distance index
• #2229 bug(operators): correct update metadata, readiness checks and CRLF startup
• #2231 bug(packets): All time does not survive reload or a shared link
• #2232 bug(node analytics): battery badge renders SVG source and mis-sizes icons
• #2258 reach: a cancelled build can cache empty scopes or a partial degree snapshot
• #2259 feat: Set HTML <title> based on current view
• #2266 bug(liveness): certain relay evidence uses another observation's route/header
• #2268 bug(rf-params): response includes blacklisted and hidden-name nodes
• #2270 FEATURE REQUEST: Add neighbor report info to observer table+details
• #2271 FEATURE REQUEST: Add heard neighbors to observer details page
• #2272 test(ingestor): hop coverage assertions depend on database setup speed
• #2281 analytics: background-fill gate reset can be overwritten by an in-flight distance build
• #2288 test(rf-health): airtime assertion races with theme refresh
• #2291 ui(nodes): label path sample links as example packets
• #2301 perf(healthz): avoid scanning the packet cache on readiness probes
• #2307 Ingestor marks a node heard from DIRECT and TRACE path hops
• #2309 ui(nodes): expose relay, traffic-share and hash details in previews
• #2310 chore: remove unused legacy Dockerfiles
• #2311 docs: consolidate historical release notes
• #2316 bug(observers): delayed detail responses overwrite a newer selection
• #2317 ci: run the standalone JavaScript gate before backend validation
• #2318 docs: archive historical root planning documents
• #2319 Any holder of the shared MQTT login can publish as any observer
• #2329 liveness: ingestor refreshes last_seen for adjacency-resolved 1-byte hops the server rejects
• #2330 observers: /neighbors reports trust the RETAIN flag and survive an observer purge
• #2331 nodes: show "actively relaying" again, from certain relay evidence only
• #2332 bug(nav): refitting an open More menu discards keyboard focus
• #2335 bug(packets): a late list load overwrites navigation before hashchange

Contributors

Pull request authors: @n30nex (33), @efiten (8), @TeTeHacko (4), @anieto (1),
@nullrouten0 (1).

Co-authors credited in the commits: @dborup, @n30nex and @efiten on #2249. The reach
limiter and most of its tests are @dborup's, the cap on waiting requests is
@n30nex's.

Issues in this release were reported by @n30nex (20), @efiten (6), @cwichura (2),
@dborup (2), @anieto (1), @Kpa-clawbot (1), @t0mk (1).

@n30nex wrote 33 of the 47 pull requests. @efiten, the interim maintainer, wrote 8
(the observer logins, the HB column, the coverage noise layer and tooltips, and the CI pin) and
reviewed and merged all 47.

Upgrade notes

• No manual migration step. At the first start the ingestor creates three tables in
  the analyzer database: observer_neighbor_reports (#2300), observer_radios, filled
  with each observer's current radio setting, and observation_radios (#2292).
• With user accounts on, users.db moves to schema version 8. Keep a copy from before
  the upgrade (see above).
• Building from source needs Go 1.26 or newer (see above).
• Opt-in: userManagement.observerLogins with enabled, hookSecret,
  encryptionKey, broker and optionally sharedUsernames. It needs
  userManagement.enabled and the EMQX hook setup from
  docs/observer-mqtt-logins.md. The secret and the key can also come from
  CORESCOPE_OBSERVER_HOOK_SECRET and CORESCOPE_OBSERVER_KEY. Keep the key apart from
  users.db and its backups: without it the mails cannot be resent and every login has
  to be reissued. The example values are in config.example.json.
• The neighbour report and the radio history need nothing from you: they fill as
  observers send reports and status messages. An observer sends the neighbour report
  with set mqtt.neighbors on in the observer firmware, or Publish Neighbours in
  OpenHop.

What's Changed

• test: make hop coverage clock checks deterministic by @n30nex in #2273
• fix(analytics): anchor resize handles to table headers by @n30nex in #2274
• fix(analytics): count roles within selected filters by @n30nex in #2275
• fix(rf-params): respect node visibility in responses by @n30nex in #2276
• fix(reach): avoid caching interrupted query results by @n30nex in #2277
• fix(analytics): keep distance indexing lazy during startup by @n30nex in #2278
• fix(packets): preserve All time across reloads and links by @n30nex in #2279
• fix(rx-coverage): English tooltips for the leaderboard score and cells columns by @efiten in #2280
• feat(observer-logins): personal MQTT logins for observers by @efiten in #2306
• ci: run JavaScript unit tests before backend validation by @n30nex in #2322
• docs: archive historical planning documents by @n30nex in #2321
• docs: consolidate historical release notes by @n30nex in #2313
• test(rf-health): wait for airtime badges after grid refresh by @n30nex in #2289
• docs(api): describe performance and clock diagnostics by @n30nex in #2303
• docs(api): describe the supported packet-list query parameters by @n30nex in #2294
• chore: remove unused legacy Dockerfiles by @n30nex in #2314
• fix(ingestor): only flood relay hops refresh nodes.last_seen (#2307) by @anieto in #2308
• fix(liveness): require evidence from each observed relay frame by @n30nex in #2298
• perf(healthz): avoid scanning retained packets after startup by @n30nex in #2305
• perf(graph): reuse indexed shortest-path workspaces by @n30nex in #2293
• fix(analytics): preserve distance invalidation during an active build by @n30nex in #2284
• fix(observers): keep detail responses tied to the current view by @n30nex in #2323
• feat(frontend): serve Leaflet, leaflet.heat and Chart.js from vendor/, not unpkg.com by @TeTeHacko in #2299
• feat(ui): identify current views in browser tab titles by @n30nex in #2287
• fix(nav): fit primary links around late account controls by @n30nex in #2286
• ui(nodes): identify path links as example packets by @n30nex in #2296
• fix(nodes): preserve names and metadata in preview panels by @n30nex in #2315
• fix(analytics): render safe, correctly sized battery status icons by @n30nex in #2282
• fix(operators): refresh update metadata and wait for readiness by @n30nex in #2285
• feat(observers): radio setting per observation from the observer's radio history (#1869) by @TeTeHacko in #2292
• deps: golang.org/x/net v0.61.0, x/crypto v0.58.0, paho.mqtt.golang v1.5.1; runtime alpine:3.24 by @TeTeHacko in #2297
• feat(observer-logins): lock the shared login out of migrated observers by @efiten in #2348
• fix(ui): preserve navigation and menu focus during delayed updates by @n30nex in #2333
• feat(observer-logins): OpenHop block in the credentials mail by @efiten in #2347
• ci(e2e): run the E2E shards on ubuntu-24.04 and fail on a broken browser install by @efiten in #2350
• feat(observer-logins): turn on neighbour reports in the firmware block of the credentials mail by @efiten in #2349
• fix(packets): read HB from the row's own raw_hex header by @efiten in #2351
• docs: use consistent snapshots for backup and restore by @n30nex in #2345
• docs(api): document node resolution and battery history by @n30nex in #2338
• fix(ingestor): open buffered writes before startup packet retention by @n30nex in #2328
• fix(ingestor): require unique hop prefixes for relay liveness by @n30nex in #2334
• feat(nodes): show relay activity from certain forwarding evidence by @n30nex in #2336
• fix(compare): exclude observer TX-only packets from overlap by @n30nex in #2346
• fix(observers): show metadata while analytics loads by @n30nex in #2339
• feat(observers): latest /neighbors report on the observers list and page (#2270, #2271) by @TeTeHacko in #2300
• fix(rx-coverage): My coverage and the observer filter on the noise layer by @efiten in #2326
• fix(reach): queue a cold request up to 3 s for a build slot before answering 429 by @nullrouten0 in #2249
• docs(release): v3.16.0 release notes and changelog entry by @efiten in #2355

Full Changelog: v3.15.0...v3.16.0