v3.15.0

v3.15.0

77 commits since v3.14.0: 27 fix, 23 perf, 19 feat, 3 test, 2 ci, 2 docs (one of them
these notes), 1 chore. Nineteen feats, so minor not patch.

The main change is how CoreScope decides that a repeater is alive. Until now a
repeater could stay green for days after it went silent, because traffic from
another repeater with the same short prefix was counted for it. That is fixed, and
it changes the status of some of your nodes after the upgrade. The rest of the
release is optional features (LoRa settings per node, observer airtime, companion
linking, smarter offline mails) and a lot of memory and speed work on the server.

Read this before upgrading

Some repeaters change from Active to Uncertain or Stale

A repeater in a path is only known by the first 1, 2 or 3 bytes of its key. With
1-byte paths, dozens of repeaters share the same prefix, and CoreScope counted a
hop like that as "heard" for whichever repeater it picked. A repeater that had been
off for days could stay Active on traffic that was really another one.

Since #2261 (closes #2260), a node only counts as heard on evidence that can only
be that node: its own packets, or a relay hop whose prefix no other repeater has.
Adverts with an invalid signature no longer count either.

What you will see after the upgrade:

• A new grey status, Uncertain, for a repeater or room that sent nothing certain
  in its silent window, while its shared prefix still shows up in paths. It may be
  that repeater, or another one with the same prefix. The nodes list and the map have
  a filter button for it, and the map dims these nodes like stale ones.
• Last Heard can be older than before for repeaters that are only heard through a
  shared prefix.
• Offline mails for watched nodes now go out for repeaters that a namesake kept
  green before. The mail says when the node is uncertain rather than gone.
• Node analytics split the activity chart into certain and uncertain traffic, and
  Availability and Longest Silence use only the certain part (#2262).

On a staging instance with 1,682 repeaters and rooms, on the day of the change: 1,497
Active, 117 Uncertain, 68 Stale. We did not count the same instance before the change.

Right after a restart, fewer nodes show Uncertain and more show Stale. The first load
skips ambiguous relay hops (#1643), so they come back as live traffic arrives. Give
it a few hours before you judge a node.

Nothing to configure. A node's own packets, such as its adverts, always count, so a
repeater that adverts within its silent window stays Active. If a node you know is
on the air shows Uncertain or Stale, its prefix is shared with another repeater and
its last advert is older than that window.

With user accounts on: back up users.db first

users.db is migrated to schema version 6 at the first start (#2174). A v3.14
binary refuses to open it after that. To roll back, you need the copy from before
the upgrade: the daily backups in backups/ next to users.db, or your own. Without
user accounts nothing changes.

The ingestor fills in channel and scope names in the background

When you add a channel key or a region key, packets stored before that keep showing
as an encrypted channel or an unknown scope. From this release the ingestor goes back
over those rows and fills in the name (#2133). It runs at every start and again when
keys are added, and creates three small tables to keep track. The server picks up
the result without a restart. There is no measured duration for a large database
yet.

Some answers are a few seconds or minutes older, in return for speed

• The nodes list is shared between identical requests for up to 15 seconds
  (#2165). A geo-filter or blacklist change clears it at once.
• The neighbor graph is rebuilt in the background every 5 minutes
  (neighborGraph.cacheRecomputeIntervalSeconds, default 300) instead of while a
  page waits (#2222). One rebuild took about 11 seconds before, on a page load.
• Server caches that have expired keep serving the old value while one refresh runs
  (#2147).

Hop names and regions can shift

Several changes make hop names and region membership more accurate, so some will
look different after the upgrade:

• 104 airport codes for Belgium, the Netherlands, Luxembourg, Germany and France were
  added to the region table (#2188, closes #2098). Narrowing a hop by the observer's
  region is now only applied to the hop the observer heard directly.
• A node is placed in a region by where it is heard zero-hop first, then by the
  fewest hops, up to 3 (#2190, closes #1879). Before, it went to every region whose
  observers heard it.
• Zero-hop adverts on a transport-direct route are now recognised (#2234, closes
  #2225).
• The packet detail and the map route view resolve 1-byte hops as one whole path
  instead of hop by hop; a hop that matches no known node shows as "unlisted"
  (#2245).
• An observer's own transmissions no longer count as something it received, in Heard
  By and in the neighbor graph (#2220).

Docker: a geo-filter save on a single-file config mount now fails visibly

If you mount config.json as a single file, saving the geo filter now returns an
error (#2137). Before, it wrote a copy inside the container that the next start threw
away. Mount the directory instead. A symlinked config.json is now written through.

LoRa settings and airtime

• LoRa radio settings per node (#2209, closes #1268) - the node page shows the
  frequency, spreading factor, bandwidth and coding rate the node reports on the
  MeshCore map. Off until you set rfParams.nodesUrl.
• Nodes with different settings than their neighbours (#2210) - a LoRa badge and a
  ?lora=differs filter on the nodes list for a node whose settings differ from the
  nodes within 50 km (rfParams.compareRadiusKm). Useful to find a repeater that was
  set up on the wrong preset.
• Observer TX airtime (#2208, closes #661) - a "TX 1h" column on the observers list
  and a tile on RF Health show how much of the last hour an observer spent
  transmitting, with a warning above 10% (dutyCycleWarnPct).

Notifications and accounts

These only apply with user accounts on.

• Smarter offline mails (#2262) - off by default
  (userManagement.notifications.adaptiveOffline). It learns how long each watched
  node is normally silent, and mails about 30 minutes into a real outage instead of
  after the fixed window.
• Alerts from an external feed (#2178, closes #2177) - point
  userManagement.notifications.externalAlerts at a JSON feed, and watched nodes in
  it get a mail.
• Link a companion to your account (#2174, #2175, #2176) - a companion app logs in
  with a device token. With clientRxCoverage.requireLinkedCompanion, coverage is only
  taken from linked companions, and the account page lists your companions with a "My
  coverage" view.
• Postal as mail provider (#2154) - userManagement.mail.provider: "postal", next
  to Brevo.
• Re-registering with a pending account updates password and name together (#2253,
  closes #2228).

Coverage and companions

• Where a companion drove and heard nothing (#2172) - the Mobile RX coverage page
  shows gaps next to receptions, for instances with clientRfSamples.
• Repeaters known only by the name a companion reported (#2239) - opt in with
  clientNodeNames.enabled. Names that companions report are used in hop resolution
  and shown on the map at an estimated position.
• Faster reach report (#2214) - opt in with retention.hopIndexDays. In the
  benchmark one reach scan went from 816 ms to 293 ms; the index took about 2 GB on a
  large database. The reach page now retries when the server is busy instead of
  showing an error (#2250).

Scopes and regions

• Scope Audit transport view (#2167, #2169) - next to what repeaters declare, a
  view of every repeater actually seen forwarding a scope.
• Region filter on Scope Audit (#2171), for both views.
• ?region= in a link works on every page (#2256, closes #2255) - not only on
  packets. Handy for links from a regional website or a Discord post.
• Region quick picks (#2170) - regionQuickPicks adds groups of regions to the
  region control, for example a country.

Pages and packets

• Pages no longer show an answer for an earlier visit (#2248, closes #2200, #2218,
  #2230) - when you clicked on quickly, a slow answer could replace the page you were
  on. Node analytics, analytics and the account page are fixed.
• Packets - the node filter keeps your pick on a slow click (#2244, closes #2243),
  grouped mode reloads when you change the observer filter (#2242, closes #2241), and
  live packets respect the channel filter (#2192, closes #2191).
• Analytics - Packets / Hour is readable on long histories (#2194, closes #2193),
  and Route Patterns hides blacklisted and hidden nodes (#2189, closes #2184).
• Node analytics say which period they cover (#2211) - for 30 days and All, the
  rates are computed over the packets in memory, and the page says from when. On one
  staging node the 30-day availability went from 19.3% to 83.3%, because days outside
  memory counted as silent before.
• Route view - "N hops" counts only real hops (#2240, closes #2206), and the sender
  and recipient decoded from the payload are marked as such (#2207).
• Favorites and home cards use the role's thresholds (#2215) - a repeater is shown
  as degraded after 24 hours and silent after 72, the same as on its node page,
  instead of after 1 and 24.
• The region filter matches packets loaded in the background (#2185) - with
  hotStartupHours, packets loaded after startup now carry the observer region.
• ?theme=dark or ?theme=light in a link sets the theme for that page load
  (#2251, closes #2236).

Faster and lighter

A large part of this release is work on the request path (#2146) and the packet
store's memory. What you notice:

• Startup to ready went from 31 to 21 seconds on one instance (#2181).
• The packet store uses less memory: for example 948 to 767 MB for the route
  patterns index (#2182) and 581 to 535 MB heap from sharing observer strings (#2183).
• The nodes list: p95 from 165 to 114 ms (#2165).
• Scope statistics for the last hour: 3.08 seconds to 16 ms (#2166).
• One reach query: 14.1 to 1.9 seconds (#2155).
• The ingestor lets live packets through between retention batches (#2153, closes
  #2151), and runs the database statistics refresh at most once a day, where one run
  took almost 8 minutes (#2158).
• Hidden browser tabs stop polling the server for the navigation counters (#2162).
• Opt-in analytics.pauseWhenIdle skips analytics nobody looks at (#2164).

The other pull requests in this group are #2147, #2156, #2157, #2163, #2186, #2187,
#2195 to #2199, #2201, #2222, #2224, #2246, #2247.

Docker and ingestor

• Run the services as a normal user (#2125, #2235) - set RUN_AS_UID and
  RUN_AS_GID. Without them the services run as root, as before.
• An interrupted background migration in the ingestor is retried at the next start
  instead of being marked as failed (#2161).

Tests, CI and docs

• Frontend coverage now counts every E2E suite (#2204, closes #2203), and the README
  badges update again (#2202, closes #2136).
• Local test runs no longer depend on the CI environment (#2160, closes #2159), and two
  flaky E2E checks are stable (#2179, #2180).
• The README is corrected and lists the optional features (#2205). SECURITY.md
  explains how to report a vulnerability (#2257).

Issues closed

29 issues, from the 76 pull requests in this range. Collected from each pull
request's closing references rather than from commit text.

• #661 feat: airtime utilization estimates per observer
• #1268 feat: enrich nodes with LoRa RF params from map.meshcore.io
• #1879 bug: /api/nodes region filter attributes by observer, not repeater home region
• #2098 iata-coords: none of the 42 observers' codes are in the table
• #2136 README badges stale since June: BADGE_PUSH_TOKEN returns 401
• #2146 Request-path latency grows with DB size
• #2148 perf: avoid allocation-class padding in retained observations
• #2151 perf: let live ingestion run between retention batches
• #2159 test: local validation assumes Actions environment and POSIX file modes
• #2177 Notifications: optional external node alerts feed
• #2184 nodeBlacklist is not applied to Route Patterns
• #2191 Packets page: live WebSocket packets ignore the channel filter
• #2193 Analytics: Packets / Hour chart unreadable on long-retention instances
• #2200 Analytics: older requests can overwrite newer filters or a later page visit
• #2203 Frontend coverage counts only 2 of the E2E suites
• #2206 Route view: DM sender/recipient rows drawn as ordinary hops
• #2218 bug(node analytics): delayed responses overwrite the page after navigation
• #2219 perf: cold region and area lookups still hold the packet-store lock
• #2221 perf: /api/nodes/{pubkey}/neighbors rebuilds the neighbor graph on the request path (~11 s)
• #2223 perf: /api/nodes/{pubkey}/paths runs two queries per transmission
• #2225 bug: zero-hop detection reads the path byte at the wrong offset for transport-direct adverts
• #2228 bug(accounts): make pending registration updates atomic and status-conditional
• #2230 bug(account): delayed session initialization replaces the page after navigation
• #2236 Feature Request: Dark Mode set via URL parameter
• #2237 server: data races between ingest and HTTP readers
• #2241 Packets: observer filter only takes effect after a page reload (grouped mode)
• #2243 Packets: picking a node suggestion sometimes does nothing
• #2255 ?region= in the URL only works on the packets page
• #2260 Node liveness counts ambiguous 1-byte hops: a silent repeater shows "Active"

Contributors

Pull request authors: @efiten (33), @A13xB0 (24), @nullrouten0 (6), @n30nex (4),
@anieto (3), @CoderNemesis26 (2), @hermes-gadget (2), @SaarMesh-Bot (2).

No co-authors other than tooling are credited in the commits.

Issues in this release were reported by @efiten (13), @n30nex (8), @anieto (2),
@A13xB0 (1), @dborup (1), @hermes-gadget (1), @Kpa-clawbot (1), @n1ghtBl00d (1),
@SaarMesh-Bot (1).

Upgrade notes

• No manual migration step.
• With user accounts on, keep a copy of users.db from before the upgrade (see above).
• Everything new is off until you set it: rfParams.nodesUrl,
  clientNodeNames.enabled, retention.hopIndexDays, analytics.pauseWhenIdle,
  regionQuickPicks, userManagement.notifications.adaptiveOffline and
  .externalAlerts, clientRxCoverage.requireLinkedCompanion,
  userManagement.mail.provider: "postal", and RUN_AS_UID / RUN_AS_GID for
  Docker. The example values are in config.example.json.

What's Changed

• fix(server): no SQL while holding PacketStore.mu by @efiten in #2147
• fix(server): drive the reach scan from the observations time window by @efiten in #2155
• perf(server): /paths reads canonical paths in batches and skips redundant collision checks by @efiten in #2157
• perf(server): battery history via the primary key, cached /api/perf row counts by @efiten in #2156
• perf(ingestor): refresh planner statistics 24h after the last refresh, not after every start by @efiten in #2158
• feat(rx-coverage): show where a companion drove and received nothing by @efiten in #2172
• feat(users): companion linking store (schema v6, device sessions, link challenges) by @efiten in #2174
• feat(server): companion linking routes, device tokens and bearer auth by @efiten in #2175
• feat(companions): linked-only ingest, account Companions and My coverage by @efiten in #2176
• feat(notify): optional external node alerts feed and the node.external event (#2177) by @efiten in #2178
• test: make local validation independent of runner environment by @n30nex in #2160
• fix(ingestor): record cancelled background migrations separately by @n30nex in #2161
• perf(store): drop the duplicate subpath count map by @A13xB0 in #2163
• perf(frontend): throttle nav stats to one refresh per 15 s, none in hidden tabs by @A13xB0 in #2162
• perf(server): share /api/nodes responses for identical queries for 15 s by @A13xB0 in #2165
• perf(scope-stats): keep the per-region count on the first_seen index by @A13xB0 in #2166
• feat(scope-audit): transport view of every repeater seen forwarding (#2142, API) by @A13xB0 in #2167
• feat(scope-audit): transport view on the Scope Audit page (#2142, page) by @A13xB0 in #2169
• fix(region-filter): setSelected notifies change listeners on a real change by @A13xB0 in #2168
• feat(region-filter): configurable region quick picks by @A13xB0 in #2170
• perf(analytics): opt-in pauseWhenIdle skips recomputes nobody reads by @A13xB0 in #2164
• test(e2e): assert the configured-scope row after the theme-refresh re-render by @efiten in #2180
• feat(scope-audit): observer region filter for both views by @A13xB0 in #2171
• (feat): UserManagement, Add Support for Postal Email Delivery Provider by @CoderNemesis26 in #2154
• test(map): settle the router's focus frame before the teardown snapshot by @A13xB0 in #2179
• perf(store): check observation duplicates without per-transmission maps by @A13xB0 in #2181
• fix(store): load the observer IATA in the background chunk loader by @A13xB0 in #2185
• perf(store): count subpaths per request instead of indexing every run by @A13xB0 in #2182
• perf(store): build hop context without a map or lowered copies by @A13xB0 in #2186
• perf(store): count path hops without decoding the JSON by @A13xB0 in #2187
• fix(hops): narrow only the directly heard hop to the observer region (#2098) by @efiten in #2188
• fix(#2191): apply the channel filter to live WebSocket packets by @SaarMesh-Bot in #2192
• fix(analytics): apply nodeBlacklist and hiddenNamePrefixes to Route Patterns (#2184) by @efiten in #2189
• feat(#2107): backfill unresolved channel/scope rows when the key set grows by @SaarMesh-Bot in #2133
• perf(store): share one copy of each observer string across observations by @A13xB0 in #2183
• fix(config): write the geo filter through a symlinked config.json by @nullrouten0 in #2137
• perf(store): cache an observation's parsed time in 8 bytes by @A13xB0 in #2196
• perf(store): build the relay-filtered candidate lists once by @A13xB0 in #2195
• ci: publish the README badges to a badges branch with GITHUB_TOKEN by @efiten in #2202
• ci: frontend coverage from every E2E suite, merged across the shards by @efiten in #2204
• docs(readme): correct stale claims and add the optional features by @efiten in #2205
• feat(observers): rolling 1 h TX airtime and a duty-cycle warning (#661) by @efiten in #2208
• fix(docker): run the Go services as an unprivileged user by @nullrouten0 in #2125
• perf(store): bulk loaders reuse their row scan targets by @A13xB0 in #2201
• perf(store): the decoded-payload cache drops fields nothing reads by @A13xB0 in #2198
• fix(nodes): place nodes in regions by zero-hop home regions (#1879) by @anieto in #2190
• feat(nodes): LoRa radio settings per node from the MeshCore map (#1268) by @efiten in #2209
• fix(nodes): node analytics rates cover the data in memory, and say so (#748) by @efiten in #2211
• fix(nodes): read the transport-direct path byte at offset 5 (#2225) by @anieto in #2234
• fix(docker): read .env last, and restore RUN_AS_UID/GID from positional parameters by @nullrouten0 in #2235
• fix(route-view): badge payload-decoded sender/recipient rows in the sequence renderer by @hermes-gadget in #2207
• fix(health): judge favorites and home status cards by the node's role by @nullrouten0 in #2215
• fix(route-view): "N hops" counts only true hops, not payload endpoints by @hermes-gadget in #2240
• perf(store): key byNode's de-dup sets by transmission ID by @A13xB0 in #2199
• perf(store): read resolved_path pubkeys in place, one copy each by @A13xB0 in #2197
• fix(analytics): Packets / Hour readable on long histories (#2193) by @A13xB0 in #2194
• fix(packets): re-fetch when the observer filter changes (#2241) by @efiten in #2242
• fix(packets): keep the node filter pick on a slow click or late search (#2243) by @efiten in #2244
• perf(reach): answer /nodes/{pk}/reach from a rolling hop index, not a path_json LIKE scan by @nullrouten0 in #2214
• 1-Byte Hop Path Inference for Packet and Map Analysis by @CoderNemesis26 in #2245
• fix(pages): drop late responses that belong to an earlier page visit (#2218, #2230, #2200) by @efiten in #2248
• fix(server): read StoreTx state under s.mu in grouped packets and other readers (#2237) by @efiten in #2246
• fix(store): resolve region and area sets before taking s.mu (#2219) by @efiten in #2247
• feat(theme): ?theme=dark|light URL parameter for one page load (#2236) by @efiten in #2251
• fix(accounts): keep pending registration updates atomic by @n30nex in #2253
• perf(ingestor): yield between committed retention batches by @n30nex in #2153
• chore: add SECURITY.md and OSS Scanner build files by @efiten in #2257
• fix(reach): retry a busy (429) reach report in the reach page by @nullrouten0 in #2250
• perf(paths): batch the resolved-path fallback for /nodes/{pk}/paths (#2223) by @efiten in #2224
• fix(observers): an observer's own transmissions are not receptions (#667) by @efiten in #2220
• perf(neighbors): rebuild the neighbor graph only in the background (#2221) by @efiten in #2222
• feat(nodes): name and show repeaters known only from the name they reported (#2238) by @efiten in #2239
• feat(nodes): flag nodes whose LoRa settings differ from nearby nodes (#1268) by @efiten in #2210
• feat(region-filter): honor ?region= in the URL on every page (#2255) by @anieto in #2256
• fix(liveness): count only evidence that can be the node alone (#2260) by @efiten in #2261
• feat(notify): adaptive offline threshold and certain node analytics (#2260) by @efiten in #2262
• docs(release-notes): v3.15.0 by @efiten in #2263

New Contributors

• @CoderNemesis26 made their first contribution in #2154
• @hermes-gadget made their first contribution in #2207

Full Changelog: v3.14.0...v3.15.0