Skip to content

v0.3.0 — the core keeps messages for sleeping devices

Latest

Choose a tag to compare

@1995parham 1995parham released this 17 Nov 14:19
· 2 commits to main since this release

The core tier now keeps QoS 1 and 2 messages for sleeping devices, so a rolling restart of the edges no longer loses what was published to them, and a core holds a 300,000-device fleet of sleeping sessions in about 650 MB.

What is new

  • The cores queue for absent sessions. Each core watches the sessions bucket and reads the QoS stream through one consumer of its own, writing messages for away sessions to a new mast_offline stream (one subject per session, newest 100). Before this, only the edge a device disconnected from queued for it, and a rolling restart of the edges lost 10,000 of 10,000 messages to sleeping devices in the drill. Now all 10,000 arrive.
  • The memory limit comes from the container. When GOMEMLIMIT is not set, mast sets Go's limit to 90% of the cgroup limit. Go does not read it by itself, and cores were OOM-killed instead of collecting harder.
  • Sized for 300,000 sleeping devices. The queuer's topic index costs about 430 bytes per away session instead of 2.8 KB. At that scale give cores at least 2 CPUs: on one, garbage collection lifts QoS 1 p99 from about 4 ms to 60–80 ms.

Fixed

  • A device's backlog is trimmed only after the device acknowledges it, so a connection that drops mid-drain no longer loses it.
  • A device that comes back is judged by when each message was stored, so it no longer misses what was published while it was away.
  • During a store outage the queuer retries until the write lands instead of acknowledging after 30 seconds.
  • A late disconnect can no longer mark a fresh connection's session away, nor recreate a session a clean start deleted.
  • A reply lost from a batched direct get is detected instead of read as the end of the queue.
  • A core with no client address now listens for routes.

Upgrading from v0.2.0

  • Upgrade the cores first. While old edges still run, their session records read as away, which can queue duplicates but loses nothing. The v0.2.0 offline bucket is delivered and then removed.
  • No config changes. The cores create the mast_offline stream on startup.
  • Known gap: when an edge is killed outright, its sleeping sessions stay marked present, and what is published to them is lost until they reconnect elsewhere. See the delivery guarantees guide.