Skip to content

v0.2.0 — sessions and QoS 1/2 across the cluster

Choose a tag to compare

@1995parham 1995parham released this 26 Oct 23:09

The cluster now keeps the promises a single node always did: a session follows its device between nodes, and a QoS 1 or 2 message acknowledged on one node reaches its subscribers on another even if the link between them drops.

What is new

  • Sessions survive a restart and follow a device between nodes (#8). Subscriptions and the queue of QoS 1/2 messages for an absent client live in JetStream KV, so a device that reconnects to a different node, or to a restarted one, gets both back. Only sessions that outlive their connection are stored: a clean session costs no writes.
  • Cross-node session takeover (#13). A client id connecting on a second node disconnects the first, through one mast.session.> subscription per node rather than one per client.
  • Cross-node QoS 1 and 2 are at-least-once (#11). A QoS 1 or 2 publish is stored on a replicated stream before it is acknowledged, and every node reads that stream through one consumer of its own. A node cut off from the core catches up on what it missed instead of losing it. See the delivery guarantees guide for what is still best-effort.
  • MQTT 5 message properties cross the cluster and survive storage, including message expiry counted from publication.
  • QoS 2 under load (#18). The embedded MQTT library wrote every acknowledgement with its own syscall; at 100k QoS 2 deliveries a second a node fell seconds behind. mast now uses mastmq/mochi v2.7.14, our maintained fork of mochi-mqtt, which coalesces them.

Fixed

  • Each message is delivered once, to the clients it was meant for: overlapping filters and shared groups used to produce duplicates.
  • Client ids are scoped to their tenant, so two tenants can use the same id.
  • A shutdown racing a connecting client could deadlock the server and leave a pod that would not terminate (#9).
  • The edge role could not open the store, and an edge started before the core's JetStream now waits instead of refusing to start.
  • The core serves metrics, health and pprof too.
  • NATS slow-consumer drops are reported as mast_nats_slow_consumers_total instead of swallowed (#12).
  • The authorization cache is keyed by credential and the HTTP backend requires a timeout.
  • Two nodes creating the same bucket at once no longer kills the loser.

Performance

Session notices ride one subscription per node instead of one per client, the three locks every message waited on are off the hot path, and decoding a plain subject no longer splits it into tokens.

Upgrading from v0.1.0

  • Two new config sections, both defaulting on: [session] expiry = "24h" and [fabric] durable_qos = true, durable_max_age = "5m". The core needs persistent JetStream storage for the new mast_qos stream and the session buckets, which it already had for retained messages.
  • Sessions held in memory by v0.1.0 are not carried over: clients with persistent sessions resubscribe once after the upgrade.
  • Linux amd64 and arm64 tarballs and a checksum file are attached to this release, for the Ansible collection.