Skip to content

Releases

v0.3.0 — the core keeps messages for sleeping devices

Choose a tag to compare

@1995parham 1995parham released this 17 Nov 14:19

The core tier now keeps QoS 1 and 2 messages for sleeping devices, so a rolling restart of the edges no longer loses what was published to them, and a core holds a 300,000-device fleet of sleeping sessions in about 650 MB.

What is new

  • The cores queue for absent sessions. Each core watches the sessions bucket and reads the QoS stream through one consumer of its own, writing messages for away sessions to a new mast_offline stream (one subject per session, newest 100). Before this, only the edge a device disconnected from queued for it, and a rolling restart of the edges lost 10,000 of 10,000 messages to sleeping devices in the drill. Now all 10,000 arrive.
  • The memory limit comes from the container. When GOMEMLIMIT is not set, mast sets Go's limit to 90% of the cgroup limit. Go does not read it by itself, and cores were OOM-killed instead of collecting harder.
  • Sized for 300,000 sleeping devices. The queuer's topic index costs about 430 bytes per away session instead of 2.8 KB. At that scale give cores at least 2 CPUs: on one, garbage collection lifts QoS 1 p99 from about 4 ms to 60–80 ms.

Fixed

  • A device's backlog is trimmed only after the device acknowledges it, so a connection that drops mid-drain no longer loses it.
  • A device that comes back is judged by when each message was stored, so it no longer misses what was published while it was away.
  • During a store outage the queuer retries until the write lands instead of acknowledging after 30 seconds.
  • A late disconnect can no longer mark a fresh connection's session away, nor recreate a session a clean start deleted.
  • A reply lost from a batched direct get is detected instead of read as the end of the queue.
  • A core with no client address now listens for routes.

Upgrading from v0.2.0

  • Upgrade the cores first. While old edges still run, their session records read as away, which can queue duplicates but loses nothing. The v0.2.0 offline bucket is delivered and then removed.
  • No config changes. The cores create the mast_offline stream on startup.
  • Known gap: when an edge is killed outright, its sleeping sessions stay marked present, and what is published to them is lost until they reconnect elsewhere. See the delivery guarantees guide.

v0.2.0 — sessions and QoS 1/2 across the cluster

Choose a tag to compare

@1995parham 1995parham released this 26 Oct 23:09

The cluster now keeps the promises a single node always did: a session follows its device between nodes, and a QoS 1 or 2 message acknowledged on one node reaches its subscribers on another even if the link between them drops.

What is new

  • Sessions survive a restart and follow a device between nodes (#8). Subscriptions and the queue of QoS 1/2 messages for an absent client live in JetStream KV, so a device that reconnects to a different node, or to a restarted one, gets both back. Only sessions that outlive their connection are stored: a clean session costs no writes.
  • Cross-node session takeover (#13). A client id connecting on a second node disconnects the first, through one mast.session.> subscription per node rather than one per client.
  • Cross-node QoS 1 and 2 are at-least-once (#11). A QoS 1 or 2 publish is stored on a replicated stream before it is acknowledged, and every node reads that stream through one consumer of its own. A node cut off from the core catches up on what it missed instead of losing it. See the delivery guarantees guide for what is still best-effort.
  • MQTT 5 message properties cross the cluster and survive storage, including message expiry counted from publication.
  • QoS 2 under load (#18). The embedded MQTT library wrote every acknowledgement with its own syscall; at 100k QoS 2 deliveries a second a node fell seconds behind. mast now uses mastmq/mochi v2.7.14, our maintained fork of mochi-mqtt, which coalesces them.

Fixed

  • Each message is delivered once, to the clients it was meant for: overlapping filters and shared groups used to produce duplicates.
  • Client ids are scoped to their tenant, so two tenants can use the same id.
  • A shutdown racing a connecting client could deadlock the server and leave a pod that would not terminate (#9).
  • The edge role could not open the store, and an edge started before the core's JetStream now waits instead of refusing to start.
  • The core serves metrics, health and pprof too.
  • NATS slow-consumer drops are reported as mast_nats_slow_consumers_total instead of swallowed (#12).
  • The authorization cache is keyed by credential and the HTTP backend requires a timeout.
  • Two nodes creating the same bucket at once no longer kills the loser.

Performance

Session notices ride one subscription per node instead of one per client, the three locks every message waited on are off the hot path, and decoding a plain subject no longer splits it into tokens.

Upgrading from v0.1.0

  • Two new config sections, both defaulting on: [session] expiry = "24h" and [fabric] durable_qos = true, durable_max_age = "5m". The core needs persistent JetStream storage for the new mast_qos stream and the session buckets, which it already had for retained messages.
  • Sessions held in memory by v0.1.0 are not carried over: clients with persistent sessions resubscribe once after the upgrade.
  • Linux amd64 and arm64 tarballs and a checksum file are attached to this release, for the Ansible collection.

v0.1.0 — first tagged release

Choose a tag to compare

@1995parham 1995parham released this 21 Sep 21:31

A multi-tenant MQTT broker built on core NATS. One binary: run it standalone at a customer site, or split it into core and edge roles for a cluster.

Why it exists

Every open-source MQTT broker that does real multi-tenancy asks you to give something up. EMQX adopted the Business Source License in 5.9, so forming a cluster from more than one node needs a paid licence key. BifroMQ has genuine native multi-tenancy but runs on the JVM. VerneMQ's source is Apache 2.0 while its official packages and images are governed by a EULA that charges for commercial use. RabbitMQ gives you vhosts but supports neither QoS 2 nor shared subscriptions. Mosquitto, NanoMQ, FlashMQ and mochi-mqtt do not cluster at all.

mast is the combination that did not otherwise exist: MQTT 5 with shared subscriptions, tenant isolation, free clustering, no JVM, and a single binary.

What works

Verified by end-to-end tests that drive a real MQTT client against a real node, and by a three-process cluster of one core and two edge nodes:

  • MQTT 3.1.1 and 5.0, QoS 0 through 2, honoured at the level the subscriber negotiated
  • Retained messages, stored cluster-wide in JetStream KV rather than per node
  • Persistent sessions — a client reconnecting with clean_start=false receives what it missed while away
  • Wills, delivered on an ungraceful disconnect and suppressed by a clean DISCONNECT
  • Shared subscriptions — $share/<group>/<filter> maps onto a NATS queue group, and each message reaches exactly one member
  • Wildcards, including foo/# matching foo itself, which NATS's > does not do on its own
  • Topics containing ., a leading /, empty levels and non-ASCII, all arriving byte-identical

Tenant isolation is structural, not a convention. Every topic is mounted under its tenant before validation, authorization or subscription ever sees it, and unmounted again on the way out. A client never learns the prefix exists, and two tenants publishing to a/b cannot reach each other.

How it is put together

Core NATS moves messages; a key-value store holds keys. Live fan-out rides core NATS, which handles tens of millions of subjects. Sessions, retained messages and offline queues live in a handful of JetStream KV buckets.

What mast deliberately never does is create a JetStream consumer per subscription. Consumers are Raft state machines, a server holds on the order of two thousand of them, and a real fleet would want hundreds of thousands. Durable state has to scale as keys, not as consensus groups.

Replacing EMQX

mast speaks EMQX v5's HTTP auth wire, including is_superuser, so an auth service written for EMQX works unchanged. Topics reach the authorization endpoint unmounted, so existing ACL rules keep working. This release has replaced a live EMQX deployment in a staging environment with no changes to any consumer.

Authentication comes two ways: ask an HTTP service, or verify a signed JWT locally. They compose — authenticate from a token while topic decisions still go to a policy server.

Install

$ docker run --rm -p 1883:1883 ghcr.io/mastmq/mast:0.1.0
$ go install github.com/mastmq/mast/cmd/mast@v0.1.0

Helm charts live in mastmq/charts.

$ mast                  # all-in-one: MQTT and storage, no cluster
$ mast --role=core      # Raft and the KV buckets
$ mast --role=edge      # MQTT listeners, joins the core as a leaf node

You do not need the split until you are big. Up to roughly ten nodes, run every process identical and let them mesh.

Known gaps

Gaps are tracked as issues rather than described as a roadmap. See the parity label for what an MQTT client is entitled to assume and does not yet get.

  • Sessions do not survive a broker restart or move between nodes (#8). A session resumes for the lifetime of the node holding it; it is not yet durable across a restart.
  • Throughput has not been characterised. The benchmarking so far measured the load generator rather than the broker — the generator saturated while mast sat at 63–115 millicores. Treat the absence of a number here as deliberate; there will be one when it is defensible.
  • There is no dashboard and no management API, and no plan for a rule engine or data bridges.
  • No connection rate limiting yet.

Apache 2.0.