Repository navigation
Releases
v0.3.0 — the core keeps messages for sleeping devices
The core tier now keeps QoS 1 and 2 messages for sleeping devices, so a rolling restart of the edges no longer loses what was published to them, and a core holds a 300,000-device fleet of sleeping sessions in about 650 MB.
What is new
- The cores queue for absent sessions. Each core watches the sessions bucket and reads the QoS stream through one consumer of its own, writing messages for away sessions to a new
mast_offlinestream (one subject per session, newest 100). Before this, only the edge a device disconnected from queued for it, and a rolling restart of the edges lost 10,000 of 10,000 messages to sleeping devices in the drill. Now all 10,000 arrive. - The memory limit comes from the container. When
GOMEMLIMITis not set, mast sets Go's limit to 90% of the cgroup limit. Go does not read it by itself, and cores were OOM-killed instead of collecting harder. - Sized for 300,000 sleeping devices. The queuer's topic index costs about 430 bytes per away session instead of 2.8 KB. At that scale give cores at least 2 CPUs: on one, garbage collection lifts QoS 1 p99 from about 4 ms to 60–80 ms.
Fixed
- A device's backlog is trimmed only after the device acknowledges it, so a connection that drops mid-drain no longer loses it.
- A device that comes back is judged by when each message was stored, so it no longer misses what was published while it was away.
- During a store outage the queuer retries until the write lands instead of acknowledging after 30 seconds.
- A late disconnect can no longer mark a fresh connection's session away, nor recreate a session a clean start deleted.
- A reply lost from a batched direct get is detected instead of read as the end of the queue.
- A core with no client address now listens for routes.
Upgrading from v0.2.0
- Upgrade the cores first. While old edges still run, their session records read as away, which can queue duplicates but loses nothing. The v0.2.0 offline bucket is delivered and then removed.
- No config changes. The cores create the
mast_offlinestream on startup. - Known gap: when an edge is killed outright, its sleeping sessions stay marked present, and what is published to them is lost until they reconnect elsewhere. See the delivery guarantees guide.
v0.2.0 — sessions and QoS 1/2 across the cluster
The cluster now keeps the promises a single node always did: a session follows its device between nodes, and a QoS 1 or 2 message acknowledged on one node reaches its subscribers on another even if the link between them drops.
What is new
- Sessions survive a restart and follow a device between nodes (#8). Subscriptions and the queue of QoS 1/2 messages for an absent client live in JetStream KV, so a device that reconnects to a different node, or to a restarted one, gets both back. Only sessions that outlive their connection are stored: a clean session costs no writes.
- Cross-node session takeover (#13). A client id connecting on a second node disconnects the first, through one
mast.session.>subscription per node rather than one per client. - Cross-node QoS 1 and 2 are at-least-once (#11). A QoS 1 or 2 publish is stored on a replicated stream before it is acknowledged, and every node reads that stream through one consumer of its own. A node cut off from the core catches up on what it missed instead of losing it. See the delivery guarantees guide for what is still best-effort.
- MQTT 5 message properties cross the cluster and survive storage, including message expiry counted from publication.
- QoS 2 under load (#18). The embedded MQTT library wrote every acknowledgement with its own syscall; at 100k QoS 2 deliveries a second a node fell seconds behind. mast now uses
mastmq/mochiv2.7.14, our maintained fork of mochi-mqtt, which coalesces them.
Fixed
- Each message is delivered once, to the clients it was meant for: overlapping filters and shared groups used to produce duplicates.
- Client ids are scoped to their tenant, so two tenants can use the same id.
- A shutdown racing a connecting client could deadlock the server and leave a pod that would not terminate (#9).
- The edge role could not open the store, and an edge started before the core's JetStream now waits instead of refusing to start.
- The core serves metrics, health and pprof too.
- NATS slow-consumer drops are reported as
mast_nats_slow_consumers_totalinstead of swallowed (#12). - The authorization cache is keyed by credential and the HTTP backend requires a timeout.
- Two nodes creating the same bucket at once no longer kills the loser.
Performance
Session notices ride one subscription per node instead of one per client, the three locks every message waited on are off the hot path, and decoding a plain subject no longer splits it into tokens.
Upgrading from v0.1.0
- Two new config sections, both defaulting on:
[session] expiry = "24h"and[fabric] durable_qos = true,durable_max_age = "5m". The core needs persistent JetStream storage for the newmast_qosstream and the session buckets, which it already had for retained messages. - Sessions held in memory by v0.1.0 are not carried over: clients with persistent sessions resubscribe once after the upgrade.
- Linux amd64 and arm64 tarballs and a checksum file are attached to this release, for the Ansible collection.
v0.1.0 — first tagged release
A multi-tenant MQTT broker built on core NATS. One binary: run it standalone at a customer site, or split it into core and edge roles for a cluster.
Why it exists
Every open-source MQTT broker that does real multi-tenancy asks you to give something up. EMQX adopted the Business Source License in 5.9, so forming a cluster from more than one node needs a paid licence key. BifroMQ has genuine native multi-tenancy but runs on the JVM. VerneMQ's source is Apache 2.0 while its official packages and images are governed by a EULA that charges for commercial use. RabbitMQ gives you vhosts but supports neither QoS 2 nor shared subscriptions. Mosquitto, NanoMQ, FlashMQ and mochi-mqtt do not cluster at all.
mast is the combination that did not otherwise exist: MQTT 5 with shared subscriptions, tenant isolation, free clustering, no JVM, and a single binary.
What works
Verified by end-to-end tests that drive a real MQTT client against a real node, and by a three-process cluster of one core and two edge nodes:
- MQTT 3.1.1 and 5.0, QoS 0 through 2, honoured at the level the subscriber negotiated
- Retained messages, stored cluster-wide in JetStream KV rather than per node
- Persistent sessions — a client reconnecting with
clean_start=falsereceives what it missed while away - Wills, delivered on an ungraceful disconnect and suppressed by a clean
DISCONNECT - Shared subscriptions —
$share/<group>/<filter>maps onto a NATS queue group, and each message reaches exactly one member - Wildcards, including
foo/#matchingfooitself, which NATS's>does not do on its own - Topics containing
., a leading/, empty levels and non-ASCII, all arriving byte-identical
Tenant isolation is structural, not a convention. Every topic is mounted under its tenant before validation, authorization or subscription ever sees it, and unmounted again on the way out. A client never learns the prefix exists, and two tenants publishing to a/b cannot reach each other.
How it is put together
Core NATS moves messages; a key-value store holds keys. Live fan-out rides core NATS, which handles tens of millions of subjects. Sessions, retained messages and offline queues live in a handful of JetStream KV buckets.
What mast deliberately never does is create a JetStream consumer per subscription. Consumers are Raft state machines, a server holds on the order of two thousand of them, and a real fleet would want hundreds of thousands. Durable state has to scale as keys, not as consensus groups.
Replacing EMQX
mast speaks EMQX v5's HTTP auth wire, including is_superuser, so an auth service written for EMQX works unchanged. Topics reach the authorization endpoint unmounted, so existing ACL rules keep working. This release has replaced a live EMQX deployment in a staging environment with no changes to any consumer.
Authentication comes two ways: ask an HTTP service, or verify a signed JWT locally. They compose — authenticate from a token while topic decisions still go to a policy server.
Install
$ docker run --rm -p 1883:1883 ghcr.io/mastmq/mast:0.1.0
$ go install github.com/mastmq/mast/cmd/mast@v0.1.0Helm charts live in mastmq/charts.
$ mast # all-in-one: MQTT and storage, no cluster
$ mast --role=core # Raft and the KV buckets
$ mast --role=edge # MQTT listeners, joins the core as a leaf nodeYou do not need the split until you are big. Up to roughly ten nodes, run every process identical and let them mesh.
Known gaps
Gaps are tracked as issues rather than described as a roadmap. See the parity label for what an MQTT client is entitled to assume and does not yet get.
- Sessions do not survive a broker restart or move between nodes (#8). A session resumes for the lifetime of the node holding it; it is not yet durable across a restart.
- Throughput has not been characterised. The benchmarking so far measured the load generator rather than the broker — the generator saturated while mast sat at 63–115 millicores. Treat the absence of a number here as deliberate; there will be one when it is defensible.
- There is no dashboard and no management API, and no plan for a rule engine or data bridges.
- No connection rate limiting yet.
Apache 2.0.