Skip to content
a built-in, autonomous Mosquitto Cluster implementation. MQTT集群.
C Python HTML C++ Makefile CMake Other
Branch: master
Clone or download
Latest commit 1f5fb6a Jun 4, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmark
client
doc initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
docker
examples
img initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
installer merge with trunk 2018/05/28 May 28, 2018
lib merge with trunk 2018/05/28 May 28, 2018
logo
man merge with eclipse/mosquitto/develop/2018/04/24 Apr 24, 2018
misc/currentcost initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
security
service initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
snap initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
src merge with trunk 2018/05/28 May 28, 2018
test merge with trunk 2018/05/28 May 28, 2018
.gitignore initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
.travis.yml initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
CMakeLists.txt merge with trunk 2018/05/28 May 28, 2018
CONTRIBUTING.md initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
ChangeLog.txt merge with trunk 2018/05/28 May 28, 2018
LICENSE.txt
Makefile merge with eclipse/mosquitto/develop/2018/04/24 Apr 24, 2018
Mosquitto.podspec initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
THANKS.txt initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
about.html initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
aclfile.example initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
appveyor.yml
compiling.txt initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
config.h merge with trunk 2018/05/28 May 28, 2018
config.mk merge with trunk 2018/05/28 May 28, 2018
edl-v10
epl-v10 initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
libmosquitto.pc.in initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
libmosquittopp.pc.in
mosquitto.conf
notice.html
pskfile.example initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
pwfile.example
readme-windows.txt
readme-zh_cn.md Update readme-zh_cn.md Jun 4, 2018
readme.md Update readme.md Jun 4, 2018
set-version.sh
travis-configure.sh initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018
travis-install.sh initial implement based on eclipse/mosquitto-develop branch Jan 19, 2018

readme.md

Mosquitto with cluster

中文版自述
In a mosquitto cluster, clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed.
The cluster is full decentralized, autonomy system without any leader or key node, to make the system with a high availablity. E.g., each node has a fault rate with 1%, then a decentralized cluster with N nodes has a service availability which is 1-1%^N.

Usage

Install mosquitto on all of the nodes and write the addresses into mosquitto.conf, e.g.,

node_name node1
node_address 192.168.1.1:1883

node_name node2
node_address 192.168.1.2:1883

Then config the loadbalancer, take above adresses as real server address. It is strongly recommend to terminate TLS on the loadbalancer, and use plain TCP inside the cluster.

Installing

> git clone https://github.com/hui6075/mosquitto-cluster.git
> cd mosquitto-cluster && vi config.mk

# WITH_BRIDGE:=yes
WITH_CLUSTER:=yes

> make && make install

Cluster Specification

Traffic cycle avoid

In order to avoid infinite PUB/SUB forwarding, the publishes from other brokers will only send to client, the subscription and unsubscription from other brokers will not be forward.

Duplicate subscription avoid

Save local clients' subscriptions and unsubscriptions by a reference counter, only forward the subscription which is fresh for local broker, and only forward the unsubscription which is no longger been subscribed by local client.

Private messages

Some private messages was introduced in order to support cluster session and retain message, i.e.,
PRIVATE SUBSCRIBE
Fix header|PacketID|Topic Filter|QoS|ClientID|SubID

PRIVATE RETAIN
Fix header|Topic|QoS|[PacketID]|ClientID|SubID|OriginalRecvTime|Payload

SESSION REQ
Fix header|ClientID|Clean Session

SESSION RESP
Fix header|ClientID|PacketID|NRsubs|sub1(topic|qos)|...|subN|NRpubs|pub1(topic|state|dir|dup|qos|mid|payload)|...|pubN|

Cluster session support

For cluster session, SESSION REQ would be broadcast for each client CONNECT which has no previous context found in local broker, remote broker which has this client's context will kick-off this client and if clean session set to false, the remote broker would return this client's session include subscription, incomplete publishes with QoS=0/1 inside SESSION RESP. This feature could be disable in mosquitto.conf, in order to endure instantaneous large number of concurrent connections at system startup phase.

Cluster retain message support

For retain message, PRIVATE SUBSCRIBE would be broadcast for each client subscription that is fresh for local broker, and if there exists a retain message, remote broker would return the retain message inside PRIVATE RETAIN. Client will receive the most recent retain message inside the cluster after a frozen window which can be configure inside mosquitto.conf.

QoS support

For QoS, all the publishes set with it's original QoS inside the cluster, and process with QoS=0, to saves the inside traffic. The actual QoS send to client decide by the broker which client connected with.

Other features

All validation, utf-8, ACL checking has disabled for the messages inside the cluster, in order to improve cluster efficiency.

Once CONNECT success with other broker, local broker will send all the local clients' subscription to remote broker, to avoid subscription loss while some brokers down and up.

The cluster current also support node/subscription recover, crash detection.

Design Philosophy

Mosquitto cluster goals

Mosquitto cluster is a distributed implementation of Mosquitto brokers with following goals:
Scalable in a horizontal fashion, which means you can expand cluster capability by increase number of brokers.
Provide continuous service under single point of failure.
Take on the role of one logical MQTT broker for millions of MQTT clients.

image

Pic1. Mosquitto Cluster Overview

While receive client subscriptin: Notification other brokers that the topic has been subscribed by me,
While receive client publish: route the message to the correct brokers which subscribed this topic.

image

Pic2. Mosquitto Cluster Private Messages

There's only one logic channel between 2 brokers, in order to reuse this channel for multi clients, some private messages has been introduced, include client id, subscription id, raw timestamp and etc., which help the broker to make a correct route decision.

image

Pic3. Mosquitto Cluster Internal Message Flow

Forwarding rules:
Forward local client subscriptions to the broker which is a fresh guy for the cluster or just recovered from a fault.
DO NOT forward internal messages to any broker.
Only broadcast local fresh subscription to other brokers.
Broadcast unsubscription until a topic is no longer subscribed by any local client.
Session messages broadcasting can be disable by configuration.
DO NOT forward PUB/SUBs under $SYS.

Benchmark

Using krylovsk/mqtt-benchmark as the benchmark tool to give a simply testing for the cluster. The bandwidth stop increase with 3 or more brokers due to client machine's bottleneck.
image

Pic4. Mean bandwidth of Mosquitto cluster

n=10k means 10000 messages to send per client, c=100 means 100 clients to start. MsgSize=1000bytes, QoS=2.

A more detailed test report is available under: https://github.com/hui6075/mosquitto/tree/develop/benchmark

Study Material

image

Pic5. Mosquitto code analyze(benchmark/mosquitto_code_analyze.jpg, w/o cluster)

Other Projects

You can’t perform that action at this time.