-
Notifications
You must be signed in to change notification settings - Fork 20
Home
Note: DPS is currently released as a technical preview for evaluation. It is very much as work in progress and is not intended for commercial deployment at this time.
Distributed Publish Subscribe for the Internet of Things, which we will refer to as DPS in this document, is a new protocol that implements the pub/sub (publish/subscribe) communication pattern. The pub/sub pattern for device to device communication is simple and powerful. There are several existing pub/sub protocols seeing heavy use in IoT applications, perhaps most notably MQTT and DDS but there are numerous other. Two characteristics of pub/sub that make it attractive for IoT uses cases are support for loose-coupling between publishers and subscribers, and inherent support for point-to-multipoint messaging. There are generally two implementation approaches: brokered (e.g. MQTT), or multicast (e.g. DDS). In brokered pub/sub systems publishers and subscribers connect to a centralized server that routes publications to matching subscribers. In a multicast pub/sub system subscribers receive messages from all publishers and selectively forward matching publications up to the application. The disadvantage of a brokered approach is that the broker is single point of failure, must be 100% available, and scales linearly with bandwidth and processing capability of the broker. Also all messages do a round-trip through the broker which puts a lower bound on communication latency. Multicast pub/sub systems are hard to scale beyond a single subnet and work much better over wired than wireless networks.
Distributed Publish Subscribe as the name implies is a fully-distributed pub/sub framework. There is no broker, devices or applications (we will just call them nodes) running the DPS protocol form a dynamic multiply-connected mesh where each node functions as a message router. The DPS framework supports a topic string syntax that will be very familiar to MQTT users and also supports MQTT-like retained messages. The mesh is boot-strapped using IP multicast, a directory service, or by explicit URL. The DPS protocol is light-weight and amenable to implementation on very small devices such as sensors that primarily publish data. The DPS architecture is well suited for applications that leverage edge computing in combination with cloud-based analytics.
Superficially DPS looks like a broker based pub/sub protocol, some of this is intentional, such as using MQTT’s topic string wild-card syntax, but the architecture is quite different. In a brokered pub/sub system publishers and subscribers typically maintain a long term connection to the broker. This is often necessary because the broker is running in the cloud and the subscriber and publishers are typically running behind a firewall, possibly NAT’d, and must establish an outbound connection to the broker to be able to communicate. DPS does not maintain long term connections, in fact connections only last long enough to send a single subscription or publication message. DPS uses hop-by-hop routing to forward publications to subscribers in the network. A DPS node with multiple network interfaces can forward pub/sub messages from one interface to another, there is no need for an end-to-end network route. In a conventional pub/sub system, publishers and subscriber send topic strings to the broker. The broker can essentially see as clear text every topic that passes through. In theory the individual elements in topic strings could be sent as hashes but that is not done currently. In DPS all publication and subscriptions are implicitly hashed and node only routes publications to nodes that have matching subscribers so there is typically no single point through which all messages pass.
DPS like other pub/sub protocols expressed publications and subscriptions as structured text strings called topic strings. A topic string is a sequence of substrings delineated by a standalone separator character. In DPS almost any character or set of characters the publisher and subscriber agree on can be used as a separator. A publication matches a subscription if the substrings and separators in the publication are the same as the substrings and separators in the publication. Subscription topic strings can also include wildcard characters as described below. These are all valid publication topic strings:
foo/bar x,y,z 1.2.3 a/b/c?val=5
In the last example “/”, “?”, and “=” are separators. Separators must standalone, two or more consecutive separators are disallowed. Subscription topics strings have the same form as publication topics strings but can include wild-card characters. DPS uses the same wildcard characters as MQTT with the same meanings: the plus sign “+” wild card matches to any substring in the same position; the hash or pound sign “#” matches any number of trailing substrings. In DPS “+” and “#” are currently the only characters that are reserved. These are some valid wild-carded subscription topic strings:
+/bar x,+,z 1.# a/b/c?val=+
In MQTT and other pub/sub protocols a subscription or publication is a single topic string. A unique feature of DPS is that subscriptions and publications can have multiple topic strings. A subscription with more than one topic string will only match publications that have matching topic strings for all of topic strings in the subscription. As an example of how this might be used consider a set of devices that publish a topic string describing device type and a topic string describing the physical location of the device. An application could subscribe to all devices at a specific location by only specifying the location topic string, all devices of a specific type by only specifying the device type, or home in on a device with a specific type at specific location by using both topic strings in the same subscription. Another unique feature of DPS is that publisher control over the kinds of wild-card matches a subscriber is permitted to use. For example, a publisher can decide that wild-card matches must fully specify at least the first N elements in order to match. This offers a publisher control over wide-open wildcard subscriptions such as “+/#”, the most generic form allowed by DPS, that will match any publication with two or more elements.