munet

The munet libraries use the muwerk scheduler to provide a comprehensive set of network functionality: WiFi connection, Access Point Mode, NTP time sync, OTA software update and MQTT communication for ESP8266 and ESP32 chips with a minimum of code:

#define __ESP__   // Platform define, add #define __ESP32__ for ESP32 (see dependencies)
#include "scheduler.h"
#include "net.h"
#include "mqtt.h"
#include "ota.h"

ustd::Scheduler sched;
ustd::Net net(LED_BUILTIN);
ustd::Mqtt mqtt;
ustd::Ota ota;

void appLoop();

void setup() {
    net.begin(&sched);  // connect to WiFi and sync NTP time, credentials read from ESP file system, (s.b.)
    mqtt.begin(&sched); // connect to MQTT server
    ota.begin(&sched);  // enable OTA updates

    int tID = sched.add(appLoop, "main");  // create task for your app code
}

void appLoop() {
    // your code goes here.
}

// Never add code to this loop, use appLoop() instead.
void loop() {
    sched.loop();
}

The library provides:

WiFi station, access point or both using configuration data from LittleFS/SPIFFS file system (s.b.). Connection to the WiFi network is established automatically. The library handles reconnect and error recovery gracefully.
Over-the-air (OTA) update is supported with one line of code [optional]
Time synchronization with NTP servers, including daylight saving handling [optional]
Connection to an MQTT server (via PubSubClient) [optional] This transparently connects the pub/sub inter-task communication that is provided by the muwerk scheduler with extern MQTT publishers and subscribers. Messages between muwerk tasks are published to the external MQTT server, and muwerk tasks can transparently subscribe to both other tasks on the same ESP and external topics via the MQTT interface.
MCUs without network hardware can be connected to a second system with network hardware via a serial link (MuSerial.h). The two connected systems appear as one system to the outside world, both can publish/subscribe. pub/sub message synchronization is handled automatically via serial link.

Dependencies

munet relies only on:

ustd. Check documentation for required platform defines.
muwerk
PubSubClient
Arduino_JSON. Note: Earlier versions used a different lib: ArduinoJson.

munet component	depends on ustd	muwerk	Arduino_JSON	PubSubClient	Network required	Platforms
Net.h	x	x	x		x	ESP8266, ESP32
Ota.h	x	x	x		x	ESP8266, ESP32
Mqtt.h	x	x	x	x	x	ESP8266, ESP32
MuSerial.h	x	x			serial link between two muwerk nodes	All (only serial port required)

Breaking Changes at Version 0.3.0

All network classes in Version 3.0.0 have been deeply refactored in order to provide more functionality and better reliability. Nevertheless there are some breaking changes.

Changed Method Signatures

ustd::Net has now two different Net::begin methods. The default one (selected when invoking net.begin(&sched);) starts the network based on configuration stored in a configuration file located at the filesystem of the device. If the device implements means for managing the configuration, this is the most versatile method. There is still a Net::begin method with a function signature similar to the older version that is intended for hardcoded devices. When starting the network with the other Net::begin method, the configuration is stored in memory and connot be changed any more at runtime.

Since the other network classes have also own configuration files when needed, the support for querying information about network services by publishing net/services/get has been removed.

The format of the net/network/state message has been extended to cover the new features, but is still compatible with older versions.

Also the interface of Mqtt::begin has changed in order to resemble the fact that all passed options are the defaults for options not set in the configuration file mqtt.json.

Changed Configuration File Format

When the device starts for the first time with the new munet library, it will detect an old configuration file and will migrate the content to the new format automatically. After the conversion, the filesystem will contain two configuration files:

net.json - the migrated version of the original net.json
mqtt.json - the new configuration file for MQTT if the mqtt service was defined in the old configuration file.

See the documentation below for details about the supported configuration options in each file.

Retained Messages

All messages published by the older version of ustd::Mqtt were flagged with the RETAINED flag. This default behaviour has been changed: It is now possible to configure if the default behaviour should be to flag the messages or not. The default is false but if the configuration file was created by migrating an older net.json, the old behaviour will be preserved in order to maintain compatibility of the device (See configuration option alwaysRetain in mqtt.json).

When this option is false (the default setting), there is a new way (for explicit MQTT topics) to specify that a message shall be published as RETAINED: instead of prefixing the topic with one exclamation point, the topic shall be prefixed with double exclamation points. E.g.: !!homeassistant/config.

Configuration

All muwerk network and mqtt configuration is stored in json formatted files in the LittleFS/SPIFFS file system of the ESP chip. In order to initialize a filesystem on a specific device, create a directory in your local file system of your project named data and place all initial configuration files like net.json or mqtt.json there.

Using platformio, the initial file system containing all files in your data directory is saved to the ESP chip by executing the following commands:

pio run -t buildfs
pio run -t updatefs

'''Note:''' This project has moved from SPIFFS (deprecated) to LittleFS for ESP8266. To continue to use SPIFFS on ESP8266 systems, define USTD_OPTION_FS_FORCE_SPIFFS. See USTD for more information about configuration options and platform defines.

In order to be able to create LittleFS file system images with platformio, your platformio.ini currently needs to contain:

board_build.filesystem = littlefs

SPIFFS and LittleFS are not compatible, if the library is updated, a new file system needs to be created and upload with pio run -t buildfs and pio run -t uploadfs.

Since ESP32 currently does not (yet) support LittleFS, ESP32 projects automatically use SPIFFS for the time being.

Minimal Required Configuration Files

The following minimum configuration files are needed in order to connect to a WiFi network and have mqtt up and running.

Minimum `net.json`

{
    "version": 1,
    "mode": "station",
    "hostname": "my-host-name",
    "station": {
        "SSID": "my-network-SSID",
        "password": "myS3cr3t"
    },
    "services": {
        "ntp": {
            "host": ["time.nist.gov"],
            "dstrules": "CET-1CEST,M3.5.0,M10.5.0/3"
        }
    }
}

Minimum `mqtt.json`

{
    "host":"my-mqtt-hostname"
}

Network Configuration

The network configuration is stored in a file named net.json.

Sample `net.json`

{
    "version": 1,
    "mode": "station",
    "hostname": "muwerk-${macls}",
    "station": {
        "SSID": "my-network-SSID",
        "password": "myS3cr3t",
        "address": "",
        "netmask": "",
        "gateway": "",
        "maxRetries": 40,
        "connectTimeout": 15,
        "rebootonFailure": true
    },
    "ap": {
        "SSID": "muwerk-${macls}",
        "password": "",
        "address": "",
        "netmask": "",
        "gateway": "",
        "channel": 1,
        "hidden": false
    },
    "services": {
        "dns": {
            "host": []
        },
        "ntp": {
            "host": [
                "time.nist.gov",
                "ptbtime1.ptb.de",
                "ptbtime2.ptb.de",
                "ptbtime3.ptb.de"
            ],
            "dstrules": "CET-1CEST,M3.5.0,M10.5.0/3"
        }
    }
}

Configuration Options Placeholder

Some of the configuration options support the use of placeholders in order to allow values that are specific to a certain devince without the need to create separate configuration files. Placeholders are written in the form of ${PLACEHOLDER}.

The following options allow the the of placeholders:

The hostname of the device. The default value of this hostname also uses a plceholder: muwerk-${macls}
The SSID for access point mode. Also here the default value uses a plceholder: muwerk-${macls}

The following placeholders are currently available:

mac: full mac address
macls: last 4 digits of mac address
macfs: first 4 digits of mac address

Top Level Configuration Options

Field	Usage
`version`	The configuration format version number. Current version is `1`. This field is mandatory.
`deviceID`	Unique device ID - will be automatically generated and saved on first start. Useful when replacing a device
`mode`	Operating mode. Can be: `off`, `ap`, `station` or `both`. Default is `ap`
`hostname`	Hostname the device will use and report to other services. May also be used to querythe DHCP server
`ap`	Configuration options for access point mode. See description below.
`station`	Configuration options for network station mode. See description below.
`services`	Configuration options for network services. See description below.

Configuration Options for Access Point Mode

The following options are stored in the ap object and apply to access point mode and dual mdoe.

Field	Usage
`SSID`	Network name of the wireless network the ESP will host
`password`	Wireless network password
`address`	Static IP address. If not defined, the default of the library is taken - usually `192.168.4.1`
`netmask`	Netmask of static IP address. Must be defined if `address` is also defined.
`gateway`	Default gateway. Does not really make sense in AP mode, but must be specified.
`channel`	Channel used for AP mode. If not specified, channel 1 is used.
`hidden`	If `true`, the network created by the AP is hidden. Default is `false`

Configuration Options for Network Station Mode

The following options are stored in the station object and apply to network station mode and dual mdoe.

Field	Usage
`SSID`	Network name of the wireless network the ESP will join
`password`	Wireless network password
`address`	Static IP address. If not defined, the address is obtained via DHCP.
`netmask`	Netmask of static IP address. Must be defined if `address` is also defined.
`gateway`	Default gateway. Does not really make sense in AP mode, but must be specified.
`maxRetries`	Maximum number of retries before giving up (and rebooting). Default is `40`
`connectTimeout`	Connection timeout in seconds. Default is 15 seconds.
`rebootonFailure`	If `true` the system reboots after reaching `maxRetries` connection failures. Default is `true`

Configuration Options for Network Service DNS Client

The DNS client is configured with an object named dns in the services object. The DNS client is only used in network station or dual mode.

Field	Usage
`host`	Array of hostnames/ip of DNS servers If empty the provided DHCP value is used

Configuration Options for Network Service NTP Client

The NTP client is configured with an object named ntp in the services object. The NTP client is only used in network station or dual mode.

Field	Usage
`host`	Array of hostnames/ip of NTP time servers from which the device synchronizes it's time. If empty the DHCP value is used
`dstrules`	optional timezone and daylight saving rules in unix format

Network Message Interface

Incoming

Topic	Message Body	Description
`net/network/get`		Returns a network information object in json format in a message with topic `net/network`
`net/network/control`	`<commands>`	Starts, stops or restarts the network (put `start`, `stop` or `restart` in the message body)
`net/networks/get`	`<options>`	Requests a WiFi network scan. The list is returned in a message with topic `net/networks`. The additional options `sync` and/or `hidden` can be sent in the body.

Outgoing

Topic	Message Body	Description
`net/network`	`{...}`	The current network state as JSON object
`net/rssi`	`<rssi value>`	The current signal value when connected to a WiFi
`net/connections`	`<connection count>`	The number of stations connected to the device in access point mode
`net/networks`	`{..}`	The result of a WiFi scan as JSON object

MQTT Configuration

The MQTT configuration is stored in a file named mqtt.json.

Sample `mqtt.json`

{
    "host": "192.168.107.1",
    "port": 1884,
    "username": "",
    "password": "",
    "clientName": "${hostname}",
    "domainToken": "mu",
    "outDomainToken": "omu",
    "lastWillTopic": "",
    "lastWillMessage": "",
    "alwaysRetain": false,
    "subscriptions": [],
    "outgoingBlackList": [],
    "incomingBlackList": []
}

Configuration Options Placeholder

Some of the configuration options support the use of placeholders in order to allow values that are specific to a certain devince without the need to create separate configuration files. Placeholders are written in the form of ${PLACEHOLDER}.

The following options allow the the of placeholders:

The clientName of the device. The default value uses a plceholder: ${hostname}
The lastWillMessage of the device.

The following placeholders are currently available:

mac: full mac address
macls: last 4 digits of mac address
macfs: first 4 digits of mac address
hostname: hostname of the device

Top Level Configuration Options

Field	Usage
`host`	Hostname or ip address of the MQTT server. This value is mandatory
`port`	Port number under which the MQTT server is reachable. (default: 1884)
`username`	Username for mqtt server authentication. (default: empty for no authentication)
`password`	Password for mqtt server authentication. (default: empty for no authentication)
`clientName`	The unique MQTT client name. (default: `${hostname}`)
`domainToken`	Common domain token for device group. (default' `mu`)
`outDomainToken`	Domain token for outgoing messages. (default: `omu`)
`lastWillTopic`	Topic of MQTT last will message. (default: `<outDomainName>/<clientName>/mqtt/state`)
`lastWillMessage`	Message content for last will message. (default: `disconnected`)
`alwaysRetain`	If `true` all messages published to MQQ will flagged as `RETAINED`. (default: `false`)
`subscriptions`	List of additional subscription to route into the scheduler's message queue. (default: empty)
`outgoingBlackList`	List of topics and topic wildcards that will not be published to the external server
`incomingBlackList`	List of topics and topic wildcards that will not be published to the muwerk scheduler's message queue

MQTT Message Interface

Incoming

Topic	Message Body	Description
`mqtt/outgoingblock/set`	`topic[-wildcard]`	A topic or a topic wildcard for topics that should not be forwarded to the external mqtt server (e.g. to prevent message spam or routing problems)
`mqtt/outgoingblock/remove`	`topic[-wildcard]`	Remove a block on a given outgoing topic wildcard.
`mqtt/incomingblock/set`	`topic[-wildcard]`	A topic or a topic wildcard for topics that should not be forwarded from the external mqtt server to muwerk.
`mqtt/incomingblock/remove`	`topic[-wildcard]`	Remove a block on a given incoming topic wildcard.

Outgoing

Topic	Message Body	Description
`mqtt/config`	`<prefix>+<will_topic>+<will_message>`	The message contains three parts separated bei `+`: prefix, the last-will-topic and last-will message. `prefix` is the mqtt topic-prefix automatically prefixed to outgoing messages, composed of `omu` (set with mqtt) and `hostname`, e.g. `omu/myhost`. `prefix` can be useful for mupplets to know the actual topic names that get published externally.
`mqtt/state`	`connected` or `disconnected`	muwerk processes that subscribe to `mqtt/state` are that way informed, if mqtt external connection is available. The `mqtt/state` topic with message `disconnected` is also the default configuration for mqtt's last will topic and message.

MuSerial - exchange of MQTT pub/sub messages between two muwerk MCUs via serial link

Two muwerk systems can be linked via a serial connection, if both run MuSerial. Pub/Sub Messages are exchanged transparently over the serial link. If one of the systems is connected to an external MQTT server, then both systems have access to MQTT pub/sub. To the outside, the system appears as one system, e.g. sensors or actors can be connected to either system, behaving identical to the outside world.

That way, a muwerk MCU without networking hardware can be connected to a network via a second muwerk system with network access via MuSerial.

See Example SerialBridge for a complete overview.

History

0.4.1 (2022-10-17): Support for ESP32C3 (platform define __ESP32_RISC__).
0.4.0 (2021-01-30): Breaking change for ustd library include: ustd include-files have now ustd_ prefix to prevent name-clashes with various platform-sdks. [queue.h clashed with ESP8266-Wifi, platform.h clashed with RISC-V SDK, hence new names ustd_queue.h and ustd_platform.h etc.]
0.3.2 (2021-01-29): MuSerial: MQTT-enable non-networked hardware via serial link, NTP Bugfix, MQTT connection state fix.
- Bugfix: NTP initialization on ESP8266 failed often, #6, due to unsafe parameter handling in current configTime() API of ESP8266, fixed.
- MuSerial: MQTT-via-serial link between two muwerk MCUs to provide MQTT access to hardware without network via serial. MuSerial runs on all platforms and can be used to network-enable hardware via ESPs. See SerialBridge.
- Fixed handling of mqtt connection state when network connection changes
CI (2021-01-38): Github actions build-check for all supported platforms: network for ESPxx, serial for all others.
0.3.1 (2021-01-20): Minor change: enable dependency-managment for Arduino library manager.
0.3.0 (2021-01-20): Next Generation Network: See section "Breaking Changes at Version 0.3.0" for caveats.
- Support for Access Point mode and Dual Mode (both network station and access point mode)
- Support for enhanced network scans (async and display of hidden networks)
- Interface for controlling network operations (start, stop, restart)
- Detailed and extensible configuration for network and MQTT
- Support for controlling the RETAINED flag in published messages
0.2.1 (2021-01-02): Small breaking change: the format of the mqtt/state has been simplified: the message contains either connected or disconnected. Configuration information has been moved into a separate message mqtt/config. Support for no outgoing domain prefix (no 'omu') fixed.
0.2.0 (2020-12-25): Initial support for LittleFS on ESP8266.
0.1.99 2020-09 (not yet released): Ongoing preparations for switch to LittleFS, since SPIFFS is deprecated.
0.1.11 (2019-12-27): New mqtt.h api functions addSubscription(), removeSubscription() that allow to import additional topics from external MQTT server. See API doc for details.
0.1.10 (2019-11-17): Add information about external prefix to mqtt/state messages. (HA registration of mupplets requirement)
0.1.9 (2019-11-17): Allow publishing to unmodified topics using ! topic-prefix. Send internal messages with topic mqtt/state
0.1.8 (2019-11-03): NTP Init was unreliable, fixed.
0.1.7 (2019-11-03): ESP32 crashes, if configTime() [for NTP setup] is called while WLAN is not connected. Fixes #2.
0.1.6 (2019-08-06): Outgoing messages (from ESP to MQTT server) are now prefixed by an additional outDomainToken in order to prevent recursions. Older versions did only prefix with ESP's hostname, but at the same time the ESP MQTT client also subscribes to topics starting with it's hostname. This design error caused duplicated messages.
0.1.5 (2019-07-24): This version uses Arduino_JSON for JSON parsing. Older versions relied on outdated versions of the older library ArduinoJson which is no longer supported with muwerk.

Errata

MQTT doesn't seem to run stable with latest PubSubClient v2.8. It is recommended to use PubSubClient@2.7 for the time being. This seems to affect both ESP8266 and ESP32

Documentation

ESP32 notes

In order to build MQTT for ESP32, PubSubClient v2.7 is needed (PubSubClient 2.8 crashes!).
SPIFFS filesystem: Optionally use this Arduino plugin to upload the SPIFFS filesystem to ESP32.

References

ustd microWerk standard library
muWerk microWerk scheduler
mupplets sensor and io functionality blocks
PubSubClient the excellent MQTT library used by munet.
Arduino_JSON JSON library for Arduino
Time zone rules: https://mm.icann.org/pipermail/tz/2016-April/023570.html

Name		Name	Last commit message	Last commit date
Latest commit History 213 Commits
.github/workflows		.github/workflows
Examples		Examples
extras		extras
.clang-format		.clang-format
.gitignore		.gitignore
.travis.yml		.travis.yml
LICENSE		LICENSE
README.md		README.md
library.json		library.json
library.properties		library.properties
mqtt.h		mqtt.h
munet.h		munet.h
muserial.h		muserial.h
net.h		net.h
ota.h		ota.h
web.h		web.h

License

muwerk/munet

Folders and files

Latest commit

History

Repository files navigation

munet

Dependencies

Breaking Changes at Version 0.3.0

Changed Method Signatures

Changed Configuration File Format

Retained Messages

Configuration

Minimal Required Configuration Files

Minimum net.json

Minimum mqtt.json

Network Configuration

Sample net.json

Configuration Options Placeholder

Top Level Configuration Options

Configuration Options for Access Point Mode

Configuration Options for Network Station Mode

Configuration Options for Network Service DNS Client

Configuration Options for Network Service NTP Client

Network Message Interface

Incoming

Outgoing

MQTT Configuration

Sample mqtt.json

Configuration Options Placeholder

Top Level Configuration Options

MQTT Message Interface

Incoming

Outgoing

MuSerial - exchange of MQTT pub/sub messages between two muwerk MCUs via serial link

History

Errata

Documentation

ESP32 notes

References

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 20

Packages 0

Contributors 2

Languages

Minimum `net.json`

Minimum `mqtt.json`

Sample `net.json`

Sample `mqtt.json`

Packages