Wishlist for v2.0.0

[marvinroger]

TODO

1st step

• The framework must be completely asynchronous. It was not doable before because TCP stuff was synchronous, now with ESPAsyncTCP, we can. The only drawback is SSL is not supported. This is currently not stable enough anyway, and it probably won't be production ready due to limited memory, something that might change with the ESP32.

  • WebServer must be implemented with ESPAsyncWebServer
  • MQTT client must be switched to AsyncMqttClient (implemented in c6facaa)
  • See for OTA, currently blocking (implemented in c6facaa)

• More modular design (implemented in Add several virtual methods to HomieNode #91, thanks to @euphi and @unaiur)

• The configuration UI must be embeddable, optionally. Captive portal must be working, so everything would be transparent to the user (implemented in Load the configuration UI into ESP8266 flash chip #84, thanks to @unaiur!)

• There must be custom settings support (Interface for adding extra settings #26)

• There must be a way to change configuration while in normal mode. This is achievable with AsyncMqttClient because there are no hard limits like PubSubClient has (change configuration in normal mode #36)

• @euphi (implemented in f1110fb):

  It should be possible to set a logger class, e.g. any class implemting the "Print" class

• Embed the magic bytes (useful for homie-ota for example) as a macro (implemented for firmware name, version and brand in e73b3af)

• No need to call registerNode (implemented in Proposal for autoregistration of HomieNodes #96, thanks to @euphi and @unaiur)

• Transparent proxy for captive portal custom configurations (implemented in Transparent proxy for captive portal custom configurations #95, thanks to @flaviostutz )

• Optimize Wi-Fi (implemented in 56088cd)

  • Use events instead of the main loop to handle Wi-Fi events
  • Store the Wi-Fi config in SDK to cut down the time required to connect to the Wi-Fi (see https://github.com/z2amiller/sensorboard/blob/b33da405795761efce0dfa02f53b6932c87958ee/PowerSaving.md)

2nd step

• Make it bulletproof. There are no hard limits in the MQTT client for topic / payload size, so this must explicitely be checked. Same thing goes for the async Web Server:
  • Make sure the MQTT payload is not too big (1000 bytes max?)
  • Make sure the MQTT topic is not too big (200 bytes max?) needs to be done in the AsyncMqttClient library
  • Make sure the JSON API request bodies are not too big (1000 bytes max?)

Breaking changes (or migration guide from v1.5.0 to v2.0.0)

• Homie.setFirmware(name, version) must be replaced by Homie_setFirmware(name, version)
• Homie.setBrand(brand) must be replaced by Homie_setBrand(brand)
• Every Homie.registerNode() must be removed
• If you've enabled Serial logging, Serial.begin() must be called explicitely in your sketch
• Remove the HOMIE_OTA_MODE in your event handler, if you have one
• config.json changes:
  • In the mqtt section:
    • mdns must be removed. Set the host and port fields
    • ssl and fingerprint must be removed. No SSL support anymore
  • In the ota section:
    • All fields must be removed, except enabled. OTA is now done over MQTT now
• The Homie.setNodeProperty() signature changed from const HomieNode& node, const char* property, const char* value, bool retained to const HomieNode& node, const char* property, const char* value, uint8_t qos, bool retained

Let's discuss

The 2.0.0 will be a breaking change. As such, we can safely change the API, the functions name, the order of the parameters... Let me know if you think there is any weirdness with the API, or the HTTP API of the config mode, etc. 😃

[jpmens]

+1 (and very much appreciated) for embedding the magic bytes for homie-ota; this will make our sketches look neater.

Regarding TLS (SSL): in spite of the huge amount of work you went to to get it to work, I have since reconsidered. Within a home, It's more important that devices are stable and "unbreakable" (eg killing them with PUBs) than having TLS. Furthermore I have the feeling that most people don't care about the security anyhow... In other words: drop TLS until ESP8266 supports it natively and asynchronously.

[piotrwachowiak]

Hi, once again thanks for great work. If it is the right place for some kind of wish list:
For me, if not TLS and/or native PKI support would be great, if only esp can handle this.
Deep sleep and battery control included in MQTT convention, like delayedHomieLoop(int sleepTime) or something like this, method which takes sleepTime as argument and handles everything. Just dreaming.
I had some troubles creating universal OLED thing, which is capable of displaying given topic/message with given params. I'd love to see it done in professional way.
It's probably just me, but event handling and subscribing is not clear for me - personally I failed with subscribe(), HomieNode relayNode("room", "relay", relayOnHandler, true) works, but I am not sure it's right way.

Best regards,

Piotr

[unaiur]

I wouldn't add custom settings. I think that we should allow to use WPS to connect to the WiFi network and use mDNS to discover a default mqtt broker service. Then, the remaining options can be configured with mqtt messages.

This way we didn't need any special BootConfig mode.

[jpmens]

+1 for configuration via MQTT.

[marvinroger]

@unaiur this would be ideal, but I am not sure it would fit everyone. WPS is probably not supported by all routers, and who actually advertise its MQTT broker via mDNS?

[euphi]

Regarding API: Make the callback function a virtual method of HomieNode.

This would allow for a more modular design.

You still can have the default implementation to call a NodeInputHandler.

[marvinroger]

@euphi Can you provide a sample code of how it would look like?

[euphi]

Proof of concept (on tag v1.5.0)
patch_virtual_callback.txt

It was necessary to remove some const. Since I haven't coded for almost 6 years, I am a little bit out of practice, but I think the const was wrong there anyhow. With the call via the method pointer, you discard the const of the Homienode object - with the member function, this is no longer possible.
And it is quite obvious that calling a callback method that expect to change a state of a object, should not be called on a const object :-)

I did not had the time to create a class derived from Homienode, but this patch at least do not break anything :-)
Due to a business trip, I won't have time to try more for at least one week. :-(

[euphi]

Due to bad weather today I found some time to make use of the new HomieNode::Inputhandler(..)

I created a new project at GitHub.

Especially have a look at https://github.com/euphi/BalkonController/blob/master/src/RGBWNode.cpp and
https://github.com/euphi/BalkonController/blob/master/src/RGBWNode.h.

It is work in progress, there are still some nodes with the old style of callback.

To use them with OpenHab, I use the following items configuration:

Number balkon_temp      "Temperatur [%.1f °C]" (Balkon, TempSens)  {mqtt="<[local:devices/balkctrl/Temperatur/degrees:state:default]"}  
Number balkon_press     "Luftdruck [%.1f hPa]"  (Balkon)            {mqtt="<[local:devices/balkctrl/Temperatur/hPa:state:default]"}  
Switch balkon_auto      "Automatik"             (Balkon)            {mqtt=">[local:devices/balkctrl/Auto/on/set:command:ON:true],>[local:devices/balkctrl/Auto/on/set:command:OFF:false]"}  
Switch balkon_pumpe     "Pumpe"                 (Balkon)            {mqtt="<[local:devices/balkctrl/Pumpe/on:state:ON:true],<[local:devices/balkctrl/Pumpe/on:state:OFF:false],>[local:devices/balkctrl/Pumpe/on/set:command:ON:true],>[local:devices/balkctrl/Pumpe/on/set:command:OFF:false]"} 
Switch balkon_valve1    "Kräuter bewässern (V1)"  (Balkon)        {mqtt="<[local:devices/balkctrl/Ventil/1:state:ON:true],<[local:devices/balkctrl/Ventil/1:state:OFF:false],>[local:devices/balkctrl/Ventil/1/set:command:ON:true],>[local:devices/balkctrl/Ventil/1/set:command:OFF:false]"}
Switch balkon_valve2    "Hecke bewässern (V2)" (Balkon)            {mqtt="<[local:devices/balkctrl/Ventil/2:state:ON:true],<[local:devices/balkctrl/Ventil/2:state:OFF:false],>[local:devices/balkctrl/Ventil/2/set:command:ON:true],>[local:devices/balkctrl/Ventil/2/set:command:OFF:false]"}
Switch balkon_valve3    "Himbeeren bewässern (V3)" (Balkon)        {mqtt="<[local:devices/balkctrl/Ventil/3:state:ON:true],<[local:devices/balkctrl/Ventil/3:state:OFF:false],>[local:devices/balkctrl/Ventil/3/set:command:ON:true],>[local:devices/balkctrl/Ventil/3/set:command:OFF:false]"}
Switch balkon_valve4    "Rosen bewässern (V4)" (Balkon)            {mqtt="<[local:devices/balkctrl/Ventil/4:state:ON:true],<[local:devices/balkctrl/Ventil/4:state:OFF:false],>[local:devices/balkctrl/Ventil/4/set:command:ON:true],>[local:devices/balkctrl/Ventil/4/set:command:OFF:false]"}
Number balkon_signal    "WiFi Signal [%.1f %%]" (Balkon)            {mqtt="<[local:devices/balkctrl/$signal:state:default]"}


Color  balkon_led       "LED-Streifen"  <slider>    (Balkon)
Dimmer balkon_led_R     "Rot [%d %%]"   <switch>    (Balkon)        {mqtt=">[local:devices/balkctrl/LEDs/r/set:state:*:default]"}
Dimmer balkon_led_G     "Grün [%d %%]" <switch>    (Balkon)        {mqtt=">[local:devices/balkctrl/LEDs/g/set:state:*:default]"}
Dimmer balkon_led_B     "Blau [%d %%]"  <switch>    (Balkon)        {mqtt=">[local:devices/balkctrl/LEDs/b/set:state:*:default]"}
Dimmer balkon_led_W     "Weiß [%d %%]" <slider>    (Balkon)        {mqtt=">[local:devices/balkctrl/LEDs/w/set:command:*:default]"}

[unaiur]

I've evolved the proposal from @euphi in a project I'm doing to control led strips with Homie.
You can see the code in homie-nodes

[TommySharpNZ]

To be able to run Homie on a battery powered device would be a great addition especially for simple environmental sensors around the house.

[powerwade]

My wishlist item is to log/send messages to a local / remote syslog server.

[jpmens]

@powerwade I think that would be out of scope, in particular as it'd mean implementing syslog protocol: Homie is MQTT.

This can very easily be accomplished by creating a subscriber (server-side) which subscribes to your MQTT messages and pushes them out via syslog; a few lines of, say, Python would do the trick.

[jayaras]

I could see syslog being handy and have thought of a few use cases where it would/could be. At least for prototyping when I don't have serial output. I've been fighting with a faulty pn532 board and it would have been handy to get some debugging info over syslog.

[flaviostutz]

I like Homie API names and the code style very much. I made a feature request on #26 regarding to captive portal.

[euphi]

Regarding syslog: It would be nice, if Homie could use another logger. Therefore it should be possible to set a logger class, e.g. any class implemting the "Print" class.

So, it is possible to create a syslog logger from another library and set it as preferred logger for Homie.

[marvinroger]

Regarding the syslog: @jpmens stated deservedly that MQTT is a mean of communication. You could just create a system (or log or whatever you want) on which you publish your debugging messages to.

I do agree with @euphi however, being able to set a Print class to log to.

[lighthousebulb]

I love the proposed Idea to set a configuration via mqtt.
But I have another suggestion:
What about a several configuration-files with different WiFi-settings for example?
The Node should scan for available Networks and choose a configuration depending on the available Networks.
For example, when I set up a Node an move it to my friends house, with different WiFi-Settings, the Node could simply choose the right config-file for the different environment.

[unaiur]

I'm working in wifi wps setup. I'm planning to add a new configuration option EnableWPS. When defined it will use the integrated settings in the ESP8266WiFi library, or start WPS pairing if no network is configured. It's nearly done and it uses very little code.

[unaiur]

Another future ticket I would like to work at in the future is startup time.

I'm planning to implement a button on battery. When the button is pressed, the ESP will wake up from deep sleep, connect to the MQTT server and publish a topic.

The problem is that it needs about 4 seconds. I will try to:
a. Save DHCP lease to persistent memory and bypass this phase while the lease isn't stale.
b. Save mDNS service also.

These steps can halve the startup time. Do you have any other idea to try to reduce startup time even more?

[unaiur]

You can see my WPS implementation at https://github.com/unaiur/homie-esp8266/tree/wps
It isn't fully tested yet, but you can taste it.

[marvinroger]

@unaiur it seems we can cut down the time needed to connect to the Wi-Fi by not calling WiFi.begin() on each boot, but let the SDK store the Wi-Fi config itself. (see https://github.com/z2amiller/sensorboard/blob/b33da405795761efce0dfa02f53b6932c87958ee/PowerSaving.md)

[marvinroger]

I am also considering removing the OTA mode and handle everything in the normal mode. MQTT would be used for the OTA, so OTA would not happen over HTTP anymore. This is now possible thanks to AsyncMqttClient and esp8266/Arduino@2f933e2

@jpmens what do you think about it? homie-ota would need some change, but I think having only one protocol to handle everything (except for the HTTP API) is a step forward.

[jpmens]

@marvinroger am I understanding correctly that the new binary firmware would be sent to Homie via an MQTT publish? If so, I'm all for it, and we can easily change homie-ota to do it.

[jpmens]

@unaiur has there been any progress with WPS for homie-esp8266? I am in a situation where this would probably be extremely useful for us.

[shogsbro]

Would it make sense to implement a captive portal while in Configure mode? I think it would only require capturing the iOS and Android test URL's, and returning a HTTP redirect.

[marvinroger]

@shogsbro the captive portal is already implemented in the latest git, but this is only useful if you've embedded the ui_bundle.gz

[thorhs]

I too would love a battery friendly deep sleep. Is that possible? I have a custom code for temperature sensing using ESP and MQTT. I would love to swap out the plumbing for Homie.

[marvinroger]

Deep sleep will be possible with #138, I don't know if there's something else you would like for deep sleep?

[karlp]

Where's the documentation on the "ui_bundle.gz" ? I can't find it in the 2.0 docs online.

[marvinroger]

@karlp just added a page: https://homie-esp8266.readme.io/v2.0.0/docs/ui-bundle

[flaviostutz]

@tiagostutz, see what an amazing v2.0 planning and execution for Homie 2.0...

[tiagostutz]

Awesome! Great expectations here...!
Now let's rush with the Config JSON API Emulator project to see if it helps build the captive portal... When I finish the README.md I'll share here to get some insights from the community.

[tiagostutz]

The first version of the emulator is ready. It's not a big deal, but maybe it can be useful:

• https://github.com/stutzlab/homie-esp8266-config-emulator
• https://hub.docker.com/r/stutzlab/homie-esp8266-config-emulator/

[averri]

The only point I don't like for v2 is removing SSL. We can't have a IoT device exchanging data with Internet with no security.

[jpmens]

@averri quite right, which is why you probably shouldn't be sending this kind of data over the Internet in the first place. Marvin has profusely explained why TLS is currently not possible. IMO you have three options:

1. Payload encryption (using whichever method you prefer). This, though, will not encrypt metadata such as topic names obviously, nor will it encrypt authentication credentials.
2. Satellite brokers. Use a broker which is (network-wise) very close to your "things". These connect via plain MQTT to the broker which then bridges out (via TLS or TLS-PSK) to upstream brokers. This is what we do and it's very reliable because the portion of the network where most hiccups occur is coupled by brokers which can elegantly retry/recover from outages
3. Use something else. (Though I'm not aware of any ESP8266 framework which comes close to what Homie offers in terms of functionality.)

[boneskull]

Marvin has profusely explained why TLS is currently not possible

Where is this? I was trying to touch AWS's IoT service, but ended up having to use a proxy.

[averri]

Hi @jpmens, I sincerely don't understand the rationale of having an IoT device which we can't use for sending data to the internet. So going through the options:

1. It's not possible to have partial encryption - or you have it or you don't have it. Neglecting security is a severe issue. See AWS IoT for example, it is very incipient, but it has strong security since the beginning.
2. Agree, this is one possible solution, but we are limiting the possibilities. What if someone wants the device to connect directly to a MQTT server on the internet?
3. It's correct to say that actually there is no other framework that comes close to Homie regarding its features. Homie it is really great, I liked it since the first time I read about it. That's why I am here contributing with critical ideas. Everybody can use something else, but Homie will continue to be unsecure if you remove TLS.

I do understand the difficulties for adding TLS to Homie at the moment, but I would like to say: don't give up because it is complex. Maybe you may not add it to v2.0.0, but I strongly recommend Homie developers to consider it as the next big feature.

Bear in mind: security will be a decision factor for adoption by many people.

[boneskull]

I have a use case that may already be covered, but I'm not sure of the canonical way to approach it.

I have a device controlling a strip of NeoPixels. One function is that it will cycle through the colors (a rainbow). The speed at which it cycles is currently configurable at a setting.

I could change the speed by exposing a property on a HomieNode. I'd like to persist this speed between resets. The only way to do this (to my knowledge) is to have both a custom setting and property handler for "speed"; I can have the property handler set the custom setting, but that seems like a smell.

Ideas:

• Allow user-defined handler for $implementation/config/set events. I don't see this event type mentioned in the docs. In my case, could inspect the custom settings, then update how the "rainbow" task is working. But I could see other users changing WiFi and other stuff on-the-fly, or causing a reset, etc.
• Persist HomieNode property state. I don't know what this would look like.

[boneskull]

...I'm not sure if I can actually write a setting from my code other than to broadcast?

[marvinroger]

@averri we don't give up, we will probably add it back in the future. 😉
@boneskull updating the configuration through $implementation/config/set reboots the device anyway. Just send your speed property retained, and your device will receive it after a reboot, nope?

[boneskull]

@marvinroger Good idea. I'm an MQTT novice.

[euphi]

Is it already possible to use platformio for latest v2.0.0? It seems there is a library.json file for 2.0.0, but in the development branch, of course.

platformio added some versioning support recently, but I'm unsure how it works?
I tried to add "Homie@2.0.0" as dependency for a project, that did not work.

Note: I ask here first to get some explanation. If no one knows if it is possible and how, I will ask again at that platformio community.

[benzino77]

Try this in platformio.ini:

...
lib_deps =
  https://github.com/marvinroger/homie-esp8266.git 
...

[segaura]

in the 'Migration guide' I read the Homie.setNodeProperty() signature has changed.
I cannot find any occurrence of "setNodeProperty" in the develop branch.

[fabiosoft]

What about an UDP logger so even debug will be completely over the air?

[stritti]

Could we close this endless issue?
If there are parts which should be extracted to separate issues, please create separate issues.

[bodiroga]

Go ahead!