Discussion Around Arrays

[timpur]

Arrays

homie/super-car/$nodes → "lights[]"

homie/super-car/lights/$name → "Lights"
homie/super-car/lights/$properties → "intensity"
homie/super-car/lights/$array → "0-1"

homie/super-car/lights/intensity/$name → "Intensity"
homie/super-car/lights/intensity/$settable → "true"
homie/super-car/lights/intensity/$unit → "%"
homie/super-car/lights/intensity/$datatype → "integer"
homie/super-car/lights/intensity/$format → "0:100"

homie/super-car/lights_0/$name → "Back lights"
homie/super-car/lights_0/intensity → "0"
homie/super-car/lights_1/$name → "Front lights"
homie/super-car/lights_1/intensity → "100"

I was looking at this example and thought to my self, that actually looks complex to subscribe to via mqtt. Wouldnt it be nicer to do this:

homie/super-car/lights/0/$name → "Back lights"
homie/super-car/lights/0/intensity → "0"
homie/super-car/lights/1/$name → "Front lights"
homie/super-car/lights/1/intensity → "100"

Thus you onlt have to sub to homie/super-car/lights/# to get all the node topics, since there all hirarical now.

Maybe also:

homie/super-car/lights/elements/0/$name → "Back lights"
homie/super-car/lights/elements/0/intensity → "0"
homie/super-car/lights/elements/1/$name → "Front lights"
homie/super-car/lights/elements/1/intensity → "100"

Just ideas, tell me your thoughts?

[timpur]

@marvinroger you said this:

homie/super-car/lights/0/$name looks like a $name attribute of the 0 property of the lights node.

This is true, never said i had a solid idea, but is this something worth thinking about...? mabe the 2nd example is a better option? idk what the solution would/could look like, just wondering if you guys have any ideas ?

[timpur]

Could be homie/super-car/lights/elements/_0/$name → "Back lights"?

[ingoogni]

For my understanding, as I miss the point in $array:

Array gives the possibility to address a whole bunch of "(not realy)nodes" at once and makes it possible to address one "(not_realy)node" in the bunch induvidualy

For the first part, you could also use one node with a $datatype boolean[] or maybe even $datatype boolean[4]. With this one could switch on/off all 4 lights at once or induvidualy.

Now front light should be dimmable, so break that out into a separate node /lights/front/intensity.

The second step is not realy needed as the controller knows the state of the lights, so you could use an integer to switch them on and dim. Ok, adressing a single led in an arry of many may become waste full.

[nerdfirefighter]

i guess i have a different opinion on what should be a node and what should be a range/array.

If i was designing the example i would have each light as its own node and i would have a meta object that is arrays of all nodes of the same type. e.g. all nodes of type light would also be in a meta array of lights.
this allows flexibility.

for a car the one i see a range being used for lights is left-indicator[0-2] for example as you have many indicators on a car but most of the time you will call them together.

front lights have high and low beam so really they are there own nodes and each range element is a low beam light (most cars have at least two). I'm being pedantic but rule number one of architecture is if it cant be reused for every situation then its not an architecture.

also the example on the definition page is incomplete as the $type node attribute is missing.

as for the mqtt layout, why not homie/super-car/light[0]/$value

[timpur]

The main point im raising this is because it would be nice to subscribe to on root topic (homie/super-car/lights/#) for a node which also includes all the array items. Currently this is not possible and the above answers doesn't help in this case ...

[nerdfirefighter]

my suggestion for this is a little left field but lets see how it goes:
for clarity in this post i am going to use the following defitions
Array - a collection of nodes
Range - a node that has a number of sub elements (e.g. led light string)

I writing this i though of a few different ways

1. To make arrays i think we could do some meta topics.
   e.g.
   homie/car/front-light/$type -> light
   homie/car/$arrays/node-types/light/front-light/$type -> light
   homie/car/$arrays/myarray/front-light/$type -> light
   these all point to the same thing, just different ways to access it.

issues with this design is that you have to publish many different topics all with the same values and it requires one or both ends to keep them all in sync.

2. make specific topics for each group but you cant reference them individually.
   homie/car/front-light/$value = 100
   homie/car/$groups/nodes/light/$value = 100
   homie/car/$groups/mygroup/$value = 100
   i like this format better as there is only one way to reference the specific node, but there is ways to perform actions across a group of nodes. meta groups would auto build based on node types, and you can also build your own groups.
   I would also suggest an attribute to get a list of all the members.
   homie/car/$groups/mygroup/$members = front-light,back-light

3. what i think you need:
   as per my def above, an array is a collection of nodes.
   so i would suggest something like:
   homie/car/nodes/front-light/$value -> 100
   where nodes is the array or group name. it could be anything you like, e.g
   homie/car/front/light/$value -> 100
   then have homie/car/$groups -> front, to list all the groups
   you can still have $nodes but that would look more like 'front/light', so it captures the path to the node via the group.

this sorta gives us the best of both worlds. e.g. if you dont want to use groups you just put all your nodes in the same group and we could default that name to be "nodes", if you want them in different groups you break them out. this also sorts out my current issue with the homie layout which is that all nodes are off the base of homie/device/ rather than in a topic that has all the nodes but no other elements.
e.g. if you call homie/car/# you get all the car attributes as well as all the nodes and all the attributes of the nodes. if you have homie/car/nodes/# then you only get all the nodes and their attributes.
You could have a $name and $description for the groups as well so that you could auto build the groups in front ends like Home Assistant.

[ingoogni]

Maybe the car lights is not the best example for array, my feeling is that as soon as, in this case, a light can have multiple states you run into problems

/american_car/rear/light/left

it can be on or off
it can be blinking
it can have a higher intensity when braking

one LED, three nodes for it (in extrema), where is in such a case the advantage for the light being pushed into an array with three other lights?

[davidgraeff]

Funnily I left arrays unimplemented in OpenHab/Eclipse smarthome (eclipse-archived/smarthome#5450) because the specification is too vague and not very efficiently designed in my opinion as well.

[euphi]

Don't make it too complicated. Please keep in mind, that someone must implement it ;-)

Main purpose of an array is to have a bunch of simple elements to switch, e.g. Inputs or Outputs.
So I like the idea to integrate it into hierarchy (The variant:

homie/super-car/lights/elements/0/$name → "Back lights"
homie/super-car/lights/elements/0/intensity → "0"
homie/super-car/lights/elements/1/$name → "Front lights"
homie/super-car/lights/elements/1/intensity → "100"

)

$name should be kept optional.

For the first part, you could also use one node with a $datatype boolean[] or maybe even $datatype boolean[4]. With this one could switch on/off all 4 lights at once or induvidualy.

@ingoogni : So how would you adress this dataype?

[timpur]

@euphi, yeah my thoughts too. Maybe something like

homie/kitchen/lights/_0/...

Not sure just, just ideas.

[ingoogni]

@euphi I'm not realy sure what you mean, lack of lingo on my side.
The point, I was trying to make is, or question I have, do we need $array when you can get the same result with what else there is in the convention. Defining a $datatype as an array of integers or bools (comma separated values) gives already al lot of information and flexibility in switching on/of stuff at once. In combination with $format enum you can target specific elements of the array (if I understand that right)

[euphi]

@ingoogni How do you want to adress a single element of an array defined as datatype (e.g bool[16])?
$format does not help (yet).

[ingoogni]

Send the whole array with only the targeted item changed compared to current state (this is why I wrote it could become wasteful with many many items, bool[123456])

[euphi]

I don't think this is a good idea because then you need to compare on receiver side, if something has changed.
For a set command this needs to be done on the device side, which is a lot of overhead. Especially if you think about the necessary architecture. Should the device-specific Homie-framework (e.g Homie-ESP8266) forward the whole array to the user application?

If so, the user has to handle it accordingly which may beyond his/her skills?
If not, the framework needs to keep track of all states of its array properties which may be an unnecessary overhead.

Homie shall be kept lightweight and easy to use for beginners, so arrays as standard datatype should not be part of the convention.

[ingoogni]

I would probably compare nothing but just overwrite all values (depending on the application).

[ingoogni]

Actually, I have no idea what the 'cost' is of something in Homie;

set up 1000 nodes to send messages to as many LEDs
set up a single node and send it a 1000 items long array of bools, loop and set the LEDs
set up $array for a 1000 LEDs

[vnodeng]

I'm trying to implement Homie arrays in my device and found this suggestion to be more or less balanced one:

homie/super-car/lights/0/$name → "Back lights"
homie/super-car/lights/0/intensity → "0"
homie/super-car/lights/1/$name → "Front lights"
homie/super-car/lights/1/intensity → "100"

The above form are perfectly suitable for programmatic parsing/processing and doesn't limit one to provide human-friendly aliases like

homie/super-car/lights/back

when necessary. For the later case an $alias property could be useful.

Also, it is a bit questionable if $array must be in the range form, because it is much more natural to just provide a count of elements - this greatly simplifies parsing in scripts. The missing start element means that arrays should be zero based.

[timpur]

A bit bias, as that was my example, but i think so too. Means you can sub to homie/super-car/lights/# and get the array node in one go.

Marvin made the point

homie/super-car/lights/0/$name looks like a $name attribute of the 0 property of the lights node.

Thats why i offered 'homie/kitchen/lights/_0/...
Use the _ to signal that it is an array item, like we do now

@borisxm, i have raised that question before about why we need a start and end index, why not just a count? I do prefer that, its simple and easy to parse. $alias could be nice.

[vnodeng]

homie/super-car/lights/0/$name looks like a $name attribute of the 0 property of the lights node.

I see no problem with this, because array elements are properties of the given node and the $array attribute merely helps to enumerate them properly. Let me step out from the car example and illustrate this differently.

Consider a moderate complexity device which can independently control 3 RGB strips. The naive approach may lead to the following layout (prefixes are omitted for simplicity):

leds/led0/mode → RGB
leds/led0/RGB → 128,128,255
leds/led0/R → 128
...
leds/led0/HSV → 0.67,0.5,1
leds/led0/H → 0.67
...
The same repeated for led1 and led2

This scheme works, but exposes too much information at a single level and not computer friendly. Using an $array we can make it slightly better:

leds/$array → 3
leds/0/mode → RGB
leds/0/RGB → 128,128,255
leds/0/R → 128
...
leds/0/HSV → 0.67,0.5,1
leds/0/H → 0.67

but it is still, too verbose. The next step would be to factor out configuration into its own topic, because it is neither changes nor can be changed frequently:

leds/config/$array → 3
leds/config/0/mode → RGB
leds/config/1/mode → HSV
leds/config/2/mode → RGB
leds/$array → 3
leds/0/RGB → 128,128,255
leds/0/R → 128
...
leds/0/HSV → 0.67,0.5,1
leds/0/H → 0.67

This looks much better, but still, RGB and HSV spaces form a nasty mess. Lets go further and try to separate them:

leds/config/$array → 3
leds/config/0/mode → RGB
leds/config/1/mode → HSV
leds/config/2/mode → RGB
leds/RGB/$array → 2
leds/RGB/0/value → 128,128,255
leds/RGB/0/R → 128
...
leds/HSV/$array → 1
leds/HSV/0/value → 0.67,0.5,1
leds/HSV/0/H → 0.67

Now we got perfect, and MQTT friendly layout with clear functionality separation. It is very easy to subscribe to RGB or HSV leds status, there is no duplicates and static configuration managed separately.

Use the _ to signal that it is an array item, like we do now

This doesn't help much, because MQTT doesn't have wildcards within topic - one can't subscribe to array changes only. Again, it is matter of proper layout which should avoid messy mixes of named and array properties.

set up a single node and send it a 1000 items long array of bools, loop and set the LEDs

MQTT isn't the best control protocol, so one has to be creative:

leds/ctl/on → 0-999 # turn on all leds
leds/ctl/on → 0-999,2 # turn on each even led
leds/ctl/on → 1-999,2 # turn on each odd led
leds/ctl/on → 0-999,10 # turn on every 10ths led

[davidgraeff]

Node Arrays as they are currently done in the spec are nothing more than multiple defined nodes. Node arrays should bring a benefit like efficiency as described in #49.

[boc-tothefuture]

Should we remove node arrays from the spec while we work them out?

Reasoning:

• We already have one breaking change in place, so we are going to a new major version already.
• There seems to be a lot of discussion that needs to happen on how arrays should be implemented and structured.
• It would be easier to not have the functionality and add it back in later than let the convention obtain wider usage an try and change arrays.

I would be happy to put in a PR to remove arrays from the current spec. Potentially, noting that the '_' char is reserved for potential future use.

[Thalhammer]

Maybe add them as an extension ?

[boc-tothefuture]

@Thalhammer -
Its possible they could be highly useful in the core convention.. My concern right now is that we have a spec for arrays that I don't think we have consensus on and they should be removed asap before changing them is really really hard.

[davidgraeff]

Example Use case:

This is actually not an example for arrays.

• You have 1..14 nodes or 1..14 properties. This is really not much.
• Your nodes/properties are not uniform, like a led-array. They almost all serve a different purpose and the purpose is probably burned into the firmware ore pre-configured before exposing via homie.

Arrays in programming languages allow to work on a uniform and consecutive memory array.
Your BITSET suggestion goes into the direction that we already thought about.

An array construct in Homie should provide to set multiple properties at once. The old "arrays", removed from the convention, did not help here. They only helped in reducing the advertising topics.

Another subject is "grouping" of nodes, so that a controller can also group related nodes. But I have suggested another solution for this already, that subject is not related to "arrays".

[Thalhammer]

I think a really good example would be addressable LED stripes (e.g. ws2812) which require the exact same properties per led (e.g. colour) and might consist of hundreds to thousands of nodes (or properties, depending on how you design your application).

A possible use case would be setting them all to the same colour using a single message instead of hundreds or setting all of them to a pattern in a single message.
Apart from reducing bandwidth usage and improving performance this also provides the ability to sync said leds.
With individual messages, there could be cases where only some of them arrive resulting in an unwanted state.

[davidgraeff]

Yup but again a bad example. Mqtt is TCP based and TCP is a stream protocol which alone puts a burden on microcontrollers with state handling and message fragmentation.
Mqtt is the absolute wrong protocol for light effects, there is artnet and numerous other protocols. Homie addresses non realtime purposes and a reasonable number of nodes and properties. The current design doesn't scale for thousands of nodes or properties and "arrays" don't solve this magically.

[euphi]

Mqtt is the absolute wrong protocol for light effects, there is artnet and numerous other protocols. Homie addresses non realtime purposes and a reasonable number of nodes and properties. The current design doesn't scale for thousands of nodes or properties and "arrays" don't solve this magically.

Updating a LED strip may have no realtime-requirements but you still may want to synchronize the update.(e.g. a LED strip that changes its pattern every minute)

So Homie could be a suitable protocol for that.

[Thalhammer]

Yup but again a bad example. Mqtt is TCP based and TCP is a stream protocol which alone puts a burden on microcontrollers with state handling and message fragmentation.

I kind of agree with this, but I don't think this is important here. I assume anyone trying to implement homie support has a micro capable enough to handle TCP. There are a lot of cheap solutions (ESP8266, ESP32, etc) which have really powerful hardware.

Mqtt is the absolute wrong protocol for light effects, there is artnet and numerous other protocols.

Totally agree, but I never talked about real time. For example, I have such a strip set up in my garden, which has a total of 2000 leds. In addition to a custom UDP based protocol for real-time, they also provide a homie interface that ties them into my home automation. Updates using homie are rather sparse (~every 30minutes).

Homie addresses non realtime purposes and a reasonable number of nodes and properties. The current design doesn't scale for thousands of nodes or properties and "arrays" don't solve this magically.

No they don't, but they reduce them a lot.

I'm against including arrays in the main convention, but we should definitely keep them as an extension.

[davidgraeff]

Sure, an extension would be fine. There are definitely different requirements regarding "arrays". What I like to have is a way for grouping nodes in 4.0 though.

I thought about a $group attribute for nodes.
And a homie/device123/$groups/groupie/$name topic per group.

Extensions could build up on this to assign a value with a single /set on a group etc.

[euphi]

Example Use case:

This is actually not an example for arrays.

* You have 1..14 nodes or 1..14 properties. This is really not much.

However, even for 2 arrays can help to have a clean implementation.

* Your nodes/properties are not uniform, like a led-array. 

Yes they are. They are a simple inputs (or outputs).

They almost all serve a different purpose and the purpose is probably burned into the firmware ore pre-configured before exposing via homie.

No, they serve the same purpose: Reading an input (or setting an output).
The device should be agnostic to the function of the input or output. I don't want to update the firmware just because the plumber changed which room is connected to which valve.
(Or better example, because more likely to happen: Reading state of circuit breakers and setting relays: If the electrical wiring changes, I dont want to update the firmware of my homie device).

Remember that homie is device-centric, not function-centric. There is no need to have a function-related information in the device.

Arrays in programming languages allow to work on a uniform and consecutive memory array.
Your BITSET suggestion goes into the direction that we already thought about.

Bitsets can be a solution. This would be somehow how the old convention was (arrays on property level).

[davidgraeff]

Homie is about exposing device functionality. It is up to the developer if he wants to expose abstract functionality, pins, memory locations, CPU registers.

Your conclusion that GPIO pins are more device centric and homie like then a coffee machine brew function is not correct in my opinion.

[euphi]

Of course it depends on the use case where to implement functionality. All I say is that it shall be possible to build function-agnostic devices and that a feature like arrays is very useful for that.

Of course, every device that is build to control a function on its own (e.g. brewing coffee or irrigating a flowerbed), should also expose this functionality (e.g. naming the irrigation valves, water level contacts etc.).

The convention just shall be able to handle all these usecases.

[boc-tothefuture]

Based on my experience over the past few days, without node arrays, we may have to reassess the values in our Timings section. It can be very difficult to publish all the topics in half a second - especially if you have any sort of logging enabled.

[davidgraeff]

Doesn't the section say half a second without any input? As long as you keep sending data, everything is alright.

[boc-tothefuture]

Doesn't the section say half a second without any input? As long as you keep sending data, everything is alright.

Nope

As soon as a device starts to publish any Homie related topic, it MUST finish with all topics within a timeframe of 500ms.

[davidgraeff]

I see. Do we solve it by resetting that timer on new input?

[Thalhammer]

I see. Do we solve it by resetting that timer on new input?

What if network latency exceeds 500ms (quite common for EDGE/GPRS) ?

[davidgraeff]

quite common for EDGE/GPRS

We are living in a 4G, soon 5G world :D On the serious side: The timeout can be extended to the TCP acknowledge timeout, which is higher. But we still need a kind of timeout. How does the controller know if it should finishing rendering otherwise?

Another idea: A controller uses the roundtrip time to the MQTT broker + a static value as reference.

[Thalhammer]

We are living in a 4G, soon 5G world :D

Maybe the rest of our planet does so, but Germany apparently does not.

Another idea: A controller uses the roundtrip time to the MQTT broker + a static value as reference.

While I like the idea, I'm not sure how you can reliably determine this.

[euphi]

With the $state announcements in Homie V3.0, I think we don't need the timing requirement at all.

The controller know that advertising information is incomplete as long as the $state is still init.

TBH it seems that I missed the Timing section, otherwise I would already have proposed to delete it when discussing the $state topic and node avertisements.

[davidgraeff]

@euphi You seem to miss that a device talks to a broker and the broker talks to the controller in the end. And the broker is not required to send messages in order. It could back up messages and send $state first for example.

[euphi]

The message order must be preserved by the controller, see http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718105

[boc-tothefuture]

@euphi - The order information is interesting (and potentially helpful). But it doesn't solve the case when the controller reconnects, since the state will be "ready" and not "init" in that case.

[euphi]

If the controller reconnects it receives the $state after node advertisements. So, if the $state has been received and is "ready" the controller can be sure that node advertisement is complete.

[boc-tothefuture]

@euphi - This is not the behavior I am seeing from mosquitto. Also, on re-reading your link. I believe that refers to the order of messages published within a topic not the order in which topics are delivered.

[wwebers]

As far as I understand your guys discuss the following "use cases" for having arrays:

• @euphi with a low level mapping of hardware pins to MQTT topics to be able to send and receive messages on that level.
• @euphi would also like to represent the state of hardware pins with some BITSET datatype
• @timpur and others state to be able to map the individual state of LEDs in a LED strip (and their colors) to MQTT topics

MQTT is unfortunately just a protocol specification for message exchange, thus I very much like this homie activity a lot.

However, I miss the following in this discussion. Where does you draw the borderline between topics and messages? The whole discussion is about arrays, by giving very low level examples of controlling the IoT devices state. I got the impression that this arrays discussion moves payload information into topics. I also got the impression that many arguments come from the "smart developers" perspective, not so much from the architectural point of view.

Thus, I prefer homie as some kind of MQTT convention defining the topic structure but not the flavor how to control my devices. Not even how to define the structure of the messages (payloads). As such, I think it would be better to remove arrays completely.

[davidgraeff]

Which brings the following topic to the table:

Release of Homie 4.
As you guys remember we need all maintainers to agree for a release.

We already agreed to remove arrays with the option to add something back if someone comes up with a brilliant solution.

I suggest to wait for the date/time issue to be solved and then do the release.

[leifclaesson]

I found this thread as I was writing my own lightweight Homie 3.x code for ESP8266, for use in my openHAB setup. I couldn't make sense of the arrays either, so finding David's comment that he didn't implement arrays in openHAB for the same reason, saved me a lot of work and headache. :)

[davidgraeff]

Closing this. If someone really needs something similar, a PR or a new issue can be opened.