Option for non-JSON output

[stoinov]

This is more of a general question: would it be feasible to add option so that each attribute to be separate entry on the MQTT path. So instead of:
zigbee2mqtt/0x00158d0002219 {"battery":"39.00","voltage":2995,"temperature":18.87,"humidity":60.87}
We get:

zigbee2mqtt/0x00158d0002219/battery 39.00
zigbee2mqtt/0x00158d0002219/voltage 2995
zigbee2mqtt/0x00158d0002219/temperature 18.87
zigbee2mqtt/0x00158d0002219/humidity 60.87

I am trying to setup my own automation without actually using HA, and I hit a roadblock with the homebridge setup - it does need simple strings to work, so I have to parse the JSON and republish them to make it work.

[hamstie]

I think this is a good idea:

• .e.g fhem server can't publish and subscribe json in an uncomplicated way
• 99% of light handling is using a single state or brightness publish

i would prefer another topic signature:
zigbee2mqtt/DEVICE/set/state ON # adding the key to the set command
or
zigbee2mqtt/DEVICE/setkv/state ON # for key value set

[cimba007]

I can only agree to that. Don't make the payload overly complicated by adding json.

[Koenkk]

@cimba007 JSON output is required for homeassistant integration. However I will add an option to disable this.

[spraot]

I've been testing these changes and it seems to work well so far. I can make a PR with the referenced changes if it seems like a reasonable way to do it to you.
I'm not sure whether I should be converting the boolean "true" and "false" strings to on/off or open/closed. Does anyone know what's standard in MQTT?

[magcode]

That could give you some inspiration: https://homieiot.github.io

[andreasbrett]

That could give you some inspiration: https://homieiot.github.io

Great choice! The new MQTT binding in openHAB supports the Homie convention and would allow openHAB users to discover MQTT devices rather than having to manually enter them in openHAB.

[dirkheiden]

I've been testing these changes and it seems to work well so far. I can make a PR with the referenced changes if it seems like a reasonable way to do it to you.

This would be a great feature (changing between JSON and separate topic). I am using openHAB and Zigbee2MQTT to bind Osram, Ikea and Xiaomi stuff. The new (faster and enhanced) MQTT binding is not working very well on formatting JSON for outgoing stuff.

Is this feature ready for testing?

[magcode]

I don't think a "real" implementation of the homie convention (and thus supporting the OH 2.4 MQTT auto discovery for example) has been considered/planned in this project yet. That would be quite some effort I guess.

[dirkheiden]

For the first step the feature described by rachetfoot:

publish_as_json: true,
publish_as_key_topics: false,

would be a great help for all openHAB users. Now sending a JSON payload like {"color":{"r": 46,"g": 102,"b": 193}} with the new MQTT binding is impossible (?). What the binding can handle out of the box is something like this:

zigbee2mqtt/mydevice/color/set 46,102,193

Now Zigbee2MQTT should accept these payloads insted of JSON messages.

Adding the homie convention to Zigbee2MQTT would be great, too.

[spraot]

My change is quite naive and only handles a single level object, converting all properties of that object to topic/message pairs. It works fine for my purposes, but to be honest, I don't think it's robust enough for merging. And with what I now know about the homie convention, I think I would prefer scrapping my approach and implementing that convention instead, however that will require a much larger effort and probably also changes to the way devices are defined.

[dirkheiden]

My change is quite naive and only handles a single level object, converting all properties of that object to topic/message pairs. It works fine for my purposes, but to be honest, I don't think it's robust enough for merging. And with what I now know about the homie convention, I think I would prefer scrapping my approach and implementing that convention instead, however that will require a much larger effort and probably also changes to the way devices are defined.

Is your change working as described in my assumption above? How can I implement it?

[spraot]

I haven't tested, but I expect that it would publish something like this:
topic: zigbee2mqtt/mydevice/color
message: {"r": 46,"g": 102,"b": 193}

And react to a similar message sent to zigbee2mqtt/mydevice/color/set

If you wanted to test and/or develop, you would just have to add my repo (rachetfoot/zigbee2mqtt) as a new remote where you installed it originally and then checkout the branch I made with the changes. (Also stop service before doing so and start again afterwards)

[dirkheiden]

(...) I think I would prefer scrapping my approach and implementing that convention instead, however that will require a much larger effort and probably also changes to the way devices are defined.

Should we open a new issue to discuss this as a request (implement the Homie convention)?

[spraot]

Up to you. However, I would like to hear whether @Koenkk is even open to this change.

[Koenkk]

What is the homie convention?

[dirkheiden]

What is the homie convention?

https://homieiot.github.io

The Homie convention defines a standardized way of how IoT devices and services announce themselves and their data on the MQTT broker.

[Koenkk]

Looks good! I think it's nice to have this in zigbee2mqtt.

[spraot]

Cool, how can I help?

[Luftloch80]

@rachetfoot
I installed your Fork and getting the mqtt in non Json format works Great
But i cannot publish anything with zigbee2mqtt/device/State/set

opt/zigbee2mqtt/node_modules/zigbee-shepherd-converters/converters/toZigbee.js:42
cmd: value.toLowerCase(),
^

TypeError: value.toLowerCase is not a function

[spraot]

Yeah, I had that issue too, and I'm not sure why because it works in the tests. However, this still works:
zigbee2mqtt/device/set

[Luftloch80]

Yes thx. This Is working

[rtreffer]

👋 I tried the homie path - highly experimental: #855

I think zigbee2mqtt might need some changed to support plugins that act on prepared messages (before toZigbee / after fromZigbee). This would simplify custom MQTT hierarchies a lot.

[Luftloch80]

Good Job.
Working with My Osram Plug, but the Osram Classic Light only Publishes this two values
No on/off or brigntness
zigbee2mqtt:info 2019-1-14 08:22:31 MQTT publish: topic 'homie/0x8418260000073d64/$stats/uptime', payload '180'
zigbee2mqtt:info 2019-1-14 08:22:31 MQTT publish: topic 'homie/0x8418260000073d64/$stats/interval', payload '60'

[rtreffer]

I haven't tested non-RGB lights yet, that's on my to-do towards end-of-week. There is a big block that looks at JSON keys (look for e.g. 'illuminance') that would likely need some additions. Am Mo., 14. Jan. 2019, 09:25 hat Luftloch80 <notifications@github.com> geschrieben:

…

Good Job. Working with My Osram Plug, but the Osram Classic Light only Publishes this two values No on/off or brigntness zigbee2mqtt:info 2019-1-14 08:22:31 MQTT publish: topic 'homie/0x8418260000073d64/$stats/uptime', payload '180' zigbee2mqtt:info 2019-1-14 08:22:31 MQTT publish: topic 'homie/0x8418260000073d64/$stats/interval', payload '60' — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADveuPGpvDMkGSWGWFoLrwu1K90TPV2ks5vDD7xgaJpZM4XoWSl> .

[Luftloch80]

Okay. Because when i Pair the Bulbs in ioBroker they work normally

[Luftloch80]

These are the payloads i get

mosquitto_sub -h localhost -t 'homie/0x8418260000073d64/#'
3.0.1
0x8418260000073d64
127.0.0.1
CA:DE:AD:BE:EF:FF
zigbee2mqtt
zigbee2mqtt
1.0.1
node
uptime
60
39363
0x8418260000073d64
OSRAM/AC03641
vendor,model,address,description
OSRAM
Vendor

string
AC03641
Model
string
0x8418260000073d64
ZigBee Address
string
LIGHTIFY LED Classic A60 clear
Description
string
ready
39423
60
39483
60
^C

[toeCUTT3R]

Hi

Newborn zigbee2mqtt user here.

I'm using Openhab2 2.4, and was reading that Openhab2 does not support output formatting for JSON.
My usecase is controlling the brigtness of some Ikea Trådfri GU10 bulbs.

I modified a python mqtt script from https://pypi.org/project/paho-mqtt/ to accept non-JSON data from openhab and format it as JSON
before sending it to the zigbee2mqtt//set topic.

The reverse scenario should be possible as well so that one could decode json data into separate topics.

I have not tested it on my own installation yet, just on the pc and on the global iot.eclipse.org broker.

Sorry about the messy looking code block, it's one of my first posts...

The data flow is:
OpenHab -> (mqtt topic) zigbee2mqtt//brightness -> Python Script -> (mqtt topic) zigbee2mqtt//set {"brightness":"123"}

Cheers

`import paho.mqtt.client as mqtt
import json
import paho.mqtt.publish as publish

The callback for when the client receives a CONNACK response from the server.

def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
client.subscribe("zigbee2mqtt/+/brightness")

The callback for when a PUBLISH message is received from the server.

def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
key = msg.topic.rsplit("/", 1)[-1]
newtopic = msg.topic.rsplit('/', 1)[0] + "/set"
cleandata = str(msg.payload) #payload arrives like b'4'
cleandata = cleandata.split("'")[1] #remove wrappings b'' - don't do this on linux
data = {}
data[key] = cleandata
json_data = json.dumps(data)
client.publish(newtopic,json_data)

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect("iot.eclipse.org", 1883, 60)
client.loop_forever()
`

The finished version running on the pi

[toeCUTT3R]

I just tried the script and can confirm, it works.
All I did was to change the mqtt broker ip to that of my own mosquitto instance and published a number for desired brigtness level, like this:

mosquitto_pub -h 192.168.1.160 -m 25 -t zigbee2mqtt/0x000b57fffeab3e34/brightness

Hope this finds its use somewhere, until either zigbee2mqtt might include a non JSON format or Openhab2 2.5 is released.

Cheers

[Koenkk]

@stoinov I've made a first implementation, you can enable it with

experimental:
  output: attribute

Let me know if it works out for you.

[stoinov]

@Koenkk I got:

/app/lib/extension/homeassistant.js:583
            throw new Error('Home Assitant integration is not possible with attribute output!');

but after this I saw all the payloads as simple strings in their respective topics. So great initial step!

Another thing I noticed zigbee2mqtt/bridge/config', payload '{"log_level":"info","permit_join":false}' - will this be stringified too?

[Koenkk]

Home assistant discovery and output attribute cannot be used together (as the home assistant discovered configuration is not compatible).

We can change that too.

[stoinov]

Two questions:

1. Do you say that HA discovery is not available yet with attribute output, or that it's not technically possible?
2. How do you handle light payloads that include objects in some of their keys? Do they get further disassembled or does it happen only on the first level of the payload object?

I don't have a light to test, but from the code it seems like only the first level of attributes are reported and there is an object it's still passed as JSON. This should be easily fixable. I am just wondering whether to add another level to the topic like topic/key/subkey or keep them on the same level like topic/key-subkey.

Both have their pros and cons, and I personally use the second approach with the ESPHome reports, as it makes it easier to handle various inputs without adding too much exception logic in my scripts. Besides it all goes to DB where it's flattened anyway.

[Koenkk]

1. It's not possible because the discovered configuration expects JSON.
2. Thanks for the PR! will take a look at it ASAP

[stoinov]

I do not understand your concern about the discovered configuration. Can you elaborate?
If you mean that HA will expect the report payload to be JSON, then as long as you send numbers and not strings you'll be fine as they are considered valid JSON.
As I mentioned before - ESPHome does publish single number payloads and have working HA Discovery for them.

[stoinov]

BTW I should probably update documentation too explaining the new behavior...

[Koenkk]

@stoinov because now all the attributes are reported to a different topic, so e.g. https://github.com/Koenkk/zigbee2mqtt/blob/master/lib/extension/homeassistant.js#L114 won't work anymore.

[stoinov]

I see. So we need couple of ifs in the discover section, to change the topics and payloads. But it will require to update to template schema I guess. Also not sure if and why there is only xy light controlling options - is this ZigBee thing or is it just your decision?

Regardless, we'd have to convert the values to RGB with something like this.

I know it's quite the change, and most of it I think I can manage myself, but I would definitely need some help understanding some of the architectural decisions.

[Koenkk]

I don't think adding discovery configurations for both JSON and attributes output is a good idea:

• It will add a lot of extra maintenance
• The MQTT JSON light won't work

There are also options to control via RGB: http://www.zigbee2mqtt.io/information/mqtt_topics_and_message_structure.html#zigbee2mqttdevice_idset

I think a better solution here would be to have a json_attributes output option, which outputs to both JSON and attributes.

[stoinov]

I agree with you that it will be a lot of extra maintenance, but only if we want to have the light as an attribute.

The optimal solution, would be to leave all light objects as JSON payloads even with attribute option. Looking at the configurations list, only the lights section has composite payload - all other objects are single level with one entry.
So we can follow my proposal from the previous comment and just exclude light objects from this discovery changes.

We can still implement double output but I don't like this solution as our primary option since it would look and feel clumsy, especially when you try to debug MQTT.

In the RG link this explains setting up, but can you actually report in RGB or HS? There is no specific setting in the configuration list with lights, only two include XY.

[Koenkk]

That doesn't sound like a good solution me, because the light objects still have to be JSON.

The reporting is indeed in XY, as this is what zigbee uses.

EDIT: What I'm wondering about, what is your use case to use both non JSON output and Home Assistant?

[stoinov]

I am trying to setup home automation without actually using Home Assistant - just node-red parsing the MQTT payloads and inserting them into InfluxDB.
My initial setup was relying on building kind of registry using the Discovery protocol and then tagging values based on matched topics. But recently I changed it to just using the topic structure itself as a way to tag each value so I am no longer relying on Discovery.
Which I guess is a good argument to just leave the current state as it is and continue with something more productive :)

Going back to the request though - my whole idea is to leave the light as a JSON object and just skip the reporting of the value_template property in the discovery payload in all other values. This way it will still work as expected with minimal change in both JSON and attribute modes.

But as I said - it's not high priority any more and we can leave it as it is.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[olialb]

@Koenkk I see. I haven't played much with lights yet, but that does makes sense.
So what is the latest status of this request? Since it got diverted with the homie discussion I am not sure if it's being worked on in parallel.

This is really useful for openHAB integration. Do you plan to make this from experimental to "advanced"

[Koenkk]

@olialb this feature will stay in for sure, not sure if it's well tested enough atm though.

[andreasbrett]

@olialb this feature will stay in for sure, not sure if it's well tested enough atm though.

I've been using it for months now without issues.

[stoinov]

I've been using it almost a year, so it's stable. But I do not have much experience with setting it up in different environments, so not sure if we have some edge cases that could break it, so any feedback is welcomed.

I use this exclusively, without HA obviously. Any other cases?

[olialb]

I am using it with openHab and it runs stable. Make the setup much more easy....
Buy the way: Thanks for this project!!! My zigbee devices (39: Ikea, Hue, Xiaomi, Osram) work perfect!

[krzyk]

I used this feature for read-only attributes, but now I have an IKEA light bulb and would like to control it without using json.

Is it possible?
What are the topics?

zigbee2mqtt/[FRIENDLY_NAME]/brightness
zigbee2mqtt/[FRIENDLY_NAME]/color/r
zigbee2mqtt/[FRIENDLY_NAME]/color/g
zigbee2mqtt/[FRIENDLY_NAME]/color/b

How about transitions, if it is no longer a single json(/transaction) it would be hard to do transition to given state, so how it is handled right now?

I know that the setting for that (and this issue) is about output but it would be good to have json-less input also.

[kianusch]

zigbee2mqtt/[FRIENDLY_NAME]/set/brightness

seems to work.

zigbee2mqtt/[FRIENDLY_NAME]/set

works with with ON|OFF payload.

Although this works also without "output: attribute" entry in the configuration.

[0xdefec71f]

Hello all,
red through this and I am not sure how to enable the feature of non-Json output. I have put

  publish_as_json: false
  publish_as_key_topics: true

in my configuration.yaml. I tried it at mqtt: and advanced: but non worked. I still get json mqtt messages only. As it is not mentioned in documentation, can somebody please help me to set this up?
Thank you!

[0xdefec71f]

figured it out. For everybody facing the same:
put output: 'attribute' under advanced: of yaml.
Greetings!