JSON-RPC Method SysVar.getAll returns invalid JSON

[unverbraucht]

I'm using raspberrymatic version 2.29.23.20171216 in conjunction with Home Assistant via pyhomematic. The module calls SysVar.getAll via JSON-RPC, which returns:

{"id": 0,"version": "1.1","result": [{"id":"40","name":"$\{sysVarAlarmMessages}","type":"NUMBER","unit":"","value":"0","channelId":"65535","minValue":"0","maxValue":"65000","isLogged":false,"isVisible":false,"isInternal":true},{"id":"1314","name":"$\{sysVarAlarmZone1}","type":"ALARM","unit":"","value":"","channelId":"65535","valueName0":"$\{sysVarAlarmZone1NotTriggered}","valueName1":"$\{sysVarAlarmZone1Triggered}","isLogged":false,"isVisible":false,"isInternal":false},{"id":"950","name":"$\{sysVarPresence}","type":"LOGIC","unit":"","value":"true","channelId":"65535","valueName0":"$\{sysVarPresenceNotPresent}","valueName1":"$\{sysVarPresencePresent}","isLogged":false,"isVisible":false,"isInternal":false},{"id":"41","name":"$\{sysVarServiceMessages}","type":"NUMBER","unit":"","value":"5","channelId":"65535","minValue":"0","maxValue":"65000","isLogged":false,"isVisible":false,"isInternal":true}],"error": null}

Decoding this throws an Exception from Pythons jsons.decode: Exception: Invalid \escape: line 1 column 58 (char 57)

Running this through jsonlint.com confirms that escaping the angled brackets (the {) is not allowed. As far as I understand in JSON backslashes are supposed to be followed by a hex utf8 sequence or another backslash.

I'm unsure if I can just delete these system variables to get it running again?

[jens-maus]

You probably know that ${sysVarAlarmMessages} is an internal abbreviation for providing multi-lingual text. However, I can currently hardly follow your problem. So I would need some more information please and some example code to reproduce the problem here locally. So please provide it. In addition, please make sure that the escaping via \{ is not performed in pyhomematic itself, which I don't know. All I can say is, that this ${sysVarAlarmMessages} identifier is an internal identifier in the CCU which at display of a webpage will be replaced with the corresponding multi-lingual text.

[unverbraucht]

Of course :)

So the JSON-RPC request receives a response of MIME type application/json with a response body that contains JSON where the curly brackets have been escaped. I've confirmed with tcpdump that this is indeed in the raw response itself:

The way this escaping works seems to be invalid in JSON, after a backslash only a hex-encoded number representing the unicode char or another backslash is legit. The backslash can be just dropped because curly brackets don't need to be escaped in JSON:

JSON decoders in browsers are much more lenient than the python one, I could decode the sequence in chrome. So I assume that is the reason why the web interface can decode it - or they replace the string in the response before parsing the JSON.

There is a small example code in this gist that uses pyhomematic to retrieve the list of devices and then the system variables, which should fail:
pip3 install pyhomematic python3 homematic-test.py [...] DEBUG:pyhomematic._hm:RPCFunctions.jsonRpcPost: API-Endpoint: http://192.168.32.242/api/homematic.cgi ERROR:pyhomematic._hm:RPCFunctions.jsonRpcPost: Exception: Invalid \escape: line 1 column 58 (char 57) DEBUG:pyhomematic._hm:RPCFunctions.jsonRpcPost: Method: Session.logout

Please let me know if you need more information.

[unverbraucht]

And I just realise that this is probably more an issue of OCCU than of RaspberryMatic? Sorry about that.

[jens-maus]

This is no problem as RaspberryMatic is more or less the testbed for OCCU and will submit fixed back to upstream once they are found to be stable

[jens-maus]

Can you please provide an example without pyhomematic where the http://XXXXX/api/homematic.cgi is used and returns invalid json? This would make debugging for me way easier.

[unverbraucht]

Sure, I've found a way to use curl from the shell.

First we need the session token:

# curl -d '{"jsonrpc": "1.1", "id": 0, "method": "Session.login", "params": {"username":"Admin", "password":"PASSWORD"}}' -o - -X POST -H "Content-Type: application/json" http://IP/api/homematic.cgi
{"id": 0,"version": "1.1","result": "vVU4HVJGWZ","error": null}

In this case the session token is vVU4HVJGWZ. Then get the variables:

# curl -d '{"jsonrpc": "1.1", "id": 0, "method": "SysVar.getAll", "params": {"_session_id_": "vVU4HVJGWZ"}}' -o - -X POST -H "Content-Type: application/json" http://IP/api/homematic.cgi
{"id": 0,"version": "1.1","result": [{"id":"40","name":"$\{sysVarAlarmMessages}","type":"NUMBER","unit":"","value":"0","channelId":"65535","minValue":"0","maxValue":"65000","isLogged":false,"isVisible":false,"isInternal":true},{"id":"950","name":"$\{sysVarPresence}","type":"LOGIC","unit":"","value":"true","channelId":"65535","valueName0":"$\{sysVarPresenceNotPresent}","valueName1":"$\{sysVarPresencePresent}","isLogged":false,"isVisible":false,"isInternal":false},{"id":"41","name":"$\{sysVarServiceMessages}","type":"NUMBER","unit":"","value":"5","channelId":"65535","minValue":"0","maxValue":"65000","isLogged":false,"isVisible":false,"isInternal":true},{"id":"1314","name":"AZ1","type":"ALARM","unit":"","value":"","channelId":"65535","valueName0":"not triggered","valueName1":"triggered","isLogged":false,"isVisible":false,"isInternal":false}],"error": null}

All the backslashes in the response here are actually wrong.

[jens-maus]

I think I found the origin where the \{ is inserted:

https://github.com/eq-3/occu/blob/master/WebUI/www/api/eq3/json.tcl#L207-L208

If you comment out these two lines (the file on your CCU/RaspberryMatic is located at /www/api/eq3/json.tcl) the response contains no \{ escape sequences anymore. However, I am not sure if this might bring up any side effects, so it would be great if you could deeply test this before I can integrate a general patch for the next update of RaspberryMatic.

[jens-maus]

I just integrated a patch to fix this issue. Please test and report back if there are other issues appearing...

[unverbraucht]

Awesome, it fixes the issue with Home Assistant. A brief use of the raspimatic web ui shows no issues.