Change in P037_MQTTImport to enable processing of JSON messages in rules

[SuksAE]

Currently the processing of JSON formated messages can only be done via the MQTTImport device.

As this device has certain limitations in processing of JSON messages, it is beneficial if it would also be possible to process JSON messages in rules. This is currently not possible, because the curly brackets used in JSON messages are handled as special characters by the rules engine.

I propose to substitute all curly brackets in unprocessed JSON messages with angle brackets before the data is put into the event queue by the MQTTImport device. So, JSON messages do not interfere with string manipulation commands in rules.

example:
Note the outcome at the last line at before and after...

rule:

On MQTTImport* Do
  LogEntry,'get: %eventname% - %eventpar% - %eventvalue1%'
  LogEntry,`get: {substring:11:17:'%eventvalue1%'}`
Endon

before:

Info   | EVENT: MQTTImport#devices/rpc/request='{"method":"setDo0","params":true}'
Info   | ACT  : LogEntry,'get: MQTTImport#devices/rpc/request - devices/rpc/request - {"method":"setDo0","params":true}'
>  LogEntry,'get: MQTTImport#devices/rpc/request - devices/rpc/request - {"method":"setDo0","params":true}'
Info   | get: MQTTImport#devices/rpc/request - devices/rpc/request - {"method":"setDo0","params":true}
>  get: MQTTImport#devices/rpc/request - devices/rpc/request - {"method":"setDo0","params":true}
Info   | ACT  : LogEntry,`get: {substring:11:17:'{"method":"setDo0","params":true}'}`
>  LogEntry,`get: {substring:11:17:'{"method":"setDo0","params":true}'}`
Info   | get: {substring:11:17:'{"method":"setDo0","params":true}'}
>  get: {substring:11:17:'{"method":"setDo0","params":true}'}

after:

Info   | EVENT: MQTTImport#devices/rpc/request='<"method":"setDo0","params":true>'
Info   | ACT  : LogEntry,'get: MQTTImport#devices/rpc/request - devices/rpc/request - <"method":"setDo0","params":true>'
>  LogEntry,'get: MQTTImport#devices/rpc/request - devices/rpc/request - <"method":"setDo0","params":true>'
Info   | get: MQTTImport#devices/rpc/request - devices/rpc/request - <"method":"setDo0","params":true>
>  get: MQTTImport#devices/rpc/request - devices/rpc/request - <"method":"setDo0","params":true>
Info   | ACT  : LogEntry,`get: setDo0`
>  LogEntry,`get: setDo0`
Info   | get: setDo0
>  get: setDo0

[tonhuisman]

Rules (and commands) already have the feature that some special characters, like percent, square- and curly- braces, can be escaped by preceding them with a backslash (\). There is code that removes these prefixes, but not yet to apply this escape mechanism. Maybe that's a better solution, with less impact, as now you don't have support for square braces..., and I wouldn't like to have a new way of handling this 😉
See hasEscapedCharacter() and stripEscapeCharacters() in StringParser.cpp

[SuksAE]

changed according description.

please review

[TD-er]

Did you test this?
Not sure if F("\\") will be 1 or 2 characters long.
Maybe better to use '\\' and see whether the compiler accepts it?

[SuksAE]

I did test it, I got following output in the console:
EVENT: MQTTImport#devices/rpc/request='\{"method":"setDo0","params":true\}'
and processing in rules also works.

It is the same code as in stripEscapeCharacters(), just the other way around...

[tonhuisman]

This got me inspired to be able to UnEscape and Escape (JSON) strings in rules, so I added that as feature, based on this PR:

(File removed, feature will be in a new PR)

Unzip the file, place the content 5597-commands-tonhuisman.diff in the root of the repo and apply the command
git apply 5597-commands-tonhuisman.diff
This will change 3 files Rules.rst, ESPEasyRules.cpp and StringParser.cpp, and add support for 2 new String functions (docs update included):

{unescape:<string-to-unescape>}
and
{escape:<string-to-escape>}

I also added the colon (:) to the characters to (un)escape and properly preserve an escaped colon in the string functions. (to make life of a user working with JSON strings easier 😉)

You can commit and push these changes in this PR, keeping it together👍

NB: These functions are only available when String Variables are included in the build.
NB2: Also have a small rule for testing:

On t2 Do
  LetStr,j1,'\{"method"\:"setDo0","params"\:true\}'
  LetStr,j2,`{unescape:[str#j1]}`
  LogEntry,'j1=[str#j1], j2=[str#j2], j3={escape:[str#j2]}'
Endon

giving this output:

00:06:26.305 : >  LogEntry,'j1=\{"method"\:"setDo0","params"\:true\}, j2={"method":"setDo0","params":true}, j3=\{"method"\:"setDo0","params"\:true\}'
00:06:26.307 : (89928) Info       | j1=\{"method"\:"setDo0","params"\:true\}, j2={"method":"setDo0","params":true}, j3=\{"method"\:"setDo0","params"\:true\}
00:06:26.309 : >  j1=\{"method"\:"setDo0","params"\:true\}, j2={"method":"setDo0","params":true}, j3=\{"method"\:"setDo0","params"\:true\}
00:06:26.311 : >  OK

Hmm, looks like it needs a little work...
Adding a few extra quotes helps, updated the code.
Oops, there's another issue in the code... working on it fixed.