Rules: Publish to MQTT not working any more

[antro31]

Checklist

• The title of this issue is "to the point" and descriptive.
• This issue describes what is happening.
• This issue describes what components are affected (e.g. name of plugin/controller)
• This issue describes how to reproduce it.
• This issue describes when it was introduced (when known) and what version is now showing the problem.

I have...

• searched the issue tracker or the forum for a similar issue. (include links when applicable)
• entered a system description using "Copy info to clipboard" on the sysinfo page. (when possible)
• entered the full filename of the used version (e.g. ESP_Easy_mega-20200305_normal_ESP8266_4M1M.bin )
• given a list of active plugins or controllers when applicable.
• filled out all applicable fields below.

Steps already tried...

• Tried a clean install (empty .bin files are included in the ZIP)
• Tested previous/other build (mention which one already tested)
• Tested on other node to make sure hardware isn't defective.
• Verified if the problem is limited to a single plugin/controller

Summarize of the problem/feature request

In the two latest versions
Publish command does not work anymore.
info log says:
ACT : Publish espsensor_01/attributes,''
instead of

ACT : Publish espsensor_01/attributes,'{ 'IP': '192.168.1.14', 'last_set': '17:42:35' }'

Expected behavior

Publication through MQTT of the following topic and content.
espsensor_01/attributes,'{ 'IP': '192.168.1.14', 'last_set': '17:42:35' }'

Actual behavior

Publication of espsensor_01/attributes,''

Steps to reproduce

1. Created rules/system/boot.txt applicable to system#boot to trigger the first timer
2. Creates rules/rules/timer.txt applicable to rules#timer at each timer request. See below for tules content.
3. Log shows publication espsensor_01/attributes,''

Note: same behaviour for ESP_Easy_mega-20200301_normal_ESP8266_4M1M

System configuration

Hardware: PLATFORMIO_ESP12E / NodeMCU

ESP Easy version: mega-20200305

Rules or log data

On Rules#Timer=1 do 
Publish %sysname%/attributes,'{ "IP": "%ip%", "last_set": "%systime%" }'
timerSet,1,60 // Set timer to trigger next event in 1 minutes
endon

[TD-er]

What was the last version your rules still worked?
Have you read the sticky issue? #2724
JSON should not have a single quote in the JSON string, but double quotes, so I don't see yet why the last given rules should not work.
Double check you don't have strange characters in your rules, which may slip in when copy/paste from some website or work document (nbsp instead of space, left/right double quote, etc)
When in doubt, also try to create a logentry using the same string as parameter.

Have you tried the "relaxed rules parsing" option in the Tools -> Advanced page?

[antro31]

Hi, last version working was 20200204.
I tried several things:

• Removed spaces in the "JSON" part of the publication ==> No effect
• Used Relaxed rules parsing ==> No effect

It overall looks to me that as soon as you try to put someting between singletons '{', this does not work anymore.
And by the way, this does not come from log entry. MQTT broker clearly does not receive anything in that case.

[TD-er]

Ah, I think I know what may have caused this and it is indeed a new change.
Never thought of this side effect.

See new replacement commands
@uzi18 Any suggestion here?

[TD-er]

OK, I have thought about it....
These new commands have some string to match. So I can reject any changes if the command does not match.
Not entirely sure though, how the nested parsing may work.
Let's have a look at it tomorrow morning. It may be hard to find a proper fix while being so tired as I am now...

[TD-er]

The breaking change was introduced here: #2828

Looks like changing the new rules commands (substring etc.) into using curly braces does break all attempts to make JSON.
So I guess I do have to try and "keep" the {...} part if it does not match a command.
Luckily these commands can never be considered JSON, as JSON needs strings to be wrapped in double quotes, so there is no conflict here.

It has to be marked as "this is JSON" or something like that or else the nested parsing will also not work. This means the { and } have to be replaced by something else if they do not match a command and after this parsing is complete, the temp characters should be changed back into their respective curly braces again.

I am thinking of using 2 characters from the "extended" ASCII code or values < 32 as temp chars, and let's hope it will not introduce any issues when Unicode is uses in strings.
So my suggestion is to use:

• 0x02 (STX - Start of Text)
• 0x03 (ETX - End of Text)

If anyone already can give some examples of Unicode that will be broken, please let me know.

[TD-er]

OK, I think I fixed it now and you can even use these new commands in generating JSON if you like:

On Rules#Timer=1 do 
logentry,'{ "IP": "{substring:0:3:%ip%}", "last_set": "%systime%" }'
timerSet,1,60 // Set timer to trigger next event in 1 minutes
endon

Results in:

285495 : Info  : EVENT: Rules#Timer=1
285506 : Info  : ACT  : logentry,'{ "IP": "192", "last_set": "09:45:38" }'
285509 : Info  : Command: logentry
285510 : Info  : { "IP": "192", "last_set": "09:45:38" }
285512 : Info  : ACT  : timerSet,1,60

[antro31]

Good. Looking forward to testing latest release then.

[TD-er]

Test build: ESPEasy_mega-20200305-18-PR_2933.zip

[antro31]

Just tested it. Rules are working now.

[uzi18]

@TD-er sorry did not get notification about your comment, maybe we can replace it with %curlyopen% %curlyclose% or shorted version %co% %cc% and parse line template at the end ?

[TD-er]

@uzi18 I have now replaced it with a single byte char, so we prevent memory allocations.
If that doesn't work (due to Unicode collisions), we can always replace it with something else.