Updated Rules parameter parsing (2019/11)

[TD-er]

This will be a sticky issue to explain once and for all how parameters are parsed in rules and how they should be parsed.

A parameter separator is typically a comma (,), but for convenience the separator can also be accompanied by a space or be a single space.

So these can be used to separate a parameter:

• comma
• comma space
• space comma
• space
• space space

This does impose some restrictions on the use of a comma or a space in a parameter.
Especially sending JSON to a MQTT controller can become next to impossible with these limitations.

In order to allow the comma or space in a parameter, you can wrap the parameter in quotes.

• ' (single quote)
• " (double quote)
• back-quote (added in builds after 2019/11/10)

There are multiple quotes available for this, to be able to use "the other quote" in your parameter.
For example in JSON, you need the double quote for string like values or keys.

Publish domoticz/in,{"idx":26,"nvalue":0,"svalue":"[AQ#TVOC]"}

This can be fixed by wrapping the last parameter with single quotes like this:

Publish domoticz/in,'{"idx":26,"nvalue":0,"svalue":"[AQ#TVOC]"}'

Just make sure to use the same quote as start and end of your parameter and don't use that character in the parameter itself.
N.B. these extra quotes are removed from the parameter when used, as well as trailing or leading spaces.

Currently there is a 'loophole' to trick the parsing into accepting spaces without having to wrap the parameter in quotes.

LogEntry,LED_turned "[VAR#5#!O]" "[INT#5#!O]" ([Plugin#GPIO#Pinstate#2])

This uses " (space quote) and " (quote space) which is abusing the parsing algorithm to trick it into using local wrapping. (it does add quotes to the log entry though)
Do not use constructions like these, since they may lead to hard to debug situations when changing only the slightest bit in the rules.
Also it is quite non-intuitive why this is working.
Better use it like this:

 LogEntry,'LED_turned "[VAR#5#!O]" "[INT#5#!O]" ([Plugin#GPIO#Pinstate#2])'

The reason this behavior was changed from before 2019/11 was that the old implementation could lead to unpredictable results.
Also there was no really strict definition for what a parameter would look like.
Meaning a parameter for one command would be parsed different from another command.

Now we have a single method for parsing a parameter, but on the downside some rules using the Publish command may no longer work when sending JSON.
These can be fixed very easily by adding the single quotes wrapping the parameter.

This is a breaking change, but it had to be like this or else we would have to deal with lots and lots of very odd reported issues where seemingly similar rules do work on one node and not on the other.
The old behavior was not only buggy (e.g. the last quote could still be present in the text), but also depending on the order of variables (wrapped in [ & ]).
Also it was impossible to add more parameters to existing commands which did parse a parameter to the end of a line.

[Misiu]

@TD-er does this fixes the TaskValueSet issue? I'm still very new to ESPEasy and rules (btw they are AWESOME!!!)

[TD-er]

Nope, that feature request is not yet implemented.
This issue I added mainly to have some reference to link when people ran into trouble because of the recent changes regarding more strict parsing of parameters.

[Domosapiens]

Using release mega-20191116
Seen in log: Too many arguments for cmd=SendTo

Seen this:
#2749 (comment)
And documentation:
https://espeasy.readthedocs.io/en/latest/Rules/Rules.html#some-working-examples
Section: SendTo and Publish
Needs an update ??

SendTo: SendTo ,

Imagine you have two ESP Easy modules, ESP#1 and ESP#2 In the Rules section of ESP#1 you have this:

on demoEvent do
  sendTo,2,event,startwatering //(to use the previous example.)
endon

?? Must be sendTo,2,"event,startwatering"

And ESP#2 has the rules according to the previous example (givemesomewater)

If you then enter this with the correct IP address in the URL of your browser:

http://<ESP#1-ip >/control?cmd=event,demoEvent
?? Must be http://<ESP#1-ip >/control?cmd="event,demoEvent"

Then ESP#1 shall send the event ‘startwatering ‘ to ESP#2.

It is also possible to directly order GPIO changes, like:

on demoEvent do
  sendTo,2,GPIO,2,1
endon

?? Must be sendTo,2,"GPIO,2,1"

[TD-er]

I updated your comment to use the correct Markdown syntax, as it is almost completely a form of documentation and not really a question, right?

The thing with commands embedded in the GET url is that you probably have to escape some of the characters, so it is best to cut/paste your command in the command dialog (tools page) first and then see what the URL has become.

[Domosapiens]

@TD-er ...yes I found out some aspects, but was asking for confirmation.

Am I the only one ??
-testing a version after 12 Nov
-running into trouble,
-looking in the error log
-seen something on github
I think this breaking change needs attention on the forum.

But how about this one?

On BoilerStat#Boiler_Stat do
SendToHTTP 192.168.1.81,80,/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]
EndOn

Error Log:

Too many arguments: cmd=SendToHTTP arg1=192.168.1.81 arg2=80 arg3=/control?cmd=taskvalueset extraArg4=10 lineLength=82
_SendToHTTP 192.168.1.81,80,/control?cmd=taskvalueset,10,4,0

Where to escape ?

SendToHTTP 192.168.1.81,80,"/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]"
Is that correct ??
or
SendToHTTP 192.168.1.81,80,/control?cmd="taskvalueset,10,4,[BoilerStat#Boiler_Stat]"
??

Both are Ok in the command dialog (thanks for the hint!)
(sorry, running production machine and therefore only 1 test/day possible ...)

[TD-er]

SendToHTTP 192.168.1.81,80,/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]

SendToHTTP has 3 parameters, so you should wrap a parameter in quotes when using space or comma in the parameter (or when in doubt :) )

The command + parameters are:

SendToHTTP,<IP address>,<Portnumber>,<url>

So the <url> should be wrapped in quotes like this.

SendToHTTP 192.168.1.81,80,"/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]"

As a reference, see the command reference page.
Or all commands there is a short description in the form I just gave you, with the , (comma) separator for the parameters.
When a parameter needs a comma or a space, then that one should be wrapped in quotes.
If you need double quote in the parameter (for example in JSON text), then use single quote or back quote to wrap the parameter. (just use the same for start/end of the parameter)

So these are all correct:

SendToHTTP 192.168.1.81,80,"/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]"
SendToHTTP 192.168.1.81,80,'/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]'
SendToHTTP 192.168.1.81,80,`/control?cmd=taskvalueset,10,4,[BoilerStat#Boiler_Stat]`

[TD-er]

I just added a flag in the Advanced settings to allow the rules parser to be a bit more tolerant in parsing the last parameter of some commands like Publish and sendToHttp.

Also the log is a bit more elaborate now in helping to fix the issue at hand.

With the tolerant flag enabled:

The default setting, with the flag disabled:

The rules being executed here:

 logentry,This is a test
 logentry,"This is a test"

[jimmys01]

IRSENDAC,'{"protocol":"GREE","power":"Off","mode":"Cool","temp":15,"swingv":"Off","swingh":"Off","light":"Off"}' is not being parsed anymore

Without the ' it works fine again
IRSENDAC,{"protocol":"GREE","power":"Off","mode":"Cool","temp":15,"swingv":"Off","swingh":"Off","light":"Off"}

[TD-er]

What build was working and what build isn't working anymore?

[jimmys01]

That I do not know since I just updated everything and not I do not remember what it was on there.

[TD-er]

Is it the last code we have on Github, or is it a nightly build?
Can you check if there is a flag set to be less strict on the rules parsing? (Tools => Advanced)

[jimmys01]

It is the latest git hub code. The issue is present with the flag enabled and disabled

IRSEND,{"protocol":"LG","data":"0x88C0051","bits":28}
>IRSEND,{"protocol":"LG","data":"0x88C0051","bits":28}
112513 : Info  : Command: IRSEND
112516 : Info  : IRTX: IR Code Sent: : Base32Hex RAW Code Send  Data: data":"0x88C0051
112626 : Info  : IRTX: IR Code Sent: LG Data: 0x88C0051 Bits: 28

Ok
113060 : Info  : Send Gratuitous ARP
IRSEND,'{"protocol":"LG","data":"0x88C0051","bits":28}'
>IRSEND,'{"protocol":"LG","data":"0x88C0051","bits":28}'
115481 : Info  : Command: IRSEND
115483 : Info  : IRTX: IR Code Sent: : Base32Hex RAW Code Send  Data: 

[TD-er]

@jimmys01 I noticed your last post again and was wondering if there is still an issue I need to look into.

[jimmys01]

@jimmys01 I noticed your last post again and was wondering if there is still an issue I need to look into.

IRSENDAC,'{"Protocol":"COOLIX","Power":true,"Opmode":"auto","Degrees":24,"Fanspeed":"auto","Swingv":"min","light":"true"}' does not work
IRSENDAC,"{"Protocol":"COOLIX","Power":true,"Opmode":"auto","Degrees":24,"Fanspeed":"auto","Swingv":"min","light":"true"}" does not work
IRSENDAC,{"Protocol":"COOLIX","Power":true,"Opmode":"auto","Degrees":24,"Fanspeed":"auto","Swingv":"min","light":"true"} works.

Tested with the latest commits

[TD-er]

IRSENDAC,"{"Protocol":"COOLIX","Power":true,"Opmode":"auto","Degrees":24,"Fanspeed":"auto","Swingv":"min","light":"true"}" does not work

This one is no surprise, as it cannot see any difference between end of parameter, or a quote in the parameter.

IRSENDAC,'{"Protocol":"COOLIX","Power":true,"Opmode":"auto","Degrees":24,"Fanspeed":"auto","Swingv":"min","light":"true"}' does not work

This one is surprising.
That could only happen if the command handling does not use the ESPEasy command parsing.
Will have a look at it.

IRSENDAC,{"Protocol":"COOLIX","Power":true,"Opmode":"auto","Degrees":24,"Fanspeed":"auto","Swingv":"min","light":"true"} works.

OK, good to know something is still working :)

[AloisKlingler]

just as a hint: the documentation regarding this is deceptive.
https://espeasy.readthedocs.io/en/latest/Rules/Rules.html and also https://www.letscontrolit.com/wiki/index.php/Tutorial_Rules#SendTo_and_Publish need to be adapted to either have the necessary quotes at the examples or at least have a hint for the "Tolerant last Parameter" Flag. :-)

Best regards
Alois

[budenn777]

Faced with the problem of saving the rules! Save button doesn't always work
Especially if multiple devices are added. In a home network, it does not work on a PC, but it works on a smartphone!

[tonhuisman]

What ESPEasy release (name of the .bin file that's installed), Operating System and Browser are used?
Do you have a script blocker active in your browser?

[budenn777]

ESP_Easy_mega_20230508_collection_F_ESP32_4M316k
I use Win7(x32) + Google Chrome

[budenn777]

[tonhuisman]

It's quite some time ago that I worked with Win7, not sure if Chrome still (fully) supports that OS, supplying updates etc. (IOW: You'd better not use Win 7 anymore).
It might be some weird interaction there. Can you try using Internet Explorer, to see if that does save the rules more reliable?

NB: This subject is more suitable to be discussed in the Forum

[budenn777]

Yes, I also assumed that win7 was outdated and checked on a new PC + win10, the problem did not go away...

[tonhuisman]

Well, if it works from a mobile phone, it isn't really an ESPEasy problem, I assume, it must be something on the PC-side, maybe your Antivirus software is blocking something because the ESP is accessed via HTTP? (Some AV software is really annoying, and they never give a clue in what is being blocked...)

[budenn777]

Ok, I'll look... :-)
While temporarily solved the problem of loading through Tools + File browser

[TD-er]

Ok, I'll look... :-) While temporarily solved the problem of loading through Tools + File browser

That's exactly what the JavaScript does when updating the rules.
It generates a HTTP POST to upload the text from the edit window and sends it to the upload URL of ESPEasy.

So if your browser blocks Javascript, or is unable to load the JavaScript (those files are loaded from a CDN server), then the save will not work.

See the documentation: https://espeasy.readthedocs.io/en/latest/Reference/ExternalHostedStaticFiles.html?highlight=CDN#external-hosted-static-files
You might want to save the rules_save.js to the file system of ESPEasy if your browser can't download this file from the CDN servers.

[budenn777]

Thank you! I will try

[budenn777]

Uploaded the files from the link https://github.com/letscontrolit/ESPEasy/tree/mega/static
But the button for loading rules didn't work. I opened the page code, there are errors, but I don't understand them. Here are three screenshots...

[budenn777]

[budenn777]

[budenn777]

[TD-er]

I don't know how to move posts to a new issue, but to be honest, this isn't quite on-topic to this (pinned) issue and thus highly confusing to those who need to read about the (breaking) changes that were made in rules parsing.

And since this was a breaking change for close to 3.5 years, I think it is no longer needed anyway. => can be closed.
Maybe you can create a new issue, where you reference to the first post you made about this subject?
It is essentially about saving rules and not how the rules are being processed.

So I will close this issue and please can you make a new issue with a title properly describing the issue?

[budenn777]

Yes, sorry for being off topic.