json update of Domoticz does not work anymore. #2660

berbergh · 2019-10-13T19:35:59Z

Checklist

[x ] The title of this issue is "to the point" and descriptive.
[x ] This issue describes what is happening.
This issue describes what components are affected (e.g. name of plugin/controller)
This issue describes how to reproduce it.
This issue describes when it was introduced (when known) and what version is now showing the problem.

I have...

searched the issue tracker or the forum for a similar issue. (include links when applicable)
entered a system description using "Copy info to clipboard" on the sysinfo page. (when possible)
entered the full filename of the used version (e.g. ESP_Easy_mega-20181001_test_ESP8266_4096_VCC.bin )
given a list of active plugins or controllers when applicable.
filled out all applicable fields below.

Steps already tried...

[x ] Tried a clean install (empty .bin files are included in the ZIP)
Tested previous/other build (mention which one already tested)
Tested on other node to make sure hardware isn't defective.
Verified if the problem is limited to a single plugin/controller

If you self compile, please state this and PLEASE try to ONLY REPORT ISSUES WITH OFFICIAL BUILDS!

Summarize of the problem/feature request

I try to update a dummy Air Quality device in Domoticz using a json syntax.
Rules sends:
sendtohttp myurl,8080,/json.htm?type=command&param=udevice&idx=663&nvalue=[CO2#PPM]
In the log it looks like:
sendtohttp myurl,8080,/json.htm?type=command¶m=udevice&idx=663&nvalue=390

The effect is that when I enter:
myurl:8080/json.htm?type=command&param=udevice&idx=663&nvalue=[CO2#PPM]
in a browser, I get an update in Domoticz.
With the sentence in Rules, I don't.
It used to work before.

Expected behavior

YOUR TEXT GOES HERE

Actual behavior

YOUR TEXT GOES HERE

Steps to reproduce

System configuration

Hardware:

ESP Easy version:

ESP Easy settings/screenshots:

Rules or log data

The text was updated successfully, but these errors were encountered:

TD-er · 2019-10-13T19:55:34Z

Hmm, that's a strange replacement of characters:
&para into ¶

Do you have any task or taskvalue containing para in the name?

berbergh · 2019-10-13T20:55:53Z

No I don't.
It also happens to other Rules lines, like:
SendToHTTP myurl,8080,/json.htm?type=command&param=switchlight&idx=531&switchcmd=Toggle
becomes:
SendToHTTP myurl,8080,/json.htm?type=command¶m=switchlight&idx=531&switchcmd=Toggle
But that one seems to work okay.

TD-er · 2019-10-14T07:19:09Z

Can you test by simply having the last part of that senttohttp sent to log using the command LogEntry ?
See the documentation - Command Reference

berbergh · 2019-10-14T21:18:04Z

LogEntry shows this:
LogEntry,/json.htm?type=command¶m=udevice&idx=663&nvalue=467&svalue=0

TD-er · 2019-10-14T21:45:30Z

So at least it is consistent there.

uzi18 · 2019-10-15T07:28:34Z

try this:
SendToHTTP myurl,8080,/json.htm?type=command& param=switchlight&idx=531&switchcmd=Toggle

TD-er · 2019-10-15T07:53:42Z

@uzi18 Just curious.
Why do you think that would work?
Either the sendtohttp command does interpret it as an extra parameter (don't think it will since there is no extra parameter defined) or it will change the space into %20

uzi18 · 2019-10-15T07:59:13Z

just a trick, to avoid conversion for &para, if convert to %20 it should work on domoticz server

TD-er · 2019-10-15T08:10:20Z

In the logentry it does convert it to ...& para....

I was just having a look at the code and this one seems strange to me:

ESPEasy/src/ESPEasyRules.ino

Lines 810 to 833 in a4148c8

    
           bool conditionMatch(const String& check) { 
        
             int  posStart, posEnd; 
        
             char compare = findCompareCondition(check, posStart, posEnd); 
        
             float Value1 = 0; 
        
             float Value2 = 0; 
        
             if (compare > ' ') { 
        
               String tmpCheck1 = check.substring(0, posStart); 
        
               String tmpCheck2 = check.substring(posEnd); 
        
               if (!isFloat(tmpCheck1) || !isFloat(tmpCheck2)) { 
        
                 Value1 = timeStringToSeconds(tmpCheck1); 
        
                 Value2 = timeStringToSeconds(tmpCheck2); 
        
               } else { 
        
                 Value1 = tmpCheck1.toFloat(); 
        
                 Value2 = tmpCheck2.toFloat(); 
        
               } 
        
             } else { 
        
               return false; 
        
             } 
        
             return compareValues(compare, Value1, Value2); 
        
           }

I recently split the functions of this code and it should not have changed functionality (apart from having the isFloat check now accepting leading spaces).
So I was wondering, why still perform the compareValues check when Value1 or Value2 is not a valid numerical value?

TD-er · 2019-10-15T08:19:33Z

I think the function should be changed.

timeStringToSeconds should return a bool if it is a valid timeString (and then convert it to the float value)
If either one of them is a valid time string (N.B. 12 is also valid as in 12 hours) and at least one is not a numerical, then consider both sides a time notation.
If not a time notation, then try converting them to a numerical.
If neither is a numerical return false.

uzi18 · 2019-10-15T08:27:00Z

maybe something is wrong with isFloat()/isNumerical() function

TD-er · 2019-10-15T08:42:56Z

Very well possible, only I cannot see what.
I have looked at them for a while but cannot see what's wrong.

What does bother me is that we now have several functions to check numericals.
For example the validIntFromString function may turn 12:34 into the valid integer value 12.
But isNumerical will return false on 12:34

TD-er · 2019-10-15T14:54:19Z

OK, took me a while, but I can now say that the change of &para into something else is a browser change in the logviewer
When printing the log to serial it does show perfectly fine.

254535 : Rule line  : LogEntry,"/json.htm?type=command&param=udevice&idx=663&nvalue=[rssi#rssi]"
254575 : Rule action: LogEntry,"/json.htm?type=command&param=udevice&idx=663&nvalue=-59"
254576 : ACT  : LogEntry,"/json.htm?type=command&param=udevice&idx=663&nvalue=-59"

(I added the "Rule line" and "Rule action" lines here for debugging to see what may change the string)

uzi18 · 2019-10-15T15:17:11Z

ok so not a bug

TD-er · 2019-10-15T15:23:17Z

Well, I'm still looking into the other parts of the code matching and parsing numericals.

berbergh · 2019-10-15T16:23:09Z

Interesting finding, but if the sentence remains intact, why does it noet work anymore. The same sentence did work some odd weeks ago.

TD-er · 2019-10-15T16:26:14Z

I did find some things with the compare functions which could in some edge cases yield incorrect results.
So I'm now working on a fix for it and will have a test build ready soon (past 20h, due to dinner and putting daughter to bed)
Would be nice though if it could still be included in the next build.

TD-er · 2019-10-15T16:34:17Z

Can you test this quick build:
normal_core_260_sdk222_alpha_ESP8266_4M1M_testrules_issue2660.bin

berbergh · 2019-10-15T16:36:26Z

Fair enough. Daughters are very important.
But for your information, I tried

And it works correctly in Domoticz, but the string is still weird:
sendtohttp myurl,8080,/json.htm?type=command¶m=udevice&idx=663&nvalue=380&svalue=0

Detecting time compare and value compare could give false matches in some cases.

berbergh · 2019-10-15T16:44:52Z

Your code won't load. The Wemos quits working and has to be power cycled

TD-er · 2019-10-15T16:47:13Z

Hmm, that's strange.
I did check it only with a custom build, and made a 'normal build' just for you to test.
This is the one I did flash to my unit (not all plugins included)
Custom 4M1M issue 2660.bin

berbergh · 2019-10-15T16:48:36Z

Something else is wrong. I can't even load the latest previous release anymore

berbergh · 2019-10-15T19:56:58Z

I managed to load your code. Unfortunately it does not help.

TD-er · 2019-10-15T20:04:21Z

OK, thanks for testing.
Can you get some logs on what it tries to send and also from a working node to see what the URL used to be?
What plugin is collecting the data?
Is there any task with the same name?

Can you give the output of the /json URL?

uzi18 · 2019-10-15T20:20:33Z

first you should check this url in your browser:
http://myurl:8080/json.htm?type=command&param=switchlight&idx=531&switchcmd=Toggle

berbergh · 2019-10-15T21:27:30Z

myurl actually is my Domoticz host on a rpi.
Your string returns this:
{
"status" : "OK",
"title" : "SwitchLight"
}

uzi18 · 2019-10-15T21:30:29Z

so it should work, you can try to use ip address instead myurl in rules

berbergh · 2019-10-15T21:31:25Z

@TD-er The device collecting the data is a MHZ-19 CO2 sensor. It is only available in test mode.
The data I see in the log is:
MHZ19: PPM value: 450 Temp/S/U values: 25/0/0.00
12871493: EVENT: CO2#PPM=450.00
12871518: ACT : sendtohttp myurl,8080,/json.htm?type=command¶m=udevice&idx=663&nvalue=450&svalue=0

And the rules are:
on CO2#PPM do
sendtohttp myurl,8080,/json.htm?type=command&param=udevice&idx=663&nvalue=[CO2#PPM]&svalue=0
LogEntry,/json.htm?type=command&param=udevice&idx=663&nvalue=[CO2#PPM]&svalue=0
endon

TD-er · 2019-10-15T22:36:47Z

Can you also give the output of the /json url of ESPeasy?
Just to see if there's something fishy with the names of the names of the plugin or other plugins.
Thing is, I also added some caching to the name lookup of the tasks + values and I just want to make sure the error is not happening in there.

Does Domoticz also have some hints in the logs to see what's going on?

berbergh · 2019-10-16T05:47:08Z

I have no clue how to get the output of espeasy. In the log of domoticz are no hints at all.

And last but not least, i am away from home for a week.

TD-er · 2019-10-16T07:39:28Z

The json page is just in the browser.
Tools => Show JSON

Or you use the url http://<ip>/json
It is just plain text, so cut and paste.

…#2660) The changed rules parsing may have added some spaces in replaced variables. Adding a trim() call to the parseString does remove any existing or added spaces in the parameter.

TD-er · 2019-10-25T08:37:33Z

I made a test build which also includes a lot of other changes.
Can you please test it to see if sendtohttp now works like it should and also if I broke something else?

berbergh · 2019-10-25T12:59:07Z

Sure. I will do that today or tomorrow. Did you by any chance include pmsx003 dust module in it? Op vr 25 okt. 2019 10:37 schreef Gijs Noorlander <notifications@github.com>:

…

I made a test build <https://www.dropbox.com/s/7jrwom3mhluspbz/ESPEasy_mega-20191016-11-PR_2667.zip?dl=0> which also includes a lot of other changes. Can you please test it to see if sendtohttp now works like it should and also if I broke something else? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2660?email_source=notifications&email_token=AA4CMGEXQXDVS3JC2J5D7LLQQKV6FA5CNFSM4JAIKOLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECHULLY#issuecomment-546260399>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4CMGAXOPGH4JN4BD42BNDQQKV6FANCNFSM4JAIKOLA> .

TD-er · 2019-10-25T13:12:02Z

Hmm P053 is included in the "testing" build and those failed to build in my test build due to size restrictions.
So I'm afraid those are not included in the build I linked.

berbergh · 2019-10-26T13:56:20Z

@TD-er Your last test build works with respect to the json update.
I tested:

Build:⋄	20104 - Mega
System Libraries:⋄	ESP82xx Core 2_5_2, NONOS SDK 2.2.1(cfd48f3), LWIP: 2.1.2 PUYA support
Git Build:⋄
Plugins:⋄	46 [Normal]
Build Md5:	727386f31985568f01f4d7798f8c07
Md5 check:	passed.
Build Time:⋄	Oct 25 2019 03:41:41
Binary Filename:⋄	ESP_Easy_mega-20191016-11-PR_2667_normal_ESP8266_4M1M.bin

Would like to have Nextion and the PMSx003 included :-)

TD-er · 2019-10-26T14:42:08Z

Can you try custom_ESP8266_4M1M_issue2660_Nextion.bin this one?

berbergh · 2019-10-26T15:43:28Z

Works fine on DUST and Nextion.

TD-er · 2019-10-26T18:06:34Z

Thanks for testing.
Good to hear I did at least fix it for this issue.
Let's hope it is now also fixed for the other related issues.

[Rules] Improve rules matching (#2660)

TD-er mentioned this issue Oct 14, 2019

MQTT sending to Openhab broken #2662

Closed

TD-er added a commit to TD-er/ESPEasy that referenced this issue Oct 15, 2019

[Rules] Improve rules matching (letscontrolit#2660)

575ec3e

Detecting time compare and value compare could give false matches in some cases.

TD-er mentioned this issue Oct 24, 2019

sendtohttp malformed request, release mega-20191003 #2681

Closed

TD-er mentioned this issue Oct 25, 2019

[Rules] Improve rules matching (#2660) #2667

Merged

TD-er mentioned this issue Oct 25, 2019

[Rules] Parsing of [task#var] syntax changed since 20190928 #2657

Closed

TD-er closed this as completed in #2667 Oct 27, 2019

TD-er added a commit that referenced this issue Oct 27, 2019

Merge pull request #2667 from TD-er/bugfix/rules_parsing_issue2660

68cae55

[Rules] Improve rules matching (#2660)

json update of Domoticz does not work anymore. #2660

json update of Domoticz does not work anymore. #2660

Comments

berbergh commented Oct 13, 2019

Checklist

I have...

Steps already tried...

If you self compile, please state this and PLEASE try to ONLY REPORT ISSUES WITH OFFICIAL BUILDS!

Summarize of the problem/feature request

Expected behavior

Actual behavior

Steps to reproduce

System configuration

Rules or log data

TD-er commented Oct 13, 2019 • edited

berbergh commented Oct 13, 2019 • edited

TD-er commented Oct 14, 2019

berbergh commented Oct 14, 2019 • edited

TD-er commented Oct 14, 2019

uzi18 commented Oct 15, 2019

TD-er commented Oct 15, 2019

uzi18 commented Oct 15, 2019

TD-er commented Oct 15, 2019

TD-er commented Oct 15, 2019

uzi18 commented Oct 15, 2019 • edited

TD-er commented Oct 15, 2019 • edited

TD-er commented Oct 15, 2019

uzi18 commented Oct 15, 2019

TD-er commented Oct 15, 2019

berbergh commented Oct 15, 2019

TD-er commented Oct 15, 2019

TD-er commented Oct 15, 2019

berbergh commented Oct 15, 2019

berbergh commented Oct 15, 2019

TD-er commented Oct 15, 2019

berbergh commented Oct 15, 2019

berbergh commented Oct 15, 2019

TD-er commented Oct 15, 2019

uzi18 commented Oct 15, 2019

berbergh commented Oct 15, 2019

uzi18 commented Oct 15, 2019

berbergh commented Oct 15, 2019

TD-er commented Oct 15, 2019

berbergh commented Oct 16, 2019 • edited

TD-er commented Oct 16, 2019 • edited

TD-er commented Oct 25, 2019

berbergh commented Oct 25, 2019 via email

TD-er commented Oct 25, 2019

berbergh commented Oct 26, 2019

TD-er commented Oct 26, 2019

berbergh commented Oct 26, 2019

TD-er commented Oct 26, 2019

TD-er commented Oct 13, 2019 •

edited

berbergh commented Oct 13, 2019 •

edited

berbergh commented Oct 14, 2019 •

edited

uzi18 commented Oct 15, 2019 •

edited

TD-er commented Oct 15, 2019 •

edited

berbergh commented Oct 16, 2019 •

edited

TD-er commented Oct 16, 2019 •

edited