Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generic HTTP don't work since ESP_Easy_mega-20200204_dev_ESP8266_4M1M.bin #3054

Closed
sracing opened this issue May 5, 2020 · 22 comments
Closed
Labels
Category: Controller Related to interaction with other platforms Status: Needs Info Needs more info before action can be taken Type: Bug Considered a bug

Comments

@sracing
Copy link

sracing commented May 5, 2020

I am using several Wemos D1 with ESP Easy since quite a while and was doing a firmware update on those ESP Easy devices.

On any of my devices, when using a firmware 'ESP_Easy_mega-20200204_dev_ESP8266_4M1M.bin' (OR NEWER), I am not able to receive data from the ESP Easy 'Generic HTTP' Controller anymore.
With any firmware 'ESP_Easy_mega-20191208_dev_ESP8266_4M1M.bin' (OR OLDER) it works without any issue.

  • Note: A very similar issue was reported on 7 Feb under Generic HTTP don't work since ESP_Easy_mega-20200204_normal_ESP8266_4M1M.bin #2878 - Generic HTTP don't work since ESP_Easy_mega-20200204_normal_ESP8266_4M1M.bin refering to a wrong URL encoding (fixed)
  • The log on a Debug dev / Kernel level does not show any failure.
  • That behaviour can be reproduced by switching between firmware versions.
  • I would not see a relation to rules as some of my Wemos D1 devices which are not working with the newer firmware are operated without using any rules - just using the 'Generic HTTP' Controller to forward values
@TD-er
Copy link
Member

TD-er commented May 5, 2020

This is communication between ESPEasy nodes?

@TD-er
Copy link
Member

TD-er commented May 5, 2020

What are you trying to send?
Can you give an example of such an URL?

@TD-er TD-er added Category: Controller Related to interaction with other platforms Status: Needs Info Needs more info before action can be taken Type: Bug Considered a bug labels May 5, 2020
@sracing
Copy link
Author

sracing commented May 5, 2020

Its communication between ESPEasy (Wemos D1 Pro) and my SmarthomeNG controller (similar Domotics or FHEM).

When I capture the HTTP requests, this is what I can see with the different firmware versions:
ESP_Easy_mega-20191208_dev_ESP8266_4M1M.bin shows:
http://192.168.178.44:8384//shNG/ws/items/esp_easy.strom.count/0

ESP_Easy_mega-20200204_dev_ESP8266_4M1M.bin shows:
http://192.168.178.44:8384/%2f%2fshNG%2fws%2fitems%2fesp_easy.strom.count%2f0

@TD-er
Copy link
Member

TD-er commented May 5, 2020

OK, that last part makes it clear it is for sure related to the fix you linked.

What is the string you use in the controller to generate this URL?

@sracing
Copy link
Author

sracing commented May 5, 2020

Under Controller Publish: /shNG/ws/items/%sysname%.%tskname%.%valname%/%value%

@TD-er
Copy link
Member

TD-er commented May 5, 2020

Check!
I will have a look at it. I think it is really similar to the other fix.
Maybe also other controllers are affected here.

@uzi18
Copy link
Contributor

uzi18 commented May 5, 2020

it is incompatibility with older release but SmarthomeNG controller should respect url encoded link, so for me it is more like bug there

@sracing
Copy link
Author

sracing commented May 5, 2020

Beside the pure url encoding, it looks like there is a specific bug if you precisely compare both outputs.
The 20200204 firmware has http://192.168.178.44:8384/%2f%2fshNG ..... with "/%2f%2f" right after the port definition - which would be encoded as "///" ( 3 x the slash character ) while it should be only 2x slash right after the port definition.

@uzi18
Copy link
Contributor

uzi18 commented May 5, 2020

@sracing try this: shNG/ws/items/%sysname%.%tskname%.%valname%/%value%
controller automaticly adds root / to controller publish string

@uzi18
Copy link
Contributor

uzi18 commented May 5, 2020

@TD-er
Copy link
Member

TD-er commented May 5, 2020

I have looked at the code quite intensively and at least in the URL encode functions in ESPEasy, there is nothing wrong so it seems.


244382: SYS : 4.00,15640.00,33.89,0.00
244398: //shNG/ws/items/ESP_Easy.test.uptime/4.00
244401: //shNG/ws/items/ESP_Easy.test.freeheap/15640.00
244405: //shNG/ws/items/ESP_Easy.test.load/33.89
244408: //shNG/ws/items/ESP_Easy.test./0.00
244470: HTTP : C008 written to client (160/160)
244470: HTTP : C008 closing connection
244492: HTTP : C008 connecting to nuc.lan:80

This is in the log.

I will now look at the link @uzi18 posted...

@TD-er
Copy link
Member

TD-er commented May 5, 2020

What we're doing is the "method of choice" according to that link.

Or at least that's what is being created as can be seen in the logs.
I has troubles getting the data from the Apache logs, so I can't see what is actually being sent, but at least the data processed in the controller is generated with the correct encoding.

@uzi18
Copy link
Contributor

uzi18 commented May 5, 2020

it can be some differents between esp/esp32
but first we should compare generic http advanced controller to generic http

@TD-er
Copy link
Member

TD-er commented May 5, 2020

Ah finally found the logs....

192.168.10.65 - - [05/May/2020:23:04:25 +0200] "GET //shNG/ws/items/ESP_Easy.test./0.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:26 +0200] "GET //shNG/ws/items/ESP_Easy.test.uptime/17.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:26 +0200] "GET //shNG/ws/items/ESP_Easy.test.freeheap/15640.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:26 +0200] "GET //shNG/ws/items/ESP_Easy.test.load/25.10 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:26 +0200] "GET //shNG/ws/items/ESP_Easy.test./0.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:27 +0200] "GET //shNG/ws/items/ESP_Easy.test.uptime/17.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:27 +0200] "GET //shNG/ws/items/ESP_Easy.test.freeheap/15640.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:27 +0200] "GET //shNG/ws/items/ESP_Easy.test.load/25.10 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"
192.168.10.65 - - [05/May/2020:23:04:27 +0200] "GET //shNG/ws/items/ESP_Easy.test./0.00 HTTP/1.1" 403 452 "-" "ESP Easy/20106/May  5 2020 11:14:39"

So just to be sure it isn't something that may have been fixed in the last few days, I will make a test build of this code.

@TD-er
Copy link
Member

TD-er commented May 5, 2020

@Tom-Bom-badil
Copy link

@sracing:
Side note - running ESPEasy here on a bunch of nodes with UDP communication to shNG, no problems since April 2018 (besides the reboot issues ESPEasy had in between, but that's a different story).

Details here, if of interest.

Probably not solving the problem, but a workaround which is more straight than untilizing the more complex http protocol.

@uzi18
Copy link
Contributor

uzi18 commented May 5, 2020

have looked into sources of shNG and it supports mqtt, it is far more reliable protocol

@uzi18
Copy link
Contributor

uzi18 commented May 5, 2020

@Tom-Bom-badil did you use generic udp protocol? screenshots are unavailable for external users... so hard to guess

udp is fast and lightweight plus no blocking :)

@TD-er
Copy link
Member

TD-er commented May 5, 2020

Well other protocol options aside, the described issue does seem like an encoding problem with the HTTP controller and since I cannot reproduce it on my node here based on the current code base, I would really like to know if this test build is working as expected or not.

If that's working as expected, you can all help into finding the best suitable protocol, as I do agree http is probably not the best one for the job.
But first let's make sure there is no hidden bug in it.

@sracing
Copy link
Author

sracing commented May 5, 2020

Nearly a fix, but ..

1 - the %valname% now seems to be a combination of the device 'Unit Name' (esp_easy) and 'Unit Number' (1) - unless there is another explanation where the '_1' comes from....
2 - root only has 1 slash where the 'old '' version had 2x the slash

Output for issue3054_ESP_Easy_mega_20200505_dev_ESP8266_4M1M.bin is:
http://192.168.178.44:8384/shNG/ws/items/esp_easy_1.strom.count/0

Output for ESP_Easy_mega-20191208_dev_ESP8266_4M1M.bin was:
http://192.168.178.44:8384//shNG/ws/items/esp_easy.strom.count/0

@TD-er
Copy link
Member

TD-er commented May 5, 2020

That is part of the %sysname%. You can see it is in front of the .
That's configurable in the settings. (Config => Append Unit Number to hostname:)

@sracing
Copy link
Author

sracing commented May 6, 2020

Makes a lot of sense, but it somewhat looks that the 'Append Unit Number to hostname:' switch was not considered in %sysname% for firmware 20191208 and older.

However, the firmware fix now works fully (I just need to disable that 'Append Unit Number to hostname:' switch after the update).

Great work. Thanks for doing this in such short time.

@TD-er TD-er closed this as completed May 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Controller Related to interaction with other platforms Status: Needs Info Needs more info before action can be taken Type: Bug Considered a bug
Projects
None yet
Development

No branches or pull requests

4 participants