[SendToHttp] Add option to add credentials in the URL #4145
Conversation
The hostname can now be: - hostname or IP - username:pass@hostname or IP
|
@thomastech Can you test this PR to see if it is now working for your cameras? |
|
I manually patched the latest official release. The "SendToHTTP connection failed" log message is no longer seen. But the cameras still ignore the commands. Here's my rules, which simply post an alternating ping/pong text message on the camera feed: Here is the logs: I've also tested with the "SendToHTTP wait for ack" enabled in advanced settings. BTW, using "bad" credentials does not change the log messages. So hard to know if my bad fortune is a credential arg or command arg issue. But as discussed before, sending the commands as a URL in a browser is always successful. Success is reported as "OK" in the browser window. Lastly, the logs do not have the "WiFi : Set TX power to " messages. Those are horrific spammers, so I had to suppress them.
|
Those can be disabled by setting the WiFi TX power to always use max power. Can you try to get a Wireshark dump when trying to send the same URL from your browser? |
|
Good to know that the logged Tx power messages can be disabled via the web interface. My cameras require basic auth (base64 encoded), so you did the right thing. Please see section 3.2 in this doc for details: I tried Wireshark years ago and gave up on it. But I'll install it again and post a dump.
|
Make sure to only include data to/from that host. |
|
I appreciate the concern about sensitive data. I don't see any, so hopefully the posted info is safe. Here's some filtered data (ip.dst == 192.168.1.240) that we can start with (local PC is *.175, camera is *.240): |
|
From the ESPEasy Device the camera commands are sent every 15 secs via the timer rules. But the camera's IP does not show any ESPEasy related http activity in Wireshark. The ESPEasy WiFi device is at 192.168.1.20 and the POE Ethernet camera is at 192.168.1.240. I have filtering set to ip.dst == 192.168.1.240. Is it possible for Wireshark to monitor WiFi and/or Ethernet data that's not sent or received by my PC? Advice on how to do this would be great. BTW, I did not install any controllers or plugins. Do I need to install a HTTP controller to use the sendToHTTP?
|
No, not for HTTP, a Controller is only needed for MQTT, because of the different server protocols / formats used. |
|
I think you can also filter with Wireshark using Anyway, it seems your browser is not using the basic authentication. |
I'm only able to capture 255.255.255.255 UDP data from the ESPEasy Device. I haven't found the magic words that allow me to monitor its http traffic going directly to the camera. No worries, I'll skip this for now.
Is this a nice way of saying that I'm out of luck? Or do you have any other secrete weapons for me to try? EDIT: I'll try the latest commit.
|
|
@thomastech Can you try again with the latest commit I added to this PR? |
See my last post I made almost at the same moment as your post. |
|
Installed the latest HTTP.cpp Not working yet. I see a http 401 in the log messages. Here's an excerpt: EDIT:
Agreed, I noticed that it uses digest authorization. The camera's API Document summarizes both (basic and digest) auth steps, summarized as follows: So a 401 Unauthorized message is expected. But the client response needs to be a base64 encoded user and password. Any advise on doing that would be appreciated.
|
|
[...]
I am sending the base64 encoded user/pass: Only difference seems to be that I send it in the header and the text suggests to send it as a response. |
|
OK, got some ideas here...
If that fails, I guess the camera really expects a host to call the |
That's what I'm doing in my browsers tests. Works fine.
The camera's login is an alternate way to send the credentials. I don't think anyone does it that way because it's more efficient to include the credentials with a command instead. But I've tried it this way and as before, works in a browser but not ESPEasy.
According to my searches on authentication, the credentials are suppose to be sent as a response to the 401. This acts as a challenge and it provides the response format. So it seems the camera is following the typical base auth method.
|
|
Well I tried it on some other devices and they accept the credentials in the header, even if it is the first query from that host. Have you tried to 'fake' accessing the login page by a separate call using sendtohttp and then the actual url you need with the credentials? |
|
@TD-er, thanks for the continued support. I appreciate your advice. Today I spent several hours tweaking ESPEasy basic auth code with the hope of finding a solution. Plus there was a lot of searching through github and tech forums. Late this afternoon I finally found the reason for all this frustration. Spoiler: The camera's API docs don't mention that basic authorization was disabled on recent firmware. Dahua and Amcrest cameras (the two brands I use) now default to Digest Authorization. Apparently this was done to improve security. And as usual I didn't get the memo. I would volunteer adding the Digest Auth feature to ESPEasy, but I don't think I'm the right guy for the job. But I can certainly help with test/validation. :) And for sure, thanks for all the help and advice. Would never have found the reason for why it does not work without your assistance.
|
|
BTW, while I was hacking/modifying code I found that the http.getString() library function used in send_via_http() is broken (see _CPlugin_Helper.cpp). It only returns empty strings on my ESP8266 build. In a perfect world the returned http GET payload message would indicate if basic versus digest authorization is required by the device.
|
|
Thanks for the update. |
There's a built-in mechanism for that. When the host returns the 401 error it also provides a payload that reports which authorization method to use. This message can be used to select the correct one.
|
Do you have a link or dump on what string will be returned? |
|
Basic Auth payload example: Digest Auth payload example: Note: The client calculates the digest authorization using the payload information. The username, password, nonce, HTTP method and URI are hashed with MD5, and then sends it back to the server. The protocol is well documented for both methods. For example, some details can be found here: BTW: As previously mentioned, the send_via_http() function makes a call to http.getString(), which should return this special payload. But getString() returns an empty string on my ESP8266 builds (did not try ESP32). I found the ESP8266HTTPClient.h header file (for getString) is missing. This does not cause a build error because it appears a pre-compiled library is used instead. That's as far as I got with my debug efforts to find the reason for the missing http GET payload. If you can get it functional I will capture a real payload from my camera to use as a reference.
|
For @thomastech to test :)
|
OK, added some code based on example code. So please let me know if this is working, or at least not blowing up your ESP. |
|
Wow, you work fast! Added the changes and the ESP8266 is running. But the camera is not responding to the text overlay commands. I have a feeling that will come soon. Your commit is correctly detecting the Digest Auth. Here's the log: Here is the 401 Unauthorized payload that is returned by the camera: Here are the authorization variables used in getDigestAuth():
|
|
It's working! The test camera's image now has a text message that toggles between "PING" and "PONG". So your Digest Auth code is a success. I need to shutdown for the day. So I'll post some details for your refreshed eyes while mine are still counting sheep. Here is what I found: The http.addHeader() function was not working with the Authorization keyword name. My temporary fix was to add this statement before calling it (see file_CPlugin_Helper.cpp):
So the updated code snippet looks like this : It's clear what this temporary fix is doing (TL;DR, zeros the string length). But I don't know if nulling the _base64Authorization var should remain a final solution. But the little details certainly point to what needs your further code review. Thomas |
|
OK, that makes sense. Maybe if this works, we can also start automating the sheep counting. That will save a lot of time ;) |
This also fixes a timeout issue on ESP32 as the WiFiClient timeout is set in seconds (!!!!!) See: espressif/arduino-esp32#6676 Extremely frustrating design choice as it has taken extreme amounts of time to debug those issues!
|
I applied your latest commit (updated 11 files). Compiles and runs. I'm using a Wemos D1 Mini with NORMAL_ESP8266_4M1M. No plugins or controllers, just rules and NTP. BTW, I've tested another command (Day/Night Profile switching) on a Amcrest 5MP Turret camera. Success. So Digest Auth is working with Dahua and Amcrest brand cameras (they are both made by Dahua). Thanks again for adding Basic and Digest Auth. I'm surprised I'm the only one that has asked for this. And feel very fortunate that you added it so quickly. Looking forward to seeing it in a future Mega release.
|
|
I also adapted all http-related plugins to use this function. And there was apparently a very nasty design choice on ESP32 which does set the timeout in seconds instead of msec as is done everywhere else. |
|
By the way, should I also create an event with the result of the HTTP call? |
Unfortunately I don't have any other http devices for the validation. I could test MQTT, does that help?
Maybe. Can you explain what this would do and how it is used?
|
Nope, MQTT controllers won't use this function call.
You can see it as some kind of "return value" so there will be an event with the last return value. For example something like this: |
Yes, that's useful. Returning the httpCode would be a nice touch. I expect that successful sendToHTTP would return code 200 (OK). And that value could be used by the rules to verify success.
|
|
Yep, that's the idea :) |
|
Tested latest commit (2 changed files). The http return code feature is working. Here is the rule I used to validate it: Tested both valid and invalid credentials. Success is code 200, failure is 401. Invalid camera command returns code 400. All good, just as expected.
|
|
Thanks for testing |
The hostname can now be:
As reported on the forum