New Power meter support: HTTP(S) + JSON (Shelly 3EM, Tasmota, Volkszähler etc.)

[berni2288]

It's finally implemented :)

• Support for any HTTP(S) JSON based Power meters like Shelly 3EM, Tasmota, Volkszähler etc.
• HTTPS is also supported without certificate checks (MITM attacks are possible)
• Requests are blocking
• Very user friendly UI with HTTP request test button
• Lots of customization options: Basic Auth; Custom request header; Timeout
• You can specify the json field (Uses https://github.com/mobizt/FirebaseJson) - Example: testarray/[2]/power
• You can choose whether you want individual HTTP requests per phase, or just one (First phase can be used for the total power of all 3 phases)
• Fixed the _lastPowerMeterUpdate safety check not working anymore in the Power Limiter (the variable was always updated regardless of the success)

[madmartin]

Hello @berni2288

I am suprised and impressed.... this is exactly what I missed some minutes ago when I started playing with this OpenDTU-branch....

I will test.

[berni2288]

I've just rebased the branch on the latest changes of development branch to fix a crash :)
Please also note that you don't have to configure all 3 phases, you can just use the first phase with the total power value, if that is available in your API.

You also have to run (cd webapp && yarn build) in order to compile the webapp before you build the firmware.

[madmartin]

You also have to run (cd webapp && yarn build) in order to compile the webapp before you build the firmware.

I got that, no problem.

Here are my testing results:

First: while playing around with the JSON path examples, I figured out that one of the json examples is not properly formatted:

diff --git a/webapp/src/views/PowerMeterAdminView.vue b/webapp/src/views/PowerMeterAdminView.vue
index ed6d0ec..785b090 100644
--- a/webapp/src/views/PowerMeterAdminView.vue
+++ b/webapp/src/views/PowerMeterAdminView.vue
@@ -170,7 +170,7 @@
 
                 <h2>JSON path examples:</h2>
                 <ul>
-                    <li>total_power - { othervalue: "blah", "total_power": 123.4 }</li>
+                    <li>total_power - { "othervalue": "blah", "total_power": 123.4 }</li>
                     <li>testarray/[2]/myvalue - { "testarray": [ {}, { "power": 123.4 } ] }</li>
                 </ul>
 

I have a tasmota-driven SML power meter adapter.
I do a read test with curl and jq. The tasmota command to send is "status 8" and must be url-encoded to status%208 - this should appear in the example section or in a documentation with usage examples.

~ $ curl -s "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq
{
  "StatusSNS": {
    "Time": "2023-04-02T13:38:36",
    "Strom": {
      "Total_in": 19902.6084,
      "L1": 219.12,
      "Meter_Number": "0a01445a470002fb2769"
    }
  }
}

The actual power consumption is the "L1" value, so the jq command

~ $ curl -s "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq '.StatusSNS.Strom.L1'
219.12

gives the correct result.

Translated to the new power meter input dialog I get this result:

The whole http get response data is 124 bytes - is that really too much?
I did not read the code yet, maybe I find the solution myself. But I wanted you to give a fast feedback.

[berni2288]

Thanks, I'll change the examples.

The response buffer is 2000 bytes I think, are you sure that your URL does not require authentication?
I can check again if I find some bug...

[madmartin]

Thanks, I'll change the examples.

The response buffer is 2000 bytes I think, are you sure that your URL does not require authentication? I can check again if I find some bug...

Yes, the tasmota device does not need authentication. The curl without the -s option shows a statistic:

martin@bln9716cm ~ $ curl  "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   126    0   126    0     0   2747      0 --:--:-- --:--:-- --:--:--  2800
{
  "StatusSNS": {
    "Time": "2023-04-02T14:08:03",
    "Strom": {
      "Total_in": 19902.6850,
      "L1": 145.94,
      "Meter_Number": "0a01445a470002fb2769"
    }
  }
}
martin@bln9716cm ~ $ 

where you can see the response is 126 byte.

[berni2288]

I have saved your json to my webserver and hit the Test button and I get "Success! Power: 145.94W". So it works for me.

Can you maybe try the following CURL command?

curl \
	--header "User-Agent: OpenDTU-OnBattery" \
	--header "Content-Type: application/json" \
	--header "Accept: application/json" \
	"http://tasmota-62a4da-1242.home/cm?cmnd=status%208"

Thanks!

[madmartin]

Exactly the same:

martin@bln9716cm ~ $ curl \
        --header "User-Agent: OpenDTU-OnBattery" \
        --header "Content-Type: application/json" \
        --header "Accept: application/json" \
        "http://tasmota-62a4da-1242.home/cm?cmnd=status%208"
{"StatusSNS":{"Time":"2023-04-02T14:37:55","Strom":{"Total_in":19902.7331,"L1":0.00,"Meter_Number":"0a01445a470002fb2769"}}}martin@bln9716cm ~ $

the same with jq:

martin@bln9716cm ~ $ curl       --header "User-Agent: OpenDTU-OnBattery"        --header "Content-Type: application/json"       --header "Accept: application/json"        "http://tasmota-62a4da-1242.home/cm?cmnd=status%208" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   125    0   125    0     0   3288      0 --:--:-- --:--:-- --:--:--  3378
{
  "StatusSNS": {
    "Time": "2023-04-02T14:39:24",
    "Strom": {
      "Total_in": 19902.7340,
      "L1": 77.93,
      "Meter_Number": "0a01445a470002fb2769"
    }
  }
}
martin@bln9716cm ~ $ 

[madmartin]

I looked at commit dfeafb2 - looks good!

I think I need some debug output on the console to see more. I have to fetch the DTU from above and connect USB...

[berni2288]

You can also find a Web Console in the menu "Info".
MessageOutput.printf prints to Web console and Serial console :)

You can replace this code in src/HttpPowerMeter.cpp:

snprintf_P(error, errorSize, "Response too large!");

with:

            snprintf_P(error, errorSize, "Response too large! Response length: %d Body: %s",
                httpClient.getSize(), httpClient.getString().c_str());

Then you get the beginning of the response body in the UI directly.

[madmartin]

Hmm.... tried with the last commit and I receive

Response too large! Response length: -1 Body: {"StatusSNS":{"Time":"2023-04-02T21:45:01","Strom":{"Total_in":19904.8576,"L1":513.89,"Meter_Number":"0a01445a470002fb2769"}}}

WTF could be wrong here?
I compiled 2 times, last time with an extra platformio run -e generic -t clean to avoid any dependency fuzz.

[berni2288]

WTF could be wrong here? I compiled 2 times, last time with an extra platformio run -e generic -t clean to avoid any dependency fuzz.

Thanks @madmartin . That helped me to identify the problem and I've fixed it now.
The Content-Length header is not returned by Tasmota it seems, or is 0. That' why the getSize() method returned -1.
And comparing a signed integer against an unsigned integer is not a good idea it seems, as -1 is indeed greater than 1999 in this case (integer overflow).

Can you test again now? thanks!

[madmartin]

@berni2288 Success, it works! Good job!

One small fix to make the console output more readable: add CR+LF

diff --git a/src/PowerMeter.cpp b/src/PowerMeter.cpp
index 6bfe7e0..da2fc2d 100644
--- a/src/PowerMeter.cpp
+++ b/src/PowerMeter.cpp
@@ -68,7 +68,7 @@ void PowerMeterClass::onMqttMessage(const espMqttClientTypes::MessageProperties&
         _powerMeter3Power = std::stof(std::string(reinterpret_cast<const char*>(payload), (unsigned int)len));
     }
 
-    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\n", getPowerTotal());
+    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\r\n", getPowerTotal());
 
     _lastPowerMeterUpdate = millis();
 }
@@ -143,7 +143,7 @@ void PowerMeterClass::loop()
         }
     }
 
-    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\n", getPowerTotal());
+    MessageOutput.printf("PowerMeterClass: TotalPower: %5.2f\r\n", getPowerTotal());
 
     mqtt();
 

[berni2288]

Thanks for testing!

[STE2k]

I can´t find the power meter settings page as posted above.. what am I doing wrong? Or where can i find these settings? Do I have to activate Dynamics Power Limit first? I checked everything, nowhere to find the menu
I used OpenDTU-OnBattery-feature-http-json-powermeter. Do I have to erase my ESP32 completely because of it is using the old web UI?

[madmartin]

@STE2k You have to reload the WebUI in the browser! Otherwise the browser continues to use the cached WebUI and code.

Internet says:
To hard refresh on Google Chrome, there are two ways you can do it:

Hold down Ctrl and click the Reload button.
Or Hold down Ctrl and press F5.

To hard refresh on Firefox, there are also two easy hotkey commands you can use:

Hold down Ctrl, Shift and the ‘R’ key.
Or Hold down Ctrl and press F5.

[STE2k]

Working, OMG.. I´m the noob. Thx

[madmartin]

No problem.

[berni2288]

That just reminded me to delete old branches. Please use the development branch instead!

[kloppy1984]

I have the same problem, and can not find the new menu items

[berni2288]

Please refer to the README.md on how to build the webapp and the firmware. All changes are merged into development. I'll lock this PR now.