Use with Shelly 3EM MQTT?

[teunito]

Hi!
Fantastic esphome component and great work!

I was wondering if it is possible to get it working with a Shelly 3EM with MQTT enabled. The Shelly does not provide a topic with total power consumption. It has a topic for instantaneous active power per channel (https://shelly-api-docs.shelly.cloud/gen1/#shelly-3em-mqtt). Can we do some sort of calculation and sum up the three channels (topics) in the YAML?
Or can i get It from the json of the HTTP status page? This looks like this:

{
   "actions_stats" : {
      "skipped" : 0
   },
   "cfg_changed_cnt" : 0,
   "cloud" : {
      "connected" : false,
      "enabled" : false
   },
   "ct_calst" : 0,
   "emeter_n" : {
      "current" : 0,
      "is_valid" : false,
      "ixsum" : 12.68,
      "mismatch" : false
   },
   "emeters" : [
      {
         "current" : 2.06,
         "is_valid" : true,
         "pf" : -0.95,
         "power" : -458.43,
         "total" : 226923.9,
         "total_returned" : 517029.3,
         "voltage" : 234.79
      },
      {
         "current" : 1.87,
         "is_valid" : true,
         "pf" : 0.81,
         "power" : 357.2,
         "total" : 269829.5,
         "total_returned" : 0,
         "voltage" : 235.29
      },
      {
         "current" : 11.34,
         "is_valid" : true,
         "pf" : 1,
         "power" : 2629.79,
         "total" : 1348406.2,
         "total_returned" : 0,
         "voltage" : 232.09
      }
   ],
   "fs_free" : 153612,
   "fs_mounted" : true,
   "fs_size" : 233681,
   "has_update" : false,
   "mac" : "E8DB84D68FFB",
   "mqtt" : {
      "connected" : true
   },
   "ram_free" : 25556,
   "ram_total" : 49928,
   "relays" : [
      {
         "has_timer" : false,
         "is_valid" : true,
         "ison" : true,
         "overpower" : false,
         "source" : "input",
         "timer_duration" : 0,
         "timer_remaining" : 0,
         "timer_started" : 0
      }
   ],
   "serial" : 92,
   "time" : "12:42",
   "total_power" : 2528.56,
   "unixtime" : 1663843375,
   "update" : {
      "has_update" : false,
      "new_version" : "20220830-080542/v1.12-3EM-gcf4f7c2",
      "old_version" : "20220830-080542/v1.12-3EM-gcf4f7c2",
      "status" : "idle"
   },
   "uptime" : 3009,
   "v_data" : 1,
   "wifi_sta" : {
      "connected" : true,
      "ip" : "192.168.1.X",
      "rssi" : -59,
      "ssid" : "AAAAA"
   }
}

Best regards!

[syssi]

Yes. You could subscribe all instantaneous power topics and use a template sensor to provide the sum:

sensor:
  - id: l1
    internal: true
    platform: mqtt_subscribe
    name: "${name} instantaneous power consumption l1"
    topic: "shellies/shellyem3-DEVID/emeter/1/power"
    accuracy_decimals: 2
    unit_of_measurement: W
    device_class: power
  - id: l2
    internal: true
    platform: mqtt_subscribe
    name: "${name} instantaneous power consumption l1"
    topic: "shellies/shellyem3-DEVID/emeter/2/power"
    accuracy_decimals: 2
    unit_of_measurement: W
    device_class: power
  - id: l2
    internal: true
    platform: mqtt_subscribe
    name: "${name} instantaneous power consumption l1"
    topic: "shellies/shellyem3-DEVID/emeter/2/power"
    accuracy_decimals: 2
    unit_of_measurement: W
    device_class: power
  - platform: template
    id: powermeter
    name: "${name} instantaneous power consumption sum"
    lambda: |-
      if (id(l1).state && id(l2).state && id(l3).state) {
        return id(l1).state + id(l2).state + id(l3).state;
      } else {
        return 0.0;
      }
    update_interval: 15s

[syssi]

Please feel to ask additional questions if the idea doesn't solve your issue. Otherwise please close the issue!

[teunito]

Okay, that looks good! I will give it a try!
On the other hand i'm looking for a solution where I have no problem if my homeassistant/raspberry is offline. I was thinking about talking directly with the shelly 3EM. Can I get it from the HTTP call and extract it?

[syssi]

Good point. Extracting the value from a json response should also be possible. Please take a look at the http request component: https://esphome.io/components/http_request.html

If you are unable to create a proper solution by combining a template sensor and the http request please ping me again.

Something like this should do the job: https://community.home-assistant.io/t/esphome-how-to-read-json-from-web/304034/20?u=syssi

[teunito]

Got it working with your links! Thanks a lot!
Here is how I did it, maybe it's useful for an example in the repository:

    id: powermeter
    name: "${name} instantaneous power consumption sum"
    lambda: |-
        if (id(powermeter).state) {
            return id(powermeter).state;
        } else {
            return 0.0;
        }
    unit_of_measurement: W
    device_class: "power"
    accuracy_decimals: 2
    update_interval: 2s

time:
  - platform: sntp
    id: sntp_time
    on_time:
      # Every 1 second
      - seconds: /1
        then:
          - http_request.get:
              url: http://192.168.178.111/status/
              headers:
                Content-Type: application/json
              verify_ssl: false
              on_response:
                then:
                  - lambda: |-
                      json::parse_json(id(http_request_data).get_string(), [](JsonObject root) {
                        id(powermeter).publish_state(root["total_power"]);
                      });

[pfeupfeu]

Thank you for this possibility to read out the Shelly3EM directly!
A small addition for shelly-3em-http-status-json.yaml: The 2nd line (id: http_request_data) is needed, to prevent a error message during compiling:

http_request:
    id: http_request_data
    useragent: esphome/device
    timeout: 5s

[pfeupfeu]

I have been using the Shelly3EM query for some time now and occasionally get this error:

[08:44:48][I][soyosource_display:442]:   Start delay: 3 s
[08:44:49][E][json:080]: Could not allocate memory for JSON document! Requested 0 bytes, free heap: 18016
[08:44:49][W][http_request:086]: HTTP Request failed; URL: http://192.168.188.71/status/; Error: connection failed; Duration: 8 ms
[08:44:55][I]....

I read it would be better using http_request_data with a global variable or a text_sensor to prevent this error.
Additionaly I wanted to have the possibility to fine-tune the update interval. That's why I rewrote the Shelly query:

globals:
   - id: getShelly
     type: std::string
     initial_value: '""'

interval:
    - interval: 1100ms
      then:
        - http_request.get:
            url: ${shelly_3em_url}
            headers:
              Content-Type: application/json
            verify_ssl: false
            on_response:
              then:
                - lambda: |-
                      id(getShelly) = id(http_request_data).get_string();
                      json::parse_json(id(getShelly), [](JsonObject root) {
                      id(powermeter).publish_state(root["total_power"]);
                      });
                - component.update: virtualmeter0

Everything else has remained the same.
I hope the allocate memory error doesn't happen again.

[syssi]

I don't see the benefit of storing the response in a global variable. Do I miss something?

[pfeupfeu]

I cant' t explain it either. But I read some reports that people had success with this measure. E.g. here:
esphome/issues#4250
I'll tell you if it makes a difference.

[pfeupfeu]

Ok, it isn't an improvement. After 2h I got this:

[pfeupfeu]

My yaml includes the Wifi signal sensor. The frequent errors seem to occur when the wifi signal is weak. That does not explains to me the allocate memory error.
The ESP is no longer responsive after some errors. It might not be a good idea to have the timeout time for http_request greater than the call frequency. Therefore I set timeout to 800ms now - hopefully it helps.

[syssi]

This is my guess: The error looks critical but it isn't. Every now and then the Shelly doesn't provide a valid response. Let's assume the connection gets dropped and there is no response at all. In this case this implementation

json::parse_json(id(http_request_data).get_string(), [](JsonObject root) {
  id(powermeter).publish_state(root["total_power"]);
});

tries to parse_json() an empty string or something else with a size of 0 bytes. This will lead to the error above. You could try to catch this case using an if condition.

[syssi]

Please try:

std::string response_data = id(http_request_data).get_string();
if(!response_data.empty()) {
  json::parse_json(response_data, [](JsonObject root) {
    id(powermeter).publish_state(root["total_power"]);
  });
}

[syssi]

The ESP is no longer responsive after some errors. It might not be a good idea to have the timeout time for http_request greater than the call frequency. Therefore I set timeout to 800ms now - hopefully it helps.

Good catch! If the requests pile up there could be a serious memory issue for sure.

[pfeupfeu]

Please try:

std::string response_data = id(http_request_data).get_string();
if(!response_data.empty()) {
  json::parse_json(response_data, [](JsonObject root) {
    id(powermeter).publish_state(root["total_power"]);
  });
}

Unfortunately no difference, the problem seems to be within id(http_request_data).get_string():

08:32:47 | [E] | [json:080] | Could not allocate memory for JSON document! Requested 0 bytes, free heap: 7008
08:32:48 | [W] | [http_request:086] | HTTP Request failed; URL: http://192.168.188.71/status/; Error: connection failed; Duration: 824 ms
08:32:49 | [I] | [soyosource_display:232] | Status frame (Soyo, 15 bytes) received

[syssi]

May be the error is raised a the http_request component already. Could you add some debug instructions to the snippet:

std::string response_data = id(http_request_data).get_string();
ESP_LOGD("Lambda", "Response data: %s", response_data.c_str());
if(!response_data.empty()) {
  ESP_LOGD("Lambda", "Parsing JSON");
  json::parse_json(response_data, [](JsonObject root) {
    id(powermeter).publish_state(root["total_power"]);
  });
}

[pfeupfeu]

No Change in the error message. Should I use a different logger: level: , not only INFO ?

09:00:42	[E]	[json:080] Could not allocate memory for JSON document! Requested 0 bytes, free heap: 17616
09:00:42	[W]	[http_request:086]	HTTP Request failed; URL: http://192.168.188.71/status/; Error: connection failed; Duration: 826 ms

[syssi]

Oh, you don't log DEBUG messages. Let's raise the log messages here to INFO to avoid other noise:

std::string response_data = id(http_request_data).get_string();
ESP_LOGI("Lambda", "Response data: %s", response_data.c_str());
if(!response_data.empty()) {
  ESP_LOGI("Lambda", "Parsing JSON");
  json::parse_json(response_data, [](JsonObject root) {
    id(powermeter).publish_state(root["total_power"]);
  });
}

[pfeupfeu]

Oh sorry, I did a mistake, I compiled the wrong version.
#76 (comment)
gives no allocation errors, only warnings when the connection is lost:

[09:32:22][I][soyosource_display:442]:   Start delay: 3 s
[09:32:26][W][http_request:086]: HTTP Request failed; URL: http://192.168.188.71/status/; Error: connection failed; Duration: 521 ms
[09:32:26][W][http_request:086]: HTTP Request failed; URL: http://192.168.188.71/status/; Error: connection failed; Duration: 516 ms
[09:32:28][I][soyosource_display:232]: Status frame (Soyo, 15 bytes) received

Combined with the shorter http_request timeout it should be stable. Thank you for your help!!

And to complete the picture, this is my current working version:

http_request:
  id: http_request_data
  useragent: esphome/device
  timeout: 500ms
  
interval:
  - interval: 1100ms
    then:
      - http_request.get: 
         url: ${shelly_3em_url}
         headers:
            Content-Type: application/json
         verify_ssl: false
         on_response:
            then:
              - lambda: |-
                  std::string response_data = id(http_request_data).get_string();
                  if(!response_data.empty()) {
                  json::parse_json(response_data, [](JsonObject root) {
                  id(powermeter).publish_state(root["total_power"]);
                  });
                  }
              - component.update: virtualmeter0