Graphs starting this past week look “odd”

[knuckleheadsmiff]

Something changed so that data from my smart meter always shows with the area under/above the graph as “solid”. Otherwise the numbers look the same. Not sure what the issue is, I’ve rebooted home assistant and it may have fixed it for an hour or so yesterday but then it started back again. Every thing looks fine in the emporia vue app.

[kzaoaai]

It's because the 1-min readings are intermittently reporting zero value. The meter reports a correct value, and then 0, then value again. This behavior started in the last few days.

[knuckleheadsmiff]

Yes I did notice it only in the last few days. At least I'm not the only one seeing it.

[kzaoaai]

Until this is fixed, I created an input_number that omits the zero values:

`
alias: Helper - House Electric Meter 1MIN Adjustment
description: omit zero values from the meter
trigger:

• platform: state
  entity_id:
  • sensor.house_electric_meter_123_1min
    condition:
• condition: or
  conditions:
  • condition: numeric_state
    entity_id: sensor.house_electric_meter_123_1min
    above: "0"
  • condition: numeric_state
    entity_id: sensor.house_electric_meter_123_1min
    below: "0"
    action:
• service: input_number.set_value
  data_template:
  value: "{{states('sensor.house_electric_meter_123_1min')}}"
  entity_id: input_number.house_electric_meter_1min_temporary
  mode: single

`

[knuckleheadsmiff]

That’s actually a good fix long term too, in the past I’d get a couple of zero reading per day in the graphs. The odds of a real zero reading are pretty low.

[akballow]

I have been noticing the same too for past few days. @magico13 please if you have the time to add the functionality to ignore 0's like the above workaround. I am sure its some API server issue on emporia's side, but having less 0 spikes is nice overall for all users.

[knuckleheadsmiff]

I guess you actually need to check if the server is actually reporting 0, or it’s an error case when the server is not correctly responding and the code is incorrectly recording 0 rather than ignoring the result? I haven’t looked at the code to know.

For some folks not recording 0 would be wrong if they are running off a battery or generator. (For me that’s not the case as I don’t have a battery but I do go negative during the day because of solar.)

[magico13]

Unfortunately ignoring zero everywhere isn't a viable option. For instance, a monitor on a dryer circuit is going to be zero quite often. If it's getting an error then I might be able to do something, but if the API reports zero with a 200 response I'm not sure that's something I can distinguish in a general way.

[kzaoaai]

Not sure if the API values follow the values reported in the emporia app, but the app does not report zeroes and its graph is correct.

Also, at least for the house meter, zero values can probably be ignored as there's very little chance a meter is going to be at zero at any point of the day, even with PV.

[knuckleheadsmiff]

Just curious, are you avoiding that the data in the 200 response is correctly formatted as you expect? It’s hard for me to follow the code so sorry (I’ve tried.)

[magico13]

So there's a fairly common issue in the API/app where the live data on the screen where you see all of the circuits will report zero or even offline and that's the endpoint that this code pulls from (one call updates all of the devices at once). But the graphs in the app use a different endpoint that runs for a single circuit and pulls historical data, which will not report incorrectly nearly as often. Although that one's also fun because it sometimes does report zero for live data and then will have a non-zero value for that time when you look at it again. I wish they'd just give us local access or push data rather than polling but they also don't really endorse this project so we're pretty low priority.

[c3p0vsr2d2]

I am also seeing the weird 0 values too - and it seems to switch from no-0-values to many-0-values every few hours for me.
I tried differentiating the 1-day values, but precision is lost, so values end up being inaccurate.
I only have one device - could the endpoint for the integration be moved to the one used by the app?

PS: Regardless, thank you for this integration.

[akballow]

Started to be very bad for me. Almost all zero values. Sounds like they are phasing out the api. Hell i would pay a monthly sub if it goes back to the old api with 1sec results.

[knuckleheadsmiff]

The input number automation suggested in the comments above work well for me to mask out the zeros for my power meter. It's the only device I track with this.

[alvinchen1]

I've oddly had a lot of experience with this issue, but not with Emporia Vue, but instead with Z-Wave kWh and W reporting. I'd suggest using one of two of the following:

1. Filter...https://www.home-assistant.io/integrations/filter/ The filter object (which you can create in sensor.yaml), can automatically weed out some of these errant 0s while still allowing you to get to 0 when appropriate.

You could, for example do this:

  - platform: filter
    name: "House Electric Meter 123 1 min"
    entity_id: sensor.house_electric_meter_123_1min
    filters:
      - filter: lowpass
        time_constant: 10
        precision: 2

2. Template Sensor. Another sensor in template.yaml, this one you have a bit more control over similar to what the automation and the input_number is. Here's an example of this:

      - name: "House Electric Meter 123 1 min templated"
        state: >
          {% set meter='sensor.house_electric_meter_123_1min' %}
          {% set metersmoothed='sensor.house_electric_meter_123_1_min_templated' %}
          {% set expectedincrease = 1 %}
          {% set metervalue=states(meter) | float(0) %}
          {% set metersmoothedvalue=states(metersmoothed) | float(0) %}
          {% if (states(meter) == "unknown") or (states(meter) == "unavailable") %}
             {{ metersmoothedvalue }}
          {%- elif ((metervalue >= metersmoothedvalue) and (metervalue - metersmoothedvalue) <= expectedincrease) or (states(metersmoothed) == "unknown") or (states(metersmoothed) == "unavailable") or (metervalue == 0) or (metersmoothedvalue == 0) -%}
             {{ metervalue }}
          {%- else -%}
             {{ metersmoothedvalue }}
          {% endif %}

In this scenario, you have a bit more granular control over what values you accept. I am first off ignoring unknown and unavailable values. Then I am only accepting when the value is increasing. Because I am not accepting negative values you would need to adjust the template for that, but using the expectedincrease, you can tune it to your preference, and basically eliminate those errant 0s. But this might help to make your numbers less volatile.

[c3p0vsr2d2]

Many thanks @alvinchen1, @kzaoaai for the pointers.

I am not sure where the zero values are coming from, and I see 3 different ways of handling the zero value.
Suppose the values sampled by the integration are, for example, {..., 500, 520, 0, 900, 700, 200, ...}

1. Option 1: Copy previous value: 520
2. Option 2: Interpolate missing value: (520 + 900)/2
3. Option 3: Energy is conserved: Replace zero value and subsequent sample by (900/2)

Which one should it be ? Would appreciate anyone shedding more light on this. TIA.

[akballow]

RIP emporia. Has anyone used an alternative that connects back to home assistant?

the lack of accuracy now with all the zeros makes a muck in all my costs calculations now too. Filters and such recommended above do not solve it completely.

[JP13090]

I've slowly been bringing in more and more Sengled Smart Plugs. They're zigbee, and I'm growing that mesh these days too, so it works for me. But they're LOCAL, and they still provide power usage information second by second even. Eventually, I'll sell my Emporia stuff to someone who just has a "connected" home instead of a smart home.

…

On Wed, Aug 24, 2022 at 3:48 AM akballow ***@***.***> wrote: RIP emporia. Has anyone used an alternative that connects back to home assistant? the lack of accuracy now with all the zeros makes a muck in all my costs calculations now too. Filters and such recommended above do not solve it completely. — Reply to this email directly, view it on GitHub <#137 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APSA3PIWGIP3FUC7DDB5W7TV2XHWBANCNFSM56QIZUDQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[baylanger]

Hey all. Nobody knows for sure if Emporia took the right decision here or not.

Feel free to check this project based on ESPHome - I haven't done it yet but that is the plan. The bonus is that the implementation runs locally - you control your data provided to HA via mqtt.

Emporia is not the first one to make such a more and others will follow. A key factor that weight on the decision likely has to do with the % of traffic generated by leaving the api "friendly". For example, Tesla's rest-api is used by many projects "heavily" using their api to fetch your car's data (location, speed, temperature, .... big "car" data). To my surprise Tesla is leaving it open but lets admit there's a difference how Tesla's official clients (cars, powerwalls, etc) uses its rest-api compared to Emporia. Tesla's cars are hitting the api constantly, the rps is already very high ... and probably not the case for Emporia.

[c3p0vsr2d2]

@akballow @baylanger I am out of the loop - is Emporia phasing out any API/etc? Or is it the zero entries that are pushing you to switch to something else?

[akballow]

@akballow @baylanger I am out of the loop - is Emporia phasing out any API/etc? Or is it the zero entries that are pushing you to switch to something else?

For the past ~ 2 weeks i would say the API is returning 50% of the time zeros. Even if you remove the zeros, any automation and cost features are useless since the missing values can not be accounted for with accuracy, Just guesses.

Emporia should just do a subscription if they can not afford the API calls. I would happily pay. I am currently looking for another product which hooks up to PG&e. There are other products which take value by reading the real value. Might look into those too.

[oziee]

might be time for me to finally try out https://gist.github.com/flaviut/93a1212c7b165c7674693a45ad52c512

[akballow]

Is this thread quite because the issue was resolved or just because everyone gave up

[knuckleheadsmiff]

Is this thread quite because the issue was resolved or just because everyone gave up

Still an issue

[magico13]

There's a template sensor provided above that can help if you've got a sensor that you know can't be zero. I haven't had time lately to implement any fixes in the integration, I'm honestly not sure if there's a generic way to "fix" it if the API is returning zeroes and not an error message. I'm open to PRs off the pre-release branch if anyone wants to give it a shot.

[apetryk2]

Is anyone else getting these messages in their logs when they are getting zero values? Might be a clue for how to fix:

2022-08-31 04:14:05.262 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 151, in _handle_refresh_interval
await self._async_refresh(log_failures=True, scheduled=True)
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
self.async_update_listeners()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 110, in async_update_listeners
update_callback()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 348, in _handle_coordinator_update
self.async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 532, in async_write_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 570, in _async_write_ha_state
state = self._stringify_state(available)
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 538, in _stringify_state
if (state := self.state) is None:
File "/config/custom_components/emporia_vue/sensor.py", line 90, in state
usage = self.coordinator.data[self._id]["usage"]
KeyError: '20803-6-1MIN'

[chrwei]

what's odd on mine is that's only on the "panel" sensors, and isn't every day, or all day. in the last week it's only done it 8/31 between 8:30am and about 1pm central time.

does the api allow re-requesting a particular minute? maybe some logic like if zero comes back, and the previous is more than 10W more, request it again...

[eagleco]

This started for me on 9/3, went away briefly until 9/6 and has been happening since. I have the "Vue Utility Connect" device (Zigbee/HAN whole house meter monitor). Went the input number filter route for now. Here's my automation yaml (what @kzaoaai posted but with code tags so you can copy and paste (I dropped the <0 condition since I don't have solar (yet))

alias: Emporia Energy 1m remove zeroes
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.energy_1min
condition:
  - condition: numeric_state
    entity_id: sensor.energy_1min
    above: 0
action:
  - service: input_number.set_value
    data:
      value: "{{states('sensor.energy_1min')}}"
    target:
      entity_id: input_number.energy_1min
mode: single

[Zyell]

might be time for me to finally try out https://gist.github.com/flaviut/93a1212c7b165c7674693a45ad52c512

Every single energy sensor of mine was having random blips in the data. It wasn't resetting to zero, just decreasing and increasing again. My Energy Dashboard was a mess. I tried to fix this in various algorithmic ways, but nothing was generic enough. I finally turned to this project and it works extremely well, even the midnight reset behavior is perfect now. I really liked this integration and have used it for almost a year. While the Emporia API endpoints used here seem to be broken at the source, the hardware works brilliantly with esphome and is entirely local.