Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utility Meter: Invalid state #90914

Closed
mamoel666 opened this issue Apr 6, 2023 · 57 comments
Closed

Utility Meter: Invalid state #90914

mamoel666 opened this issue Apr 6, 2023 · 57 comments

Comments

@mamoel666
Copy link

The problem

Since I upgraded to 2023.4 I get theses warnings in the logs:

Logger: homeassistant.components.utility_meter.sensor
Source: components/utility_meter/sensor.py:438
Integration: Verbrauchszähler (documentation, issues)
First occurred: 10:24:03 (5 occurrences)
Last logged: 10:24:36

Invalid state (unknown > 86.093)
Invalid state (unknown > 52.027)
Invalid state (unavailable > 41865)
Invalid state (unavailable > 494.2)
Invalid state (unknown > 23.53)

What version of Home Assistant Core has the issue?

core-2023.4.0

What was the last working version of Home Assistant Core?

2023.3.6

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Utility Meter

Link to integration documentation on our website

https://www.home-assistant.io/integrations/utility_meter

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.utility_meter.sensor
Source: components/utility_meter/sensor.py:438
Integration: Verbrauchszähler (documentation, issues)
First occurred: 10:24:03 (5 occurrences)
Last logged: 10:24:36

Invalid state (unknown > 86.093)
Invalid state (unknown > 52.027)
Invalid state (unavailable > 41865)
Invalid state (unavailable > 494.2)
Invalid state (unknown > 23.53)

Additional information

No response

@home-assistant
Copy link

home-assistant bot commented Apr 6, 2023

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (utility_meter) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of utility_meter can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign utility_meter Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)


utility_meter documentation
utility_meter source
(message by IssueLinks)

@pergolafabio
Copy link

I'm seeing the same thing now since 2023.4

2023-04-06 10:01:15.237 WARNING (MainThread) [homeassistant.components.utility_meter.sensor] Invalid state (unknown > 0.021)

image

  - name: Water Usage l to m3
    unit_of_measurement: "m³"
    icon: mdi:water
    state_class: measurement
    state: >
      {{ states('sensor.water') | float(0)/1000 }}

water_usage_hourly:
    source: sensor.water_usage_l_to_m3
    name: Water Usage Hourly
    cycle: quarter-hourly
    delta_values: false
    unique_id: water_usage_hourly

@Joo01
Copy link

Joo01 commented Apr 6, 2023

Have same warnings in log since installing of 2023.4

Logger: homeassistant.components.utility_meter.sensor
Source: components/utility_meter/sensor.py:438
Integration: Verbrauchszähler (documentation, issues)
First occurred: 11:10:42 (35 occurrences)
Last logged: 11:21:28

Invalid state (unknown > 23.8007)
Invalid state (unknown > 32.2407)
Invalid state (unknown > 17.0871)
Invalid state (unknown > 6.6341)
Invalid state (unknown > 497.6042) 

@Mariusthvdb
Copy link
Contributor

Mariusthvdb commented Apr 6, 2023

this happens on each restart, but, if based on template sensors at least, also on each template reload, hence the 'unavailable'.

Logger: homeassistant.components.utility_meter.sensor
Source: components/utility_meter/sensor.py:438 
Integration: Nutsmeter (documentation, issues) 
First occurred: 10:15:29 (25 occurrences) 
Last logged: 10:17:43

Invalid state (unavailable > 24.6149)
Invalid state (unavailable > 368.3253)
Invalid state (unavailable > 349.50390000000004)
Invalid state (unavailable > 84.8711)
Invalid state (unavailable > 434.37500000000006)

@dgomes
Copy link
Contributor

dgomes commented Apr 6, 2023

This is not an error message... this is a warning.

It means that utility meter is unable to fully track consumption values from the source sensor because the source sensor is either unavailable or producing unknown values.

Utility Meter continues to work... it just skipped a some state changes from these invalid states to good states...

The warning is there, so that you know that the source sensor needs to be fixed...

@mamoel666
Copy link
Author

This is not an error message... this is a warning.

That's what I reported in the issue.

This warning wasn't there before the upgrade to 2023.4 and I think it's quite usual, that many sensors are not available at startup or am I wrong? I can live with this log message but there may be many users that are worried about this.

@DmytroKorniienko
Copy link

I have tons of warnings here: #90930 and hope this can be fixed.

@pergolafabio
Copy link

Hmm, that issue is not related to this one, your utility meter just doesn't have a value, you need to fix that... Or just disable it, since it doesn't work anyway:-)

The issue here is only 1 warning after restart HA, when integrations are not yet ready , when integration is loaded, the warning stops

@DmytroKorniienko
Copy link

Hmm, that issue is not related to this one, your utility meter just doesn't have a value, you need to fix that... Or just disable it, since it doesn't work anyway:-)

The issue here is only 1 warning after restart HA, when integrations are not yet ready , when integration is loaded, the warning stops

Why you think so? My sensors still working, but with a warnings, same situation for other users.

Screenshot_2023-04-07-02-45-11-901_com android chrome

@pergolafabio
Copy link

There must be one sensor with an unknown value, check that one.. also your log error is different, and we only see it one time , not spammed, so your utility meter is just not working, but best discuss this in your issue, it's not related to this one

@DmytroKorniienko
Copy link

DmytroKorniienko commented Apr 7, 2023

so your utility meter is just not working

You are wrong, but kk, discussion stopped.

@Xirt
Copy link

Xirt commented Apr 7, 2023

I believe we can rule out "non working meters" given the amount of users reporting the issue starting exactly after moving to 2023.4.0. It might be that at initialization there is no value yet which results in the error. I however do not recall seeing this warning earlier, while now - after upgrading to 2023.4.0 and also 2023.4.1 I have similar records in my log:

Logger: homeassistant.components.utility_meter.sensor
Source: components/utility_meter/sensor.py:438
Integration: Utility Meter (documentation, issues)
First occurred: 10:13:23 AM (4 occurrences)
Last logged: 10:13:23 AM

Invalid state (None > 13088.174)
Invalid state (None > 4380.855)
Invalid state (None > 3059.067)
Invalid state (None > 14409.962)

The underlying logic in my setup is that I create sensors first (based on my integrations / meters) after which the utility_meters are created for my dashboards:

# Power / Gas Consumption Statistics
template:
  - sensor:
    - name: 'Power Usage'
      state: "{{ (0 - states('sensor.power_produced')|float(0) + states('sensor.power_consumed')|float(0)) }}"
      unit_of_measurement: "W"
    - name: 'Total Energy Consumed'
      state: "{{ (states('sensor.energy_consumed_tariff_1')|float(0) + states('sensor.energy_consumed_tariff_2')|float(0)) }}"
      availability: "{{ states('sensor.energy_consumed_tariff_1')|float(0) != 0 and states('sensor.energy_consumed_tariff_2')|float(0) != 0 }}"
      unit_of_measurement: "kWh"
    - name: 'Total Energy Produced'
      state: "{{ states('sensor.solaredge_lifetime_energy')|multiply(0.001)|float(0) }}"
      availability: "{{ states('sensor.solaredge_lifetime_energy')|float(0) != 0 }}"
      unit_of_measurement: "kWh"
    - name: 'Total Energy Returned'
      state: "{{ (states('sensor.energy_produced_tariff_1')|float(0) + states('sensor.energy_produced_tariff_2')|float(0)) }}"
      availability: "{{ states('sensor.energy_produced_tariff_1')|float(0) != 0 and states('sensor.energy_produced_tariff_2')|float(0) != 0 }}"
      unit_of_measurement: "kWh"
    - name: 'Total Energy Usage'
      state: "{{ (states('sensor.total_energy_produced')|float(0) + states('sensor.total_energy_consumed')|float(0) - states('sensor.total_energy_returned')|float(0) ) }}"
      unit_of_measurement: "kWh"

utility_meter:
  gas_consumed_today:
    source: sensor.gas_consumed
    cycle: daily
  energy_consumed_today:
    source: sensor.total_energy_consumed
    cycle: daily
  energy_produced_today:
    source: sensor.total_energy_produced
    cycle: daily
  energy_returned_today:
    source: sensor.total_energy_returned
    cycle: daily
  energy_usage_today:
    source: sensor.total_energy_usage
    net_consumption: true
    cycle: daily

The same actually also causes an error on my side that I believe is new (one time at (re)start and everything works fine afterwards):

Logger: homeassistant.helpers.event
Source: components/utility_meter/sensor.py:427
First occurred: 10:13:23 AM (4 occurrences)
Last logged: 10:13:23 AM

Error while processing state change for sensor.total_energy_consumed
Error while processing state change for sensor.total_energy_produced
Error while processing state change for sensor.total_energy_returned
Error while processing state change for sensor.total_energy_usage
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/event.py", line 287, in _async_state_change_dispatcher
    hass.async_run_hass_job(job, event)
  File "/usr/src/homeassistant/homeassistant/core.py", line 593, in async_run_hass_job
    hassjob.target(*args)
  File "/usr/src/homeassistant/homeassistant/components/utility_meter/sensor.py", line 427, in async_reading
    _LOGGER.warning("Invalid state %s", new_state.state)
AttributeError: 'NoneType' object has no attribute 'state'

In my view, something "structurally" changed causing the warnings and errors. This might require a code fix or changes by people using the functionality: I leave it up to the experts to indicate what's is needed, but I hope this more elaborate scenario helps to narrow down on the RC :).

@realthk
Copy link
Contributor

realthk commented Apr 7, 2023

Same here, these warnings after each restart since 2023.4.0:

2023-04-07 10:14:13.753 WARNING (MainThread) [homeassistant.components.utility_meter.sensor] Invalid state (unknown > 1018.55)
2023-04-07 10:14:13.772 WARNING (MainThread) [homeassistant.components.utility_meter.sensor] Invalid state (unknown > 1018.55)
2023-04-07 10:14:13.782 WARNING (MainThread) [homeassistant.components.utility_meter.sensor] Invalid state (unknown > 1018.55)

In my opinion, a warning like this should contain the entity_id of the utility meter causing it! Otherwise how to correct any issues, if you don't know what entity has problem?
Luckily because of this unique value, I could find these are daily/monthly/yearly utility meters for my fridge energy consumption, all defined like this, only with different names and cycle values:

fridge_daily_energy:
  name: Fridge daily consumption
  source: sensor.fridge_energy
  cycle: daily
  unique_id: utility_fridge_daily_energy

sensor.fridge_energy here is provided by a Zigbee plug through Z2M, so I've changed this device to send retained MQTT messages, and it seems to cleared the issue, so I guess this happens when MQTT is already started but Z2M not yet, so no value for sensors like this.

@dgomes
Copy link
Contributor

dgomes commented Apr 7, 2023

The WARNING (not error!) message was indeed introduced in release 2023.4

The objective was for users to understand that their utility meter is skipping readings due to mis behaving source sensors.

The best approach is to fix the SOURCE sensor, from various issues z2m looks to be a recurring culprit ...

@dgomes
Copy link
Contributor

dgomes commented Apr 7, 2023

CC @Wesley-Vos

@realthk
Copy link
Contributor

realthk commented Apr 7, 2023

Yes, but

  • a warning without specifying which utility_meter has problem isn't really helpful, rather annoying
  • during startup this isn't a problem at all, just later starting of a component
    I think most of us likes to have clean logs without even warnings if possible.

@Xirt
Copy link

Xirt commented Apr 7, 2023

Thanks for pushing the improvement in the logging @dgomes .

I do however also agree with @realthk on the second comment: in my case the source (not being Z2M) is only "misbehaving" at the (re)start / (re)load of Home Assistant and unfortunately at every (re)start / (re)load. After all has been initialized there is no issue anymore (and if there would be an issue, then I would like to see it in the log). With the current setup, warnings could potentially be seen as "spam to skip" in the logs.

@dgomes
Copy link
Contributor

dgomes commented Apr 7, 2023

updated the PR to only activate warnings after HA is marked as started...

@Mariusthvdb
Copy link
Contributor

Mariusthvdb commented Apr 7, 2023

What about the template reload ? Will we be safe there too now?

@dgomes
Copy link
Contributor

dgomes commented Apr 7, 2023

This is not a fix... this is a ignore until HA is marked as started

@dgomes
Copy link
Contributor

dgomes commented Apr 7, 2023

I've added yet another PR that might address the template issue

#91035

@Mariusthvdb
Copy link
Contributor

Mariusthvdb commented Apr 7, 2023

I've added yet another PR that might address the template issue

#91035

thx Diogo, hope it will see to it. appreciated!

btw, since trigger based template sensors retain their state during restarts, would it be an idea to rewrite the used (regular) template sensors to trigger based template sensors? In the hope these states wont go unavailable during these essential moments.

@dgomes
Copy link
Contributor

dgomes commented Apr 7, 2023

@Mariusthvdb that would be a different issue

@drthanwho
Copy link
Contributor

I'm seeing the warning about unavailable entities for several of my entities coming from an esphome device which I'm guessing is due to the integration not having the entities available on time. But I see the PR added to wait for HA to have started so I'm not sure why I still get those warnings.

@Mariusthvdb
Copy link
Contributor

Mariusthvdb commented Apr 9, 2023

updated to 2023.4.2 now and the PR changed the Warning message to include the entities in play, which is really good, thanks!:

Logger: homeassistant.components.utility_meter.sensor
Source: components/utility_meter/sensor.py:447 
Integration: Nutsmeter (documentation, issues) 
First occurred: 15:17:39 (13 occurrences) 
Last logged: 15:17:39

Alle lampen binnen yearly received an invalid state change coming from sensor.lights_inside_total_device_energy (unavailable > 350.89)
Alle lampen buiten daily received an invalid state change coming from sensor.lights_outside_total_device_energy (unavailable > 85.21)
Alle lampen buiten yearly received an invalid state change coming from sensor.lights_outside_total_device_energy (unavailable > 85.21)
Alle lampen daily received an invalid state change coming from sensor.lights_total_device_energy (unavailable > 436.1)
Alle lampen yearly received an invalid state change coming from sensor.lights_total_device_energy (unavailable > 436.1)

ofc, I would still hope these could be prevented too, as the appear on template reload....

@dgomes
Copy link
Contributor

dgomes commented May 13, 2023

Please wait for 2023.5.3, then report back

@pimw1
Copy link

pimw1 commented May 13, 2023

Aah ok, nice. I was not aware that a (potential) fix is coming. Thanks for mentioning.

@pimw1
Copy link

pimw1 commented May 14, 2023

I've installed 2023.5.3. The same error appears when restarting Home Assistant.

@dgomes
Copy link
Contributor

dgomes commented May 14, 2023

For esphome devices in which you might trust their reliability you should use periodically_resetting: false as per #88446

@pimw1
Copy link

pimw1 commented May 14, 2023

I've set periodically_resetting:true because the esphome device sensor will go back to 0 when i reboot the esphome device.

That being said, i am unable to understand how this is impacting the issue at hand. The issue that i am facing, is that a reboot of home assistant generates the warning "an invalid state change coming from sensor.gang_watermeter_totaalverbruik."

The esphome device itself is running nicely in the background. In my case, it is a water meter which is only giving a pulse once every half hour or so (whenever i use 1 liter of water).

In practice, this means that the sensor value of the esphome device stays unchanged while home assistent is restarted. Yet, the warning appears in the logs of home assistant.

@pimw1
Copy link

pimw1 commented May 14, 2023

image
To give you an impression :)

@dgomes
Copy link
Contributor

dgomes commented May 14, 2023

The problem is that HA does not know if your water meter made a change while it was unavailable, therefore in the next reading coming from the meter the utility_meter will either skip that reading (the default solution, which leads to missing values) or it will take into consideration the previously recorded state (before the meter went unavailable - that's the periodically_resetting: false)

Previous to the introduction of this option, this issue was not logged, basically because there was no alternative... (silent error)

sure we can remove the warning message, but you will miss some important (?) readings and wonder what's wrong (nothing is wrong, the utility_meter just doesn't have enough information to deal with a sensor that came back from unavailable)

@pimw1
Copy link

pimw1 commented May 14, 2023

The sensor itself is fully working and rock-solid. The issue the booting of Home Assistant itself. By definition, when home assistant is starting, sensors will often be unavailable until home assistant is fully started. This leads me to a different conclusion: the utility meter is not correctly handling the startup of Home Assistant.

@dgomes
Copy link
Contributor

dgomes commented May 14, 2023

Some more clarification, periodically_resetting is not about resets that happen programmatically (like nighly reset)

It is to be used by devices that reset during unavailability (those unreliable sensors that loose power/connection often)

The default behaviour for several years was to consider sensors as unreliable and just ignore their first state transition from unavailable to available, effectively only tracking changes while HA is working.

@pimw1
Copy link

pimw1 commented May 14, 2023

It is to be used by devices that reset during unavailability (those unreliable sensors that loose power/connection often)

I'm aware of that :) My sensor is designed to jump back to 0 when i shut it down from power. That happens sometimes (like once every month or so, whenever i have an electrical job to do at home). That is expected behaviour. And utility meter is handling that scenario perfectly.

The issue is imho the warning message itself, which is not recognizing that home assistant is still in startup phase.

@pimw1
Copy link

pimw1 commented May 14, 2023

The best scenario would be that:

  • utilty meter is showing a warning message when a sensor is really unavailable, to make the end-user aware
  • utility meter is handling the startup of home asisstant in a good way, so it won't send a warning when everything is working just fine (like in my case).

@gfrancesco
Copy link

I think some of the issues that I discussed in #72943 (my last comment for example) can be related. The periodically_resetting PR addressed some of them I think, but I'm not sure.

In addition I noted that in 2023.5.3 I cannot read anymore any utility meter if the source sensors are unavailable, because they also turn unavailable. This is rather problematic and never occurred before. A common use case where this is evident is having a utility meter attached to a solar inverter: when the inverter shuts off at night you cannot read the meter measuring daily, weekly or yearly energy.

@dgomes
Copy link
Contributor

dgomes commented May 19, 2023

@gfrancesco good point, I've reverted the unavailability of utility_meter in #93274

@pimw1
Copy link

pimw1 commented May 19, 2023

Would that also fix the warning "an invalid state change coming from sensor.xxxxx" when rebooting home assistant?

@dgomes
Copy link
Contributor

dgomes commented May 20, 2023

Sorry guys, but #93274 got closed

You will need to find a fix in which your inverter component doesn't go unavailable 🤷‍♂️

@pimw1
Copy link

pimw1 commented May 20, 2023

I think there are 2 puzzles at hand:
-what to do with sensors that go unavailable, like the solar convert. I've no idea how to handle that.
-what to with home assistant reboots. By definition, sensors will be unavailable until home assistant is fully booted. It makes no sense at all to show a warning during reboots.

@gfrancesco
Copy link

I appreciated the effort @dgomes, might be that the goal of the PR was misunderstood?
Just to recap, let's see if I got it correctly:

  • There was an issue with unwanted warnings in log;
  • The PR to fix that broke how the utility meter worked since years (being readable even when the source sensor is unavailable);
  • The PR to restore the original behaviour was rejected.

In my opinion, it doesn't make much sense, especially considering that this is not an edge case at all. Source sensors go unavailable all the time, that's the norm not an exception, and it was correctly handled for years! This is further aggravated if your source sensor is battery driven and just measures intermittently, imagine water consumption for a sprinkler for example. To save battery the source sensor would be unavailable most of the time and the water consumption would be readable only when the sprinklers are on, which would not be very useful.

Even if you think how physical utility meters works, it doesn't make any sense. Imagine if you cannot read how much water you used this year just because the little plastic spinner of the sensor broke.

@pimw1
Copy link

pimw1 commented May 20, 2023

I appreciated the effort @dgomes

Me too, thank you dgomes for all the help and effort you put into this.

@dgomes
Copy link
Contributor

dgomes commented May 20, 2023

There has been a long issue with what happens to helpers when their sources are unavailable, and it made sense to me to make the utility_meter unavailable when it's source is unavailable.

From this issue discussion I realised that my previous judgement was not correct and therefore created a new PR to revert not the handling of unavailable sources but the unavailability of the utility_meter itself - this last PR was closed.

@gfrancesco
Copy link

Do you have a link to the issue you are referring?
I agree with your point of view, I just don't understand how it's supposed to be fine to willingly break user facing behaviors that are in place since years for the sake of having some common rule on how to handle helpers.
Helpers components are very different, e.g. text boxes, buttons, utility meters, etc. Maybe trying to force some common behaviors to all of them is not a good idea?
From a user perspective, it's hard to imagine why an utility meter should be unavailable. It's a counter, the normal behavior would be to either increase it's value or stay fixed, but it should always show the count. Imagine if you can read a car mileage only if it's wheels are moving, not very handy.
Anyway, sorry for the rant, I know it's not your fault and you tried your best, it's just that it will take quite some work to fix this problem on the user side, I'm not even sure how to do it at the moment.

@dgomes
Copy link
Contributor

dgomes commented May 21, 2023

It's not a single issue, you can search this repository issues.

@issue-triage-workflows
Copy link

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

@martinszelcel
Copy link

Hi, the problem still occurs :(
I have a PV inverter that goes offline after sunset. Because of it its utility meter also goes unavailable. I'm using this utility meter in template sensors to calculate home energy usage, so all template sensors based on it are also unavailable.

I'm searching for a workaround for this issue, because currently I can't see all stats on the dashboard after sunset. I've tried to create a separate template sensor that is set to 0 instead of unavailable but this causes wrong calculations on the utility meter. The values get doubled/tripled during the day (probably when the inverter loses connection).

Can we get any info if this problem will be resolved or can we use some other workaround to fix this issue?

@github-actions github-actions bot removed the stale label Aug 22, 2023
@gfrancesco
Copy link

gfrancesco commented Aug 23, 2023

It has been made clear that, according to the dev team, this is not a bug. It is a behavior that has been introduced deliberately. See my posts above, anyway I won't hold my breath for this to be changed since the PR was closed.

At the moment I'm using a partial workaround using trigger template sensors. Essentially the source sensor used by the utility meter becomes the trigger sensor. The trigger sensor references the energy meter of your inverter, and it only updates when there is a new valid read coming from your inverter.

For example if your inverter goes down and its energy meter reads unavailable, the trigger sensor is not updated, does not go unavailable, and keeps reporting the latest number, thus the utility meter also keep showing a good read.

Here's an example of one of my trigger sensors:

- trigger:
    - platform: state
      entity_id: sensor.inv_1_e
      not_to:
        - "unavailable"
        - "unknown"
        - "none"
  sensor:
    - name:         "Inv-1 E NU"
      state:        "{{ states('sensor.inv_1_e') }}"
      state_class:  "total_increasing"

@issue-triage-workflows
Copy link

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

@issue-triage-workflows issue-triage-workflows bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 28, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Dec 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests