history_stats sensor does not reset until slightly after midnight for the next day

[bengtner]

The problem

History_stats gives a wrong value if the sensor used as the base for the measurement is "on" over midnight and you want to measure a daily value.This problem has been reported before (#72357) but this issue was closed without action.

The following sensor is used to keep track of the daily usage of a water tap:

- platform: history_stats
    name: Waterfill daily
    entity_id: switch.poolwater
    state: "on"
    type: time
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"

This speciIfic night switch.poolwater was open less than 5 minutes:

The history stat sensor should present 3m14s for July 27th and 22s for July 28th. But it continued to count all day ( today) even if the switch turned of 22s after midnight. In the morning it looked like this:

What version of Home Assistant Core has the issue?

2022.7.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

history_stats

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

Not sure if this problem occurs everytime the sensor is open over midnight. I've seen this problem before monitoring another sensor, but I thought it was corrected in a later release. See https://community.home-assistant.io/t/strange-history-statistics/433815 for additional details.

[probot-home-assistant]

history_stats documentation
history_stats source
_{^{(message by IssueLinks)}}

[CaptTom9]

I may have the same issue. I have two history_stats entities which are based on an "on" condition (one for water heating, one for air conditioning.) I haven't had either one on for a couple of weeks. Today when I started them, one showed it had been on for 11.7 hours, the other 8.5. The history_stats values appear to have incremented normally after that, as the systems cycled on and off. Here's what one of them looked like today shortly after it was returned to service after being shut down for 2 weeks:

I’m running HA OS 8.2 and HA 2022.7.6.

[tdejneka]

I encountered the same problem.

In the chart below, note how the sensor begins to accumulate time immediately at 00:00 on Aug 2. I only noticed the anomaly mid-afternoon the following day (the AC had definitely not been running non-stop for 15 hours). I restarted Home Assistant to clear the problem.

[CaptTom9]

Today my two history_stats values are back to normal. So far, for me it's been a one-time thing; the value was zero for 8 days, then when it started accumulating again it started with a seemingly random value rather than starting with zero. I say random because this was not a previous value. In both cases the values were out of what would be a normal range.

[Zeunas]

I have the exact same issue since updating to the newest versions HA Core. I have a Tablet uptime sensor based on a Tablet switch being on/off but where before the values were being reported accurately, now it shows that my daughter has been spending like 19hours in a day on her tablet (when I know it's been much much MUCH less than that.

I can provide more details on this, just let me know what kind of info would be useful (have nothing wrong on the logs though).

[Arquiteto]

This is still an issue on 2022.9.6
I have a climate heating tracked by history stats sensor and the heat is turned off at midnight. History stats was still counting it as active time after midnight until it turned on and off again.

[CaptTom9]

It happened again a few days ago. I have a history_stats sensor which tracks how long my 1st floor (zone 1) heating is “on” over the course of the day. Yesterday it only came on just once. After that, the “Zone 1 on today” sensor suddenly went to 13.3 hours:

In the second chart, notice that the binary “Heat 1st” sensor had no state before that. I haven’t needed heat for longer than the purge_keep_days in Recorder, so the previous state change had rolled off. The sensor changed to “on” at around 1302. It stayed on for about 15 minutes and changed to “off.”

The 13.3 hours which were added to history_stats looks like the elapsed time since midnight, not since the sensor changed to “on.”

[CaptTom9]

Here's another occurrence. This time it suddenly jumped from around 2 hours to over 10.5 hours after restarting HA.

[iliasl]

I have exactly the same issue. My heating turns off exactly at midnight, yet history stats assume it never turned off

As a temporary fix, I change my accumulation period to start from 0:00:01 (1 second after midninght) and this fixed mine. There must be a bag if the state changes exactly at the same time as the accumulation window starts.

    type: time
    start: '{{ now().replace(hour=0).replace(minute=0).replace(second=1) }}'
    end: '{{ now() }}'

Below is the situation after the "fix"

[o0larrylurex0o]

I had the same problem this morning, I was about about to apply the 1 second fix, but after the HA update to 2022.12.7 my history stats was ok again (0min instead of 7h)

Was this corrected ?

[fenty17]

I'm having issues with this. The history_stats sensor resets slightly after midnight, meaning that plotting any statistics graphs with'max' picks up the value from the previous day if it was higher than today's. In my case it's how long the heating has been on each day, but I get lots of double days with the same value.

[shirou93]

My sensor showing currently time not sensor state time.

entity_id: binary_sensor.status_tv state: "on" type: time start: "{{ now().replace(hour=0, minute=0, second=0, microsecond=0) }}" end: "{{ now() }}"

[brunoluvizotto]

Same problem here.

For a daily on time integration of a sensor, if the sensor is on after midnight and turns off after some time, history_stats acts as if the sensor is on until the next time it turns on and off again.

Pretty sure that would happen if it's a monthly time integration of a sensor and the sensor is on when a new month starts - it's just 30 times less frequent.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[fenty17]

This definitely still needs attention as I'm pretty sure it hasn't been addressed. I have a workaround in place but it should just do what it says it does.

[WoosterInitiative]

This definitely still needs attention as I'm pretty sure it hasn't been addressed. I have a workaround in place but it should just do what it says it does.

@fenty17 You suggest that you've got a workaround; can you elaborate?

Seems that this should not be allowed to go stale. Partially because I'm not aware of a way in the UI to fix the discrepancies. I've had to modify directly in the db, which is super ugly.

[fenty17]

@WoosterInitiative - I was following another similar GitHub issue and got the workaround from there: #84840 (comment)

That issue has just been closed as 'not planned'. I'd much rather things like this got attention than voice assistant and Matter progress, but that's just my take.

[jata1]

Hi all - I have a related/similar issue with history stats. See thread

https://community.home-assistant.io/t/history-stats-and-graph-card-not-calculating-correctly/600896/40

the root cause is that the 12am start does not happen until 30 seconds after midnight so the max for the new day holds the max from the previous day.

strangely, if you chart using group_by max for the day, the data is correct but if using LTS (statistics) then max/day returns the max from the previous day (until the current day max > previous day max). Chart below shows the issue quite well.

I need to use LTS (as I require more than 10 days of data) but the LTS data in not correct.

[jata1]

@WoosterInitiative - I was following another similar GitHub issue and got the workaround from there: #84840 (comment)

That issue has just been closed as 'not planned'. I'd much rather things like this got attention than voice assistant and Matter progress, but that's just my take.

@fenty17 - thanks for the link and tip to reset the entity state. I have implemented this and will see how it works. Whilst setting up the python_script i noticed there is a HACS script that does the same thing and a bit more. You can just add it through the HACS GUI

[wadey]

I think I'm seeing the same issue, basically when I upgrade Home Assistant and it restarts, my history_stats type:time based on a binary sensor values are wrong for the next 24 hours, with odd spikes. See the screenshot, you can see the time period after I did the restart with all of the odd spikes:

My issue might actually be #80871

[jata1]

The fix i implemented a few weeks ago has stopped working in HA 2023.9. I use an automation to reset the state to 0 at 11.59.59. This is still working but the state is then updated again to the previous state before the reset.

Since 2023.9

When working correctly, it looks like this...

[rolandas-valantinas]

Still a problem

Seems to be fine in history

But when used in statistics graph previous days max is is being picked up

[to4ko]

having exactly same with mine history_stats

  - platform: history_stats
    name: PC S Stats
    entity_id: switch.pc_s
    state: "on"
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"

while the switch pc_s got turned off yesterday

[jata1]

I have a possible work-a-round for the issues with the history_stats reset after midnight. It is working very well for my use case (to be able to track daily hrs/use of a template switch).

The solution is to use a Utility Meter Helper and use the History_stats sensor as the target entity. The Utility Meter can be set to daily and this resets correctly at midnight.

Let me know if this works for you and if you need any further help.

[rolandas-valantinas]

I've also tried to offset start, but that hasn't changed anything - sensor values are correct but statistics of same sensor are not
start: "{{ now().replace(hour=0, minute=1, second=0, microsecond=0) }}"

[rolandas-valantinas]

@jata1 if you just need charts best work around is to use https://github.com/alexarch21/history-explorer-card
These charts work as expected, however as I want raw data for some analysis it doesn't solve the HA problem - card has export CSV but export just gives back raw data which I've posted above

[jata1]

@jata1 if you just need charts best work around is to use https://github.com/alexarch21/history-explorer-card These charts work as expected, however as I want raw data for some analysis it doesn't solve the HA problem - card has export CSV but export just gives back raw data which I've posted above

I see. Your use case is different than mine. I did post about another work-a-round that involved using a python script and an automation to reset the entity to 0 just before midnight (11.59.59) and this worked until 2023.8/9 but stopped working consistently.

Why can't you use the history_stats and a utility meter as I describe above and then just export the stats from the utility meter?

[Arturfrain]

I have a possible work-a-round for the issues with the history_stats reset after midnight. It is working very well for my use case (to be able to track daily hrs/use of a template switch).

The solution is to use a Utility Meter Helper and use the History_stats sensor as the target entity. The Utility Meter can be set to daily and this resets correctly at midnight.

Let me know if this works for you and if you need any further help.

Hello, i have the problem that i use the history_stat for daily tracking of the filter cycle of a hot tub. The filtercycle starts as soon as the electricity price is low (i use Tibber with hourly changing electricity prices). I reset the history_stat for the binary filter sensor at midnight. I observed that when the history_stat sensor is reset at midnight and the Automation switches on the filter cycle due to cheap electricity at the same time (midnight) the history_stat stays at zero for hours even when filter is running. Only after restarting Homeassistant the correct history _stat value is displayed again. Looks to me that maybe the filtering starts seconds before history _stats reset and then history_stats does not recognize any change an stays at zero.
I will try a short wait time of one Minute in my Automation before switching on filter. Maybe that fixes the problem. If not i would like to try your solution with the utility meter.
Do you use the history _stat sensor with reset at midnight and only transfer the values to the utility meter and use the utility meter instead of the history_stat sensor ? Could it be that the short time delay transferring the data to the utility meter solved the problem because it is a timing problem ? Hope you understand what i mean.
Can you explain a little bit more detailed your approach with the utility meter ?
Thank you

[jata1]

@Arturfrain - just checking to see if your issue was solved by starting the automation just after midnight? That sounds like a reasonable approach to test (try 1 or 2 minutes after 12am).

I am solving a different issue using the utility meter approach but it might still help you... My issue was that my stats were not calculating the correct max/day as the history_stat sensor would not reset until just after midnight but I do think the issues are probably related.

My understanding of your issue is that your sensor/switch is 'on' before the history_stats resets and then the switching on event is missed. I doubt my solution will work for you as the underlying issue has not been solved.

To answer the specific question - yes - I do use the utility meter to track the same data as the history stats and this works for calculating the time on/per day correctly.

[Arturfrain]

@jata1 : thank you for your answer. Yes at the moment it looks like my added wait time of 1 minute before starting the Action in my Automation seems to solve the problem. I still will observe the behavior through the next days. But it seems to me that my problem was as I thought related to a timing problem. I guess that the history_stats after resetting need a state change before it starts to actively check the sensor’s state. But this is only a guess and conclusion of my observation and maybe not correct. I will post in some days if my solution works stable. Maybe that helps others with same problem. Nevertheless i would love to understand if my interpretation is valid.

[Arturfrain]

@jata1 : my approach with short wait time before starting the Action in the Automation still works. The history_stats sensor updates on-values correctly after reset at midnight.

[CendaL]

I use utility meter workaround as the max value in statistics is wrong when you use the recommended approach for tracking time:

    ...
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"

The hourly reset is done only during the first minute which causes max to still use the previous hour value. I don't know how to fix it (I've checked the sources).

The workaround is to set up a Minute Time Tracker (I use 15 seconds precision):

template:
  - trigger:
    - platform: time_pattern
      seconds: "/15"
    sensor:
      - name: Minute Time Tracker
        state_class: measurement
        unique_id: minute_time_tracker_id
        unit_of_measurement: min
        state: "{{ (as_timestamp(now())/60) | round(2) }}"

Then I use utility meter to track states:

utility_meter:
  topeni_tracker:
    source: sensor.minute_time_tracker
    name: Topení tracker
    cycle: hourly
    tariffs:
      - on
      - off

And automation to update it:

alias: Select Topení tracker
description: ""
trigger:
  - platform: state
    entity_id:
      - binary_sensor.topeni_on
condition: []
action:
  - if:
      - condition: state
        entity_id: binary_sensor.topeni_on
        state: "on"
    then:
      - service: select.select_option
        data:
          option: "on"
        target:
          entity_id: select.topeni_tracker
    else:
      - service: select.select_option
        data:
          option: "off"
        target:
          entity_id: select.topeni_tracker
mode: single

It's a little bit more work to set it up but work correctly.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[rolandas-valantinas]

This is still an issue. Such a fundamental logic shouldn't require multiple sensors and workarounds to work.

Following documentation should result in correct MAX value, but that is not the case and therefore this is an issue yet to be resolved