Long term statistics are incorrect because recorder.statistics.get_last_statistics converts to user-defined units

[jaharkes]

I noticed this when I was capturing volumetric data in cubic feet where the long term statistic is normalized to cubic meters.

The sensor is configured as {state_class: total_increasing, device_class: gas, unit_of_measurement: ft^3} and the values in the states table are stored as their original cubic feet numbers. The statistics table values are stored converted to cubic meters, but the summed value show some sort of exponential growth.

5309|2021-09-03 00:12:00.374019|44|2021-09-02 23:00:00.000000|||||5226.8934465076|0.0566336931997284
5346|2021-09-03 01:12:00.949944|44|2021-09-03 00:00:00.000000|||||5227.1199812804|5229.11998127974
5383|2021-09-03 02:12:10.469552|44|2021-09-03 01:00:00.000000|||||5227.2332486668|189891.862521871

Comparing to statistics of sensors that do look normal, the last columns is supposed to add just the small differences between measurements. The following logged entries are relevant to these bad data jumps.

2021-09-02 21:12:00 INFO (Recorder) [homeassistant.components.sensor.recorder] Detected new cycle for sensor.gas_meter, value dropped from 5226.8934465076 to 184585.99993983016
2021-09-02 22:12:10 INFO (Recorder) [homeassistant.components.sensor.recorder] Detected new cycle for sensor.gas_meter, value dropped from 5227.1199812804 to 184593.99993982754

In these messages, the first value is the m^3 value, while the second value is (a close approximation of) the original ft^3 data.

This is caused by the fact that the relevant code in components/sensor/recorder.py is using statistics.get_last_statistics which returns values converted for display (in my case ft^3) which are then compared against the normalized values from fstates which are m^3.

core/homeassistant/components/sensor/recorder.py

Line 341 in 8af0cb9

unit, fstates = _normalize_states(hass, entity_history, device_class, entity_id)

core/homeassistant/components/sensor/recorder.py

Line 387 in 8af0cb9

last_stats = statistics.get_last_statistics(hass, 1, entity_id)

core/homeassistant/components/recorder/statistics.py

Line 375 in 8af0cb9

def get_last_statistics(

[jaharkes]

Sorry for the 'assign' @emontnemery, but looking at the history of these files I think you are probably more familiar with these parts of the code.

[probot-home-assistant]

recorder documentation
recorder source
_{^{(message by IssueLinks)}}

[ghost]

This bug also can cause the calculated sum to return as 'infinite' and halt the processing of all long term statistics.

[emontnemery]

Thanks @jaharkes for the excellent analysis of the error 👍