History chart doesn't show actual temperature

[Eugen4]

Home Assistant release (hass --version):
0.57.2

Component/platform:
history

Description of problem:
Temperate chart doesn't show actual temperature.
When same temperature values comes from sensor by mqtt, it is ignored by history component due to no changes in state. When state changes (after minutes or hours) 2 dots on the charts connects by single straight line. This way of chart drawing is not correct because doen't represent actual temperature.

Expected:
I expect to see on the chart actual temperature - horizontal line until state is changed.
See attached example:

• blue line - as is
• red line - expected and actual temperature

Problem-relevant configuration.yaml entries and steps to reproduce:

sensor:
  - platform: mqtt
    state_topic: "home/bedroom/temperature"

[cgtobi]

I don't see how the red line would be more correct than the blue line, as you would rather have a transition between two points in time, than a volatile jump in temperature. And at that small scale you'd always have these minor deviations without having a higher precision sensor. Even if it doesn't look very nice I think the current implementation is perfectly fine.

[TD22057]

@cgtobi The sensor is reading 22.0, 22.0, 22.0, 22.0, 22.1. The red line is what the sensor is reading so it's correct and the blue line is wrong. The longer the sensor value stays at a constant value, the more wrong the blue line becomes. It isn't an issue that it's a volatile jump, it's an issue that the plotting value is not what the sensor said it was. If we were fitting/smoothing values, then we could discuss accuracy vs smoothness but that's not what's happening here.

[skweee]

I agree that this behaviour is a bug in many cases. The history_graph as well as the history component should be configurable.

This is a graph for the climate.eq3btsmart valve attribute using a template sensor. From 0:00 to 6:00 the valve was closed (i.e. 0% open). Then at 6:00 the state changed to 100% opened. The graph however implies that the valve opened gradually over night, which in fact it didn't do. The same behaviour can be observed from 11:00 to 16:00. For this use case the graph is not producing the right output.

A workaround for this would be to set force_update to true for the device (which unfortunately is not yet supported in climate.eq3btsmart). But this workaround would produce unnecessary log entries in the database, which in my opinion is a poor solution.

A better solution would be to let the user specify the behaviour for the history_graph:

• interpolate=true: where the user wants to interpolate lower precision sensors
• interpolate=false: where the user wants the value of the last sensor log to be displayed until the next state change

[disrupted]

@skweee exact same issue here. was also trying to visualise the data of my eQ3 thermostat and saw this bug as well. think it's a general issue either (not platform-exclusive).

[skweee]

@disrupted I agree, the interpolate idea above was not meant to be platform-specific but as an addition for the history_graph. I updated my comment above to make this clearer.

[skweee]

I locally patched the climate.eq3btsmart and sensor template components and hard-coded force_update=true. This is definitely not the correct solution, as the database is now growing in size quickly and the history charts are slower. At least I get to show what in my opinion the graphs should look like:

In my opinion, there should never be any graph interpolation for devices with force_update=false, as you can't know whether missing data points are due to the sensor being too coarse-grained or due to unchanged states.

[TD22057]

What would be really nice is if the history kept track of where there is no data vs where the data was constant. Then it could write a nan record to cover the interval when there is no data which would be a great addition to the graph. Fixing that would also fix the problem in the front end that only shows the last time the value changed, not the last time the value was updated which is really confusing.

[ebudalen]

I am having the excact same problem for showing my graph of power usage. Hope there soon will be a fix for this.

[xnyhps]

I've noticed this too with a number of my sensors. The precipitation graph from buienradar looks like a sawtooth wave on days where it's 0.0 almost all the time, but occasionally > 0. I also have some template sensors to convert another sensor which is in seconds into minutes, which also doesn't force updates.

I'm not very familiar with the internal architecture of home-assistant, but wouldn't it be better to make force_update default to True when a state is numerical?

[bokub]

I do have the same problem with my brightness sensor, because the value stays equal to 0 during the night

Right now, home assistant checks if it's about to write a value equal to the last one, and does nothing when that's the case.

Time	Value	Write
00:01	3	✅
00:02	3	❌
00:03	3	❌
00:04	3	❌
00:05	4	✅

Maybe it could write a maximum of 2 similar values, and just edit the time of the last one while the value doesn't change ?

Time	Value	Write
00:01	3	✅
00:02	3	✅
00:03	3	🕒 Change the last log time from 00:02 to 00:03
00:04	3	🕒 Change the last log time from 00:03 to 00:04
00:05	4	✅

[akloeckner]

I was going to file the very same bug report for a pilight weather sensor. Here are my fifty cent to the proposed solutions:

• interpolate = true/false option for history_graph: This would be helpful to have. It would be the correct solution for the plotting problem. But it's a rather manual solution and does not cover the case of simply "missing values". Sometimes sensor readings just don't reach my home assistant. I would like to see something more "automatic". PLUS this option. And the default should be false. As this is (I guess) also the way, home assistant will issue state change events.
• force_update = true: This sounds as a quick-n-dirty workaround. Good to know, this option exists. But it's no real solution, I think. As you state before, it will overcrowd the log.
• keeping track of no data vs unchanged data: I think, it is very hard to guess this difference from within home assistant. How would we determine this. Have we missed the value? Or was there no update from the sensor? So, I think, a failsafe implementation of this idea is simply not possible.
• updating the last log point: This comes very close to the "automatic" solution I would like to see! But: Is it at all possible to update a previous log entry? Or maybe even better: Delete the previous duplicate entry and add a new log entry?

[akloeckner]

I just tried to add force_update: True to my pilight sensor. But it does have no effect. Is there anything else, I should change? Also, an explanation here might be a good idea...

https://github.com/home-assistant/home-assistant.github.io/edit/next/source/_components/sensor.markdown

[bokub]

Looks like the new charts of 0.64 fix this, right ?

Before

After

[ebudalen]

yes, it's fixed here too. I think this issue can be closed.

[disrupted]

@bokub that's super awesome ! big fan of the new chartJS integration

[skweee]

Yep, can confirm that. The new charts fix this issue.

[bokub]

@Eugen4 can you close your issue please ?