window_state does not change, configuration issue? #181
Comments
|
Same issue here in a different room: no open window state and a slope jumping around wildly: |
|
And in a different room the slope is at zero for many hours, even though the temperature changes: |
|
Hello @maia , It works on my side with exactly the same configuration. So you need positive value and the slope should be under 0.05 during 30 sec for the detection to be effective. Here is a test I just do and it works:
The configuration is: The only difference is that I'm in over_switch VTherm type but it should not have an impact. |
|
Ok, I just made a test (in dev environment) with a over_climate VTherm and you are right, it seems not working at all. Thank for the head up ! |
|
Thanks. I hope the temperature slope issue is a part of the problem. In case it's not related at all, let me know so we can create a separate issue. Good luck on bughunting! |
|
It was simply not implemented for over_climate VTherm or a regression due to an ancien issue. |
|
Thanks. Should this also fix the slope calculations, as reported in the screenshots above? |
|
Non there is no need to fix something on slope calculation. There is a temporal filter to avoid mini-slope deviation due to precision +/- 0.1 gap. It should works fine event if the curve seems a little agitated. |
It does not work for me. I might need to create a new issue, as this one is closed. |
|
No, we will continue here. |
|
Please see the image here: |
|
Mine:
I know this will not help you, but this works fine with 4.0.2 in my context. To have more information, you will to turn the debug level in logs and check those logs: previous logs gives all the details. Be careful in debug log level, the log will growth rapidly. So just turn it on, take the log and set the level to info back. The algo is based is forgetting abnormal values (too quick or too high) to avoid some thermometer high jump (there is one in my screenshot which is ignored). EDIT: i agree with you, your graph shows it don't works |
|
Unfortunately the log lines do not contain the name of the entity, so they might mix in the log. But anyways, here's a chart plus the debug log file using 
|
|
@jmcollin78 Can the problem with the cycle calculation that you attempt to fix in 4.0.3 cause the issues with slope calculation? |
|
Normally not. The slope is time based : Delta Temp / Delta time. |
Well, if delta time is zero or just a few seconds, it might mess up the slope? I havn't looked into the slope calculation, but it would be great if the slope is usable to predict the room temperature in the next cycle (in 5 minutes). But maybe that would need two slopes, one short term and one medium term. The "medium term" slope would be something like a regression based on all measurements of the past 30 minutes used to predict the temperature in 5 minutes. Having an algorithm deal with a predicted temperature in 5 minutes might increase its performance and reduce over/undershooting: "If the status quo continues, will the temperature rise or fall in the next 5 minutes and will we need additional heating or not?" By the way, the slope is also an aspect of different room physics. To me it feels like it would be great to have one place in the config where one defines the physics (from "slow to heat, slow to cool down" to "rapidly changes to heating and open windows") and that single setting defines some (user hidden) variables that influence slope calculation, self regulation etc. |
|
Yes that's is why I have a minimal delay (10 sec) between 2 measurements. The algorithm don't like the noise giving +0.1 and -0.1° sometime many times per minutes. delta_t=0.269 delta_temp=0.100 new_slope=0.371 last_slope=0.07340745304406945 slope=0.222 delta_t is very short (16 sec) and 0.1° in 16 sec means a big gap which produce some bad slope measurement. My temperature curve are mostly smooth and that's why don't experience your problem. I guess my magic value of 10 sec in not accurate for your case. Maybe we should adapt it to 30 sec in your case. If you can have a try it is located in open_window_algorithm.py line 15. The difficulty here is to remove some aberrant point (see my temperature graph). So I have temporal filter, aberrant point removal based on threshold, and low pass filter with mobile mean ( val(N) = mes(N) x 0.5 + mes(N-1) x 0.5) |
|
I will try that before I have to reboot HA the next time. Rebooting is always as issue as the entire thread mesh network needs to recreate itself and that takes a while and sometimes some devices don't join. But in general I think this part of the code could be drastically improved by using more than the last 2 temperature measurements and the time inbetween, this will always be inexact. I don't have a pseudo-code solution yet though. |
|
I include the fix into the https://github.com/jmcollin78/versatile_thermostat/releases/tag/4.0.4 |
|
The slope now has less wild swings. Unfortunately it's not able to detect open windows: And when I opened the window, the slope in that room didn't change more than when the window was closed: From all I've read about unsupervised time series forecasting today (e.g. see this link) I believe the biggest improvement is to use an exponential smoothing for the temperature: Take the temperature for the past 10 minutes (one measurement per minute, if not available then calculate from the available data points), calculate a EMA for the past few minutes and use the difference between the last two smoothed points as slope. |
|
I do an exponential mobile average calculation but not on the temperature itself but on the slope (which is not suppose to change on a wide range too fast). I will have a look at this. Maybe something like: temp(N) = temp_measurement (N) x 0,60 + temp(N-1) x 0,40. This will largely smooth the temperature curve but not too much which will introduce too much latency in detection. I have to make some simulation with your measurements. |
jmcollin78
added
enhancement
|
Jean-Marc, I have a couple of suggestions:
def calculate_ema(previous_ema, current_measurement, num_periods):
if previous_ema is None or previous_ema == 0:
return current_measurement
alpha = 2 / (num_periods + 1)
smoothed_value = alpha * current_measurement + (1 - alpha) * previous_ema
return smoothed_value(I'm just starting to learn python, so I hope that's valid)
|
Agree.
Yes
yes. I always use 0.5 myself to get a relatively dynamic curve, but this can be variabilized for sure. Thank you for your suggestion.
each time the temperature change. So We must integrate the delta time to be correct. That's why I ignore to frequent measurement. I cannot have another cycle for this calculation and having it calculated at cycle only will be not suitable for long cycle. I have to find a formula for which we can take into account the dt factor. I think that's why I smooth the slope (which dtemp / dt) and not smooth the temp only. EDIT: are you really a statistician or like me just a sunday stastistician user ? |
|
I agree, it's a goal to have a place in code where measurements can reported the moment they happen and are processed, to be available for slope calculation, regulation etc. – and this must take the timestamps of the measurements into account, by e.g. creating a 1 minute timeframe and if multiple measurements are taken in that single minute then we use the average and if there's no measurement over a few minutes then we calculate the sliding average between the available measurements.
I don't have academic training in statistics but statistics have been a rather large part of my profession over the past 20 years, from classical surveying to predictive analytics, financial markets and language models - and anything inbetween. |
|
I've seen that it's common to use EMA for irregular time series. In that case the input is a list of tuples containing the timestamp and corresponding measured temperature value. The timestamps should be in ascending order. The EMA is calculated by using the distance between two measurements to calculate the alpha. See this explanation, scroll down to "Irregular Intervals". Also see this pdf paper (page 9). EDIT: The decay/half-life (denominator G) in the first example is the one point where one can define if the EMA should only look at very recent data (e.g. 10 minute) or longer time frames. |
This seems very easy to implement. I will try this maybe this week-end. |
I agree, it is promising. I also found this alpha calculation, it probably is the better choice and maybe you want to consider it instead (in C++ and Ruby, isn't there any AI translation service into Python?): double alpha = 1 - std::exp(std::log(0.5) * time_decay / halflife);
double ewma = alpha * measurement + (1 - alpha) * previous_ewma;alpha = 1 - Math::E ** (Math.log(0.5) * time_decay / halflife )
ewma = alpha * measurement + (1 - alpha) * previous_ewmaAs for using the data: You could calculate the EMA and keep all calculated EMA values of the past 60 minutes in memory (or only keep a maximum of one per minute, it is smoothed anyway). This would allow to calculate a slow and a fast slope (e.g. 60 minutes and 10 minutes). And having this in a central place even allows to use the EMA values for a regression calculation to calculate the slope at some point in the future. It will allow predicting the temperature in the future (in 5 minutes) and this value could be used in the regulation algorithm. As for the EMA calculation of irregular time series, I've seen that it might be useful to have an upper limit for alpha in case the last measurement was too long ago. For example when using a half life of 10 minutes a measurement that is 60 minutes ago (if there's nothing inbetween) would contribute to the smoothed value with 1,5%, giving the current measurement 98,5% relevance. It could be wise to limit the alpha to e.g. 4x the half life ( |
|
Planned for 4.2.0 (next release). Maybe next week or this week if i have got enough time. |
|
Finally, I succeed to implement the EMA algorithm, add a ema_temp new attribute which is the smoothed temperature value and I use it into the automatic window detection algorithm. https://github.com/jmcollin78/versatile_thermostat/releases/tag/4.2.0.alpha1 I create a Discussion announcement if we want to discuss more about this. EDIT: |
|
Hello @maia , I have publish a alpha8 pre-release which fixes all slope artifacts. Can you have a try with these release ? The other think I have discovered this integration which is really helpfull to display graphs with zoom and much more. I have beautiful analysis graph now:
|
|
Maybe I have a precision issue. When the temperature is slowly decreasing I should have a small negative value and not 0 which is case here. Maybe the unit °C/min is not accurate and a unit as °C/hour would be better. My room is loosing 1° in 3 hours. This make 0,33 °C/hours = 0,005 °C/min -> slope is 0. |
|
Another room:
|
|
I agree, a slope measured in °C/hour is better interpretable for humans. And anything users configure themselves should be as understandable as possible. Regarding alpha8, which I installed yesterday, the calculated slopes unfortunately still cause false positives when the temperature changes by only 0,1°C and looking at these charts I couldn't say which threshold I would want to use. (I opened the windows in all three rooms at the same time and for the same duration, with 0°C outside) Unfortunately I'm currently rather busy and don't have time to investigate possible solutions for a better calculation of the smoothed temperature. But as this smoothing is partly relevant also for an improved regulated temperature, I'm hoping that someone can come up with a solution. Because otherwise I'd want to suggest to change the default cycle from 5 minutes to 1 minute (for everything, not just the slope), I'm not yet aware of any downside to having shorter cycles. 
|
|
Because you have a very volatile thermometer you certainly need to "smooth" more the EMA. I will expose the possibility to change the parameters of EMA calculation (in the configuration.yaml to avoid adding more configuration parameters in the UI). I think this is the solution for your case. You will then have more "latency" and detection will be delayed but we cannot have both : slow latency and heavy smoothing. |
The recent screenshots show different temperature sensors, one of them is from Aqara and as far as I know, it's a pretty frequently used sensor. Ideally 90% of the users do not need to change parameters, neither in yaml nor the UI. It should work out of the box. I think you personally don't see these issues as your rooms change temperature very, very rapidly, so any 0,1°C change the sensor reports is only a blip in your charts. |
|
An additional info to alpha8: I'm still seeing strong spikes in the calculated slope: |
|
Thank I will try to reproduce with the value. I wonder how long will your battery support this behavior. EDIT: |
|
Here's a zoom. Not sure how to export the ema values, the lovelace card I usually use only allows to export entities, not attributes. Will look into it. Also I'll try a 1min average smoothing of the raw temperature as input for vtherm. The batteries (2xAAA) hold for 1-2 years. While powering a display. BLE is very efficient it seems. |
|
I have released the 4.2.0 with the capbility to use your own parameters for EMA calculation. |
VTherm is not changing the window_state to "open" when the slope drops below the configured value:
I've configured all VTherm entities using the option "window detection" with the suggested value for temperature slope of 0.05 (yaml see below):
Unfortunately while there are changes in the slope, the window_state does not change to open (on):
By the way, in the configuration you ask for a "decrease" with the info: "Recommended value: between 0.05 and 0.1. Leave empty if automatic window open detection is not use". So I assume one needs to put a positive value in there, because a negative value of a decrease would be an increase in temperature? Otherwise, if the configuration requires a negative value, you shouldn't ask for a decrease but for a change and also change the recommended value to negative value.
PS: Why does the slope jump around so much, if the room temperature is usually pretty stable? I don't understand the slope when comparing it with the room temperature in the screenshot.
Here is the configuration:
The text was updated successfully, but these errors were encountered: