Feature Request: Simulate power fluctuation to integrate better into ne energy integration

[NachtaktiverHalbaffe]

I mainly want to use this component to integrate my devices to the energy integration if they haven't energy measurements built in. So the workaround is like using this component, feeding them into the Riemann integral integration and feeding that integral integration into a utility meter.

Now the problem is: The Riemann integration only calculates the integral if the state of the underlying sensor changes. This can have issues if using the fixed mode with a device that runs for hours like an desktop during workday or even for weeks or years like my homeserver. As a consequence the state didnt change, the riemann integral doesnt calculate the energy used so it doesnt appear in the energy panel (like my homeserver as one of my biggest energy usages).

So my idea (if possible): Add fluctuation into the fixed mode. Every e.b. 10 minutes (or another poll time) change the value of the sensor like +-0.01, so the state changed event is triggered. If you fluctuate this with switching signings, this fluctuation will negate itself out so in mean it didn't affect the real energy usage. This would help to better suit the needs if it is used for the energy integration with fixed mode for devices that doesn't change it state that often.

Don't know if this is possible in code, but if possible this would be great

[bramstroker]

Sounds like a good idea. Will see what I can do.

[ThomDietrich]

Is it needed to change the value to trigger updates in other parts of HA? @bramstroker do you remember that we discussed this before 🙃😋

[NachtaktiverHalbaffe]

@ThomDietrich I dont know what it needs so that the Riemann integral update is triggered. But it only triggers right now if the power of the powercalc sensor changes and all other attributes are constant. And in the fixed mode it only changes if the underlying entity changes from on to off which e.g. my server does only when maintance is done. If it would have an attribute last_reset, it would maybe also trigger the riemann integral if this value changes.

[ThomDietrich]

My primary argument is "I agree that a regular 'update' of the sensor value will play nicely with other integrations like riemann or influxdb."
Your solutionizing suggestion to introduce a random change might not be needed :)

[NachtaktiverHalbaffe]

If the outcome is that my riemann and so on gets triggered every one hour or less im happy. How its done doesn't matter to me, this fluctuation concept was my first idea. Glad that you are working already on a solution/have some concepts for it.

As far as I know the new energy and statistics integration "measure-interval" is hourly , so that interval should be aimed that all sensors like riemann which are tied to the powercalc entity in fixed mode gets triggered in this time.

[bramstroker]

Is it needed to change the value to trigger updates in other parts of HA? @bramstroker do you remember that we discussed this before 🙃😋

I can remember you asked to prevent too many state changes. I.e. only once every 10 minutes.
This issue is about the other way around. We need HA to have the state of the power sensor change frequently so the integral sensor is behaving correctly. I don't know that much about HA internal architecture to know if we can trigger a state change without changing the value. So the approach @NachtaktiverHalbaffe proposes is kind of a workaround, but maybe the only possible solution.

[NachtaktiverHalbaffe]

If we think in the hourly interval of the statistics and energy integration, we only need to do that every 30 minutes (or two times an hour). With very little change like 0.01 and alternating with adding and sustracting this change from the specified value, the mean shouldnt be affected. E.g. if 50 W is specified as fixed power. Then having the first half hour 49.99W and the second half hour 50.01W should both result in 50Wh.

Of cause this is only a mathematical concept and a first idea. Have you already asked in the devs channels of the home assistant discord server about an another way of fixing this? There should be plenty devs that know much more about the architecture and possible ways of fixing this than me.

[nepozs]

Ideally stable power draw does not exist in real life (as it depends on mains voltage fluctuations), but I don't think that artificial modifying static power draw is proper way to ressolve problem with Riemann integral.

BTW I've also noticed this problem, but I've suspected platform: intergration (Riemann).

[NachtaktiverHalbaffe]

Yeah the Riemann integration seems to only calculate retrospective if the state changes (or then push the solution). But overall integrating a constant value which didnt change for long time is a special cornercase. Another possible solution is to trigger the "state_changed" event if this is possible, with this you dont need to manipulate the state itself. But dont know if manually triggering this event in a custom component is possible

[ThomDietrich]

@bramstroker you misunderstood me :) I fully support the feature requested here, but hope we can avoid the hacky workaround.

My recent work on improvements in https://www.home-assistant.io/integrations/statistics gave me the impression that a sensor can be updated (without changing the value, or by only changing an attribute) and other integrations or automations will react. However this is not confirmed!

I agree with @NachtaktiverHalbaffe, maybe it would be best to ask the developers community.

[bramstroker]

@ThomDietrich no problem :-). I think I have found the way to force a state update but without actually changing the value. Did some digging through the code and found the place where the state change events are triggered in home assistant core. There I noticed a force_update flag which entities can implement. This seems the way to go and a lot of other integrations also set this to True. I don't have time to implement and test this now, but maybe this weekend.

[NachtaktiverHalbaffe]

Perfect. If you wish, I can again hop on the master branch and test the fix when everything is finished.

[bramstroker]

I did found some time to implement, but I don't have time to test with the riemann integral yet. It is merged to master. @NachtaktiverHalbaffe could you try the master branch and let me know your findings?

[NachtaktiverHalbaffe]

Nice! I hopped on the master branch an report tomorrow on the testing if the first few hour-intervals are over

[bramstroker]

Great. btw I have set the update interval to 30 minutes. We can tweak this when needed

[NachtaktiverHalbaffe]

The force update works atm in my environment.

The scan interval should be discussed/tweaked. The optimal way imo is to let the user specify the poll time via the config and providing a useful standard value. The problem with 30 min is, it can update in the worst case at :01 and :31 in each hour interval, so the second half hour isnt represented in an interval. With e.g. 10 min you would only miss the last 10 min in the worst case. But this isn't a real technical discussion rather than a design/conceptional discussion.

[bramstroker]

Great news that it's working for you. I will set to 10 minutes nu default and make it configurabele soon

[bramstroker]

It was quite a struggle how to do it, but now it's possible to change the poll interval. The default is 10 minutes.

You can change the interval by adding the following to the configuration.yaml

powercalc:
  scan_interval: 00:00:30

This will be 30 seconds. When you want for example 20 minutes you set to 00:20:00.

Has been merged to the master branch. I did some testing on my installation and seems to work fine.

@NachtaktiverHalbaffe are you able to do some final testing?

When everything is ok I will issue the next release.

[cascer1]

I'm testing this setting right now and I'm not sure how to see if it works. I have an integration sensor that monitors the result of a powercalc sensor, and set the scan_interval to 00:01:00. I expected the value of the integration sensor to increment every minute as a result of the update, but this does not seem to be the case.

How can I check what's happening?

[NachtaktiverHalbaffe]

@cascer1 Which version do you have installed? You have to install the master branch and not a release (eg. v0.27). The powercalc sensor should update the state every 1 minute in your case. Under dev--> states you can select your powercalc sensor and check the time when the sensor was last updated. If this updated time updates every 1 minute, then you have misconfigured your integration sensor.

@bramstroker I jumped on the updated commit and tell tomorrow if it worked. One improvement I already have: I would cut out the hour out of scan_interval because the energy integration updates the data around every hour, so scan_intervals>1h makes no sense imo. It prevents the user to specify an unuseful scan_interval. Of course, only if it is not a big inconvenience if you implement this.

[cascer1]

@NachtaktiverHalbaffe Thanks for the reply. I think I know what the issue is. Namely, I've made a template entity that adds the power usage of all my powercalc entities together so I have a single usage number for all my lights. That entity doesn't update when the underlying powercalc entities do.

[NachtaktiverHalbaffe]

@cascer1 Yeah I have a template which generates the config for all my power sensors and if I want to make energy groups, I do an template which sums up kWh-Sensors (so basically integration sensors). This way it works perfectly. Summing up power sensors in an template wont work as you found out

[cascer1]

@NachtaktiverHalbaffe So if I understand correctly, you've set up a template that makes integration entities for each of the power sensors automatically? If so, could you share it? I'd rather not manually type out all the integration sensors for each lamp :p

[NachtaktiverHalbaffe]

It's a template I found made by Frenck. You can enter it under dev--> Template and the copy&paste it into your config. So copy&paste is still done manually, but you dont have to write out the config every time

{% for state in states -%}
{%- if state.attributes.unit_of_measurement == "W" and state.attributes.device_class == "power" -%}
- platform: integration
  source: {{ state.entity_id }}
  unit_prefix: k
  round: 2
{% endif -%}
{%- endfor -%}

[bramstroker]

It's a template I found made by Frenck. You can enter it under dev--> Template and the copy&paste it into your config. So copy&paste is still done manually, but you dont have to write out the config every time

{% for state in states -%}
{%- if state.attributes.unit_of_measurement == "W" and state.attributes.device_class == "power" -%}
- platform: integration
  source: {{ state.entity_id }}
  unit_prefix: k
  round: 2
{% endif -%}
{%- endfor -%}

This is cool! I was thinking about it, but maybe I will be able to create integration sensors for every powercalc sensor automatically within the component, so you don't need any additional configuration.

[cokeman0]

only issue, is that _power is not "private" to powercalc. I'd love to see it updated to _powercalc or _calc_power
Then we can make templates...

[NachtaktiverHalbaffe]

@cokeman0 I dont understand the problem/usecase you adress. My template doesn't filter for "_power" in the entity id or name, it filters out all entities with device class power. And if you want to use your power sensors for the energy integration, then you want all power sensors and not only the ones from powercalc. Otherwise, with the attribute "name" in the config you can override the name if you need your own naming scheme.

@bramstroker Nice Idea. My suggestion is that HA Core will support power sensors for energy integration in mid-term. So could be the case that this feature gets obsolete in any given time, but until this happens this would helpful 👍

[cokeman0]

@NachtaktiverHalbaffe thats to have a template, the sums up the power my lights are using, I have one for smartplugs, and one for vampire devices (devices not monitored, by a any device).

in my HA i now have: (lys = light)

so if powercalc had a "private" name, one can make a template. like:

 -  platform: template
    sensors:
      total_smartplugs_watt:
        friendly_name: Total smartplugs watt 
        unit_of_measurement: W
        value_template: >
          {% set ns = namespace(states=[]) %}
          {% for s in states.sensor %}
            {% if s.object_id.endswith('_smartplug') %}
              {% set ns.states = ns.states + [ s.state | float ] %}
            {% endif %}
          {% endfor %}
          {{ ns.states | sum | round(2) }}

so when adding a new smart plug, I only need to add "_smartplug" to the entity reporting the watts. and it all added up

[bramstroker]

@cokeman0 There is a new option now in the configuration entity_name_pattern. This will also change the entity_id accordingly so this allows you to postfix powercalc sensors with powercalc instead of power.
This is in the master branch. Maybe I'll have a look into having a seperate pattern of entity name and entity id. But for now it is one option.

[cokeman0]

Can't wait to test :-)

[bramstroker]

Closing this issue. When there are any other questions you are free to create a new issue.