Riemann Sum right integration providing unexpected result

[jyavenard]

The problem

I noticed that with my calculation for export/import when using the right method, I always go widely inaccurate results; even though from a mathematical standpoint it should be the best method.

I use a iotawatt sensor provides readings of a defined output for both import and export:
The sensor is defined as a calculation:
Export = (Phase1 + Phase2 + Phase3 + Tesla1 + Tesla2 + Tesla3 + Pool + Solar1 + Solar2 + AC1 + AC2 + AC3 + HotWater) min 0
Import = (Phase1 + Phase2 + Phase3 + Tesla1 + Tesla2 + Tesla3 + Pool + Solar1 + Solar2 + AC1 + AC2 + AC3 + HotWater) max 0
values for Import would be negative.

As the energy screen requires data that is positive; I use a template for that:

- platform: template
  sensors:
    power_grid_import:
      friendly_name: "Grid Power Import"
      unit_of_measurement: W
      device_class: power
      value_template: "{{ states('sensor.iotawatt_output_import')|float|abs }}"
      availability_template: "{{ states('sensor.iotawatt_output_import') not in ['unavailable', 'unknown', 'none' ] }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_output_import', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_output_import', 'state_class') }}"
    power_grid_export:
      friendly_name: "Grid Power Export"
      unit_of_measurement: W
      device_class: power
      value_template: "{{ states('sensor.iotawatt_output_export')|float|abs }}"
      availability_template: "{{ states('sensor.iotawatt_output_export') not in ['unavailable', 'unknown', 'none' ] }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_output_export', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_output_export', 'state_class') }}"

I then use the integration integration as followed:

- platform: integration
  source: sensor.power_grid_export
  name: energy_grid_export_wh_r
  unit_prefix: k
  method: right

- platform: integration
  source: sensor.power_grid_import
  name: energy_grid_import_wh_r
  unit_prefix: k
  method: right

This however provides extremely inaccurate result .
Consider the history plot for the export power reading:

However, when you look at the graph out of the integration:

What appears to happen is that the power sensor will read 0, and with a utility_meter reset at 0
The first read that is not 0 is at around 9:00AM and has a value of 188W
This is calculated as 188W*9h = 1.69kWh for the first entry; which is obviously wrong.

In that graph again, we can see that it then reads 0 again, and then 2000W 30 minutes later and that again is calculated as an extra 1kWh

So you end up shortly with an energy calculated significantly higher that what it actually is.

With Trapezoidal you also see a similar issue, though not as bad ; left works fine as it only measure from the first read (left side)

This are the value plotted using all 3 methods; the actual value reported by the energy meter is 3.9kWh export and 19.3kWh import.

As you can see, the Right value (on the left) is almost 50% over-valued.

What is version of Home Assistant Core has the issue?

core-2021.8.8

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

integration (Riemann Sum)

Link to integration documentation on our website

https://www.home-assistant.io/integrations/integration/#energy

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant]

integration documentation
integration source
_{^{(message by IssueLinks)}}

[probot-home-assistant]

Hey there @dgomes, mind taking a look at this issue as it has been labeled with an integration (integration) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[jyavenard]

I just defined a new integration sensor and I could tell that it had no data in it until it received a non-zero value from the source. It seems to be treating 0 as the absence of data rather than a plain 0.

[dgomes]

I'm having problem understanding your issue :(

The 3 methods necessarily produce different results (that's why they are configurable), there is a large error associated to this method when sampling is sparse (for best results power sensors should update every few seconds and not whenever the value changes), these are well know limitations of this approach.

In this issue you discuss the integration integration, but you don't share a graph of such sensor. If I consider the utility_meter graph as using an integration sensor, it also doesn't make sense... the integration sensor is configured to provide kWh and the utility_meter is showing Wh

[jyavenard]

I thought my explanation was pretty clear and complete.

I did provide the graph of what was integrated; it's the "Grid Power Export" one.
And while yes, a Riemann sum will provide different results according to the method use; the difference is actually quite small and will eventually even become irrelevant over time.

The sampling isn't sparse at all. The sample is happening every 30s, not when the value change. Between 5PM the prior day and 9AM the sampling read was 0.
Querying more often than 30s is however not necessary in this instance as the source will sample over 30 times per second and the power value read is the average of all those reads.
As such, polling every 30s or 1/30th of a second will yield the exact same integration value.

However, what was integrated was as if there was no data between the first sample reading 0 and the last one as if the first read that was not 0 was the average over the past few hours.
And so it does first_non_zero_value*(how_many_hours_since_last_non_zero) and leads to invalid data.

This is the problem I'm reporting.

[dgomes]

"Grid Power Export" picture above is showing W as unit (makes sense its Power), but integration is about transforming power into energy, according to your configuration if must show kWh and be named "energy_grid_export_wh_r", so where is that graph ?

30sec polling should be OK.

If you are truly providing new state update every 30 seconds that multiplication does not happen... values are updated on every state update of the source.

[jyavenard]

"Grid Power Export" is the source (pretty name of sensor.power_grid_import), and it's in W; hence the name.
"Daily Exported Energy" is graph plotted from:

utility_meter:
  daily_exported_energy:
    source: sensor.energy_grid_export_wh_r
    name: Daily Exported Energy
    cycle: daily

I put it through a daily utility meter because the scale makes it easier to understand what's going on; in the graph in the first post however, what is plotted is correctly sensor.energy_grid_export_wh_r
So you do have the graph, just the relevant part of it.

You don't need to explain to me what an integration is about, nor power vs energy, I promise you :)

If you are truly providing new state update every 30 seconds that multiplication does not happen... values are updated on every state update of the source.

Well, this is the problem I'm trying to report !

It doesn't perform the integration for every discrete measurements!
Once again, it ignores all measurements where the value was 0.
The results shown is this graph would be what I would expect if the series was:
(00:00:00, 0)
(09:00:00, 188.1)
(09:00:30, 205)
...

rather having a series that is:
(00:00:00, 0)
(00:00:30, 0)
(00:01:00, 0)
(00:01:30, 0)
... repeat
(08:59:30, 0)
(09:00:00, 188.1)
(09:00:30, 205)

why all the data are ignored I don't know. Be it in the integration integration, or the template sensor that performs abs(state("sensor.iotawatt_output_export")) is yet to be determined.
But something in the chain is broken.

I hope I've clarified the situation better now, so you don't dismiss this problem because you assume I'm misunderstanding power vs energy or that I don't know what a Riemann sum is.

[dgomes]

I'm not trying to dismiss your problem :)

It's just that often the problem reported is not the actual problem (its related but it's not the problem) and I'm only the maintainer of the integration integration.

Can you check that your template sensor is actually updated every 30 seconds ? (don't know if you can look into de recorder DB for state update)

The integration sensor treats 0 as any other value, only exceptions are UNKNOWN and UNAVAILABLE states reported by the source, in which case the integration skips those updates.

[jyavenard]

It's just that often the problem reported is not the actual problem (its related but it's not the problem) and I'm only the maintainer of the integration integration.

I didn't say the problem was with the integration integration, the bot did it :)

I'm reporting a bug/problem. I can only go so far as helping to describe the problem, the cause and their outcome. Identifying on where the problem is actually occurring is beyond my knowledge (for now)

Logging when the sensor is updated (from the original iotawatt , the template or the integration) is exactly what I'm currently discussing in the discord channel :)

only exceptions are UNKNOWN and UNAVAILABLE states reported by the source, in which case the integration skips those updates.
If you look in the template calculating the absolute value; you'll see that I added an argument to the template to not even attempt to update the sensor then because it would store zero in that sensor after a reboot until a new value was read and that too was screwing up the daily meter.

[jyavenard]

Ok.

The issue is with the iotawatt integration, it just doesn't lodge a state change when a new read occur and the value is identical to the previous one.

[dgomes]

OK looks like the root cause as been identified

Personal comment: Since iotawatt is actually a core integration, I think the integration could be done by iotawatt internally (not requiring the use of integration), it would provide end-user with better UX