Split sensors in live and statistical

[DarwinsBuddy]

enhancement(statistics-sensor): Move statistics sensor to separate entity
Increase scan interval to 60 minutes

[codecov]

Codecov Report

Merging #89 (4faf048) into f/statistics (05365db) will decrease coverage by 5.19%.
The diff coverage is 1.12%.

@@               Coverage Diff                @@
##           f/statistics      #89      +/-   ##
================================================
- Coverage         29.56%   24.37%   -5.19%     
================================================
  Files                 9       12       +3     
  Lines               433      521      +88     
  Branches             55       64       +9     
================================================
- Hits                128      127       -1     
- Misses              301      390      +89     
  Partials              4        4              

Impacted Files	Coverage Δ
custom_components/wnsm/base_sensor.py	0.00% <0.00%> (ø)
custom_components/wnsm/config_flow.py	0.00% <ø> (ø)
custom_components/wnsm/live_sensor.py	0.00% <0.00%> (ø)
custom_components/wnsm/sensor.py	0.00% <0.00%> (ø)
custom_components/wnsm/statistics_sensor.py	0.00% <0.00%> (ø)
custom_components/wnsm/utils.py	0.00% <0.00%> (ø)
custom_components/wnsm/api/client.py	67.96% <75.00%> (-0.25%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[DarwinsBuddy]

please test thoroughly. 🙏

[reox]

Thanks! I just spun up a docker image and it works ;)
But I guess you cannot move from the f/statistics branch to the new one, because the statistics sensor is now called _statistics?
However, you should be able to migrate from the current main branch, right?

I dare to switch now to this branch on my "production" system and keep it running for some days...

[salzrat]

is it safe to copy these files over the main branch installed from hacs?

[reox]

is it safe to copy these files over the main branch installed from hacs?

I never did that. I installed the repository manually by cloning it directly from github.
Maybe it works, not sure though if hacs then gets confused...

[salzrat]

I think there shouldn't be any problem with HACS as it basically does just that. But can you just overwrite the last main version with this branch and it will continue to work, or should one reinstall and run through setup again?

[reox]

I think there shouldn't be any problem with HACS as it basically does just that. But can you just overwrite the last main version with this branch and it will continue to work, or should one reinstall and run through setup again?

I guess that it should work, just as before. I use the statistics branch now for several months and I think I will have to reset the statistics, because the statistics are now tracked in another entity. However, the entity just tracking the meter reading is unchanged for this branch, thus it should keep working.
The easiest would be to spin up a docker container and test the change there. The only issue is, to thoroughly test that change, you would need to have some historical data already.

[salzrat]

so the meter reading entity is the one which goes into the energy dashboard?

[DarwinsBuddy]

@reox I intentionally created another sensor with a suffix for the statistical readings to be downwards compatible. Users checking out the branch/PR and testing already should be capable of cleaning up their set up on official upgrade imho.
The default user should not be bothered with such tasks.

If you are confident that this split is okay, please approve and fast-forward merge my f/statistics_split into f/statistics.
Afterwards I'd approve yours and merge it into main, unless there is still some things left to discuss.

Thank you for the huge effort taken! ❤️

[reox]

That depends how you configured it. If you came from the main branch, then you would only have the meter reading entity. In the statistics branch, I swapped this by a sensor that read the hourly consumption instead of the meter reading.
Now, with the statistics_split branch, you have the original meter reading entity and a new entity called XXX_statistics which gathers the hourly statistics.
In the energy dashboard, you would probably want to configure the statistics entity only, because it has the correct consumption at the correct date. The other entity with the meter reading could however be used for automations (but is always delayed 24h+)

[reox]

If you are confident that this split is okay, please approve and fast-forward merge my f/statistics_split into f/statistics.
Afterwards I'd approve yours and merge it into main, unless there is still some things left to discuss.

I switched to this branch yesterday on my HA system, and I will keep it running for a week or so. I guess that should be enough testing then ;)

[DarwinsBuddy]

If you are confident that this split is okay, please approve and fast-forward merge my f/statistics_split into f/statistics.
Afterwards I'd approve yours and merge it into main, unless there is still some things left to discuss.

I switched to this branch yesterday on my HA system, and I will keep it running for a week or so. I guess that should be enough testing then ;)

Agree. Let's see how it goes and talk in a week :)

[reox]

Agree. Let's see how it goes and talk in a week :)

yep! I was already shocked to see "update of the sensor has failed" but these were just timeouts... 😀

Maybe add TimeoutError as an exception to the update function and just write a warning to the log?

[DarwinsBuddy]

@reox like this? 🙃

[reox]

hehe, yes ;)

btw, for the logging style: I would not use the old % style, but either format strings everywhere or pass the to formatting strings as parameters to the log function:

var = "world"
logger.debug('hello %s', var)  # pass argument to logger for lazy expansion of string
logger.debug(f'hello {var}')  # use the f-string power, but don't profit from lazy expansion
logger.debug('hello %s' % var)   # I do not like it ;)

or what do you think?

[DarwinsBuddy]

hehe, yes ;)

btw, for the logging style: I would not use the old % style, but either format strings everywhere or pass the to formatting strings as parameters to the log function:

var = "world"
logger.debug('hello %s', var)  # pass argument to logger for lazy expansion of string
logger.debug(f'hello {var}')  # use the f-string power, but don't profit from lazy expansion
logger.debug('hello %s' % var)   # I do not like it ;)

or what do you think?

Usually I'd agree, but for debug logging lazy evaluation is computational more efficient. Most of the time the log level will be INFO or ERROR, but usually not DEBUG.

having the f'' syntax leads to the string being formatted every time the interpreter is passing that line regardless if it's needed or not. lazy evaluation is only performed in case of the string being used by the respective logging function.

I am also a fan of f-formatting but it has its drawbacks.

[reox]

But if you write 'hello %s' % 'world', then this string is also directly evaluated, or not?
Thus in that case you have no advantage over f-strings.
Only if you pass the values as arguments to the debug function, you have the lazy evaluation - or not?

[DarwinsBuddy]

afaik the %-notation will not evalulate the string before the function call.
https://news.ycombinator.com/item?id=19991634

passing it to the logging function may be an option, but I do not know how this will be formatted eventually.

for debug logging it introduces obsolete computational effort for string interpolation

[salzrat]

so you're about to merge into the main branch? should I still wait?

[reox]

yes, but it might take some time until it is in main.

[salzrat]

so I just copied the f/initial_import over my current installation and restarted HA. When should I start to see anything new? Or do I have to re-login (and how do I do that)?

[reox]

you should see a second sensor with the name AT001....._statistics. If you configure this sensor to track the energy, you should also see the hourly consumption in the energy dashboard.
Or you use a widget like these:

Track yesterday's and day-before-yesterday's consumption (Similar to smartmeter-web):

type: horizontal-stack
cards:
  - type: statistic
    entity: sensor.atXXX_statistics
    icon: mdi:transmission-tower-import
    name: Gestern
    period:
      calendar:
        period: day
        offset: -1
    stat_type: change
  - type: statistic
    entity: sensor.atXXX_statistics
    icon: mdi:transmission-tower-import
    name: Vorgestern
    period:
      calendar:
        period: day
        offset: -2
    stat_type: change

Graph of the consumption

chart_type: bar
period: hour
days_to_show: 2
type: statistics-graph
entities:
  - entity: sensor.atXXX_statistics
    name: Energy Consumption
stat_types:
  - state

[salzrat]

ok, yes, I see this sensor, but if I click on it it says "no statistics found"... how long should it take to get populated with something?

[reox]

yep, that is correct. This sensor has no state and thus you cannot show it in the entity viewer... Homeassistant can be weird :D
You have to setup such a widget or use the energy dashboard.

[salzrat]

Oh great!
But two questions:

• in the energy dashboard, it populated everything except today - will this still show up?
• how can browse the data outside the energy dashboard? in the history dashboard, nothing is shown because there are no states, just statistics?

[reox]

in the energy dashboard, it populated everything except today - will this still show up?

It works the same way as in the smartmeter-web. The current day will show up typically on the next day at around 02:00 to 03:00. That's how they designed it...

in the history dashboard, nothing is shown because there are no states, just statistics?

Yes. That is due to the way the statistics import works / how homeassistant works in general. As of writing, there seems to be no way to import historical states as well. There are many discussions in the HA community forum about that.

how can browse the data outside the energy dashboard?

As I said, use the above shown widgets or create something similar. With the statistics-graph and statistic widgets, you can query for this data.

[salzrat]

Ok, understood. Really weird how HA works here.

But is there a way to show the energy together with another entity in a graph? I use the epex spot plugin to give me epex spot prices for awattar, so it would be great to be able to compare consumption and prices in a single graph!

[reox]

You can add more entities into the statistics-graph, "normal" entities work in those as well.

[salzrat]

I tried, but the epex entities don't show up... Actually, the only entities that show up in the statistics graph config panel are the things that also show up in the energy panel...

[salzrat]

Ok, apexcharts to the rescue (wow that is really a great addon):

header:
  show: true
  title: Spot vs Energy
span:
  start: day
  offset: '-2day'
yaxis:
  - id: kw
    min: 0
    opposite: true
  - id: price
    min: 0
all_series_config:
  show:
    legend_value: false
  curve: stepline
  stroke_width: 2
series:
  - entity: sensor.epex_spot_at_price
    yaxis_id: price
    unit: ct/kWh
    type: line
    transform: return x * 1.2 * 1.03 / 10
  - entity: sensor.atxxxxxxx_statistics
    yaxis_id: kw
    type: line
    statistics:
      type: state

[salzrat]

Ok, now I want to compute the daily average price, but without an entity with states, this seems impossible, as I can't access the statistics?

[salzrat]

Maybe an alternative would be to have a sensor that has the hourly values as attributes as soon as they are available. Then this could be somehow combined in a template sensor maybe...

[reox]

Maybe an alternative would be to have a sensor that has the hourly values as attributes as soon as they are available. Then this could be somehow combined in a template sensor maybe...

No, that does not work that way. The hourly data is added in the API for a whole day (or even longer period of time) at once, typically on the next day at 02:00 UTC.
To make the state available you would now have to process each value, wait 15min, process the next, etc. But then each state would be on the wrong time.

The general problem is, that HA works in a way that a sensor state is measured at a given time and is not attached with a time of measurement.
There are currently two workarounds: 1) use the SQL Database directly, but this is unsafe and each change in the database will mess up everything or 2) use the statistics, which has a more or less stable API but cannot back-date states.
In the HA community discourse, this is still an open feature request to be able to import old states.

[salzrat]

No, what I meant is similar to what the epex data does. Please look here:
https://github.com/mampfes/ha_epex_spot

The epex entity has the current price, but as an attribute, it has a map of the complete hourly data for the current and next day if available. Using apexcharts, this can be plotted nicely (see bottom of epex page).

For the smartmeter, you could create an entity that changes the state whenever the new hourly data becomes available. It doesn't really matter what the state is - it could even be the date of the last available hourly data. As attributes, you attach a map of the past 24-48 hours of hourly data. In this way, in a template one could access the current state, and read the hourly data (which should be in some format (timestamp,kwh) and do something with it...

[reox]

Ah okay, I see. That sounds like an easy fix, but I'm still not sure it will get rid of all problems :D

[salzrat]

Sure, it would be something to try on a separate branch maybe :) I could test whether it can do what I would like it to do :)

[DarwinsBuddy]

@salzrat fyi https://github.com/DarwinsBuddy/WienerNetzeSmartmeter/releases/tag/v1.1.0

[salzrat]

Btw, I managed to get the average price based on hourly consumption using an SQL sensor. It calculates the daily average price, and also stores the epex average price and difference to it as attributes. It does this for the 3rd day in the past, since there we can be reasonably sure to have energy values:

SELECT * FROM (
  WITH querydate AS (SELECT date('now','-3 days', 'localtime') AS date)
  SELECT *, daily-epex AS diff FROM (
  SELECT date(states.last_updated_ts, 'auto', 'localtime') AS date,
    (SELECT states.state*1.2*1.03/10
     FROM "states"
     WHERE states.metadata_id = (SELECT metadata_id FROM states_meta WHERE entity_id = 'sensor.epex_spot_at_average_price')
     AND date(states.last_updated_ts, 'auto', 'localtime') = querydate.date
    ) "epex",  
    SUM(states.state*1.2*1.03/10*statistics.state) / 
      (SELECT SUM(statistics.state)
       FROM "statistics"
       WHERE statistics.metadata_id = (SELECT id FROM statistics_meta WHERE statistic_id = 'sensor.atxxxx_statistics')
       AND date(statistics.start_ts, 'auto', 'localtime') = querydate.date
      ) "daily"
  FROM "states", querydate
  INNER JOIN "statistics" ON datetime(states.last_updated_ts, 'auto') = datetime(statistics.start_ts, 'auto')
  WHERE states.metadata_id = (SELECT metadata_id FROM states_meta WHERE entity_id = 'sensor.epex_spot_at_price')
    AND statistics.metadata_id = (SELECT id FROM statistics_meta WHERE statistic_id = 'sensor.atxxx_statistics')
    AND date(states.last_updated_ts, 'auto', 'localtime') = querydate.date
  )
)

I can then plot this using apexcharts and compare it to the epex average price by offsetting that by -3 days.

[salzrat]

Is there a way to resynchronize? I am missing a couple of days of energy consumption in the sensor (energy dashboard), but they have since appeared in the smartmeter portal, and I would like them to appear...

[reox]

No, unfortunately the only way would be to remove the integration, cleanup the statistics and re-import everything.
The problem is, that we cannot import statistics between already imported statistics. At least thats what I know. Maybe there is a special API that can do that...

[salzrat]

So maybe you could add an option that does that? Removing and reading seems intimidating…

[DarwinsBuddy]

I'm currently traveling and have not mich time to investigate further, but I've got potentially an idea how to quickly remove old statistical data. under development tools in hassio there is a service called purge entities which can basically remove this data. (don't know for sure if it's removing recorder imported ones, but it's maybe worth a try).

…

On Wed, May 10, 2023, 14:55 salzrat ***@***.***> wrote: So maybe you could add an option that does that? Removing and reading seems intimidating… — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADX2VGWRKMZ5HWY7H7VQXLXFP6DLANCNFSM6AAAAAAWP4YQUI> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[reox]

I created a new issue for that: #140