Since SENEC Application v0825,v0826 & v0827 no STATISTIC data is provided [No Data for multiple Sensors]

[marq24]

Problem

When your SENEC.Home will be updated to v0825 (also after the update to v0826) the integration is not able to provide data for a couple of sensors..

Affected Sensors (that's all the total sum data displayed in kWh):

• House consumed
• Solar generated
• Battery charged
• Battery discharged
• Grid Imported
• Grid Exported
• Wallbox charged

The reason behind is, that SENEC has decided to exclude 'STATISTIC' information from the response of the 'lala.cgi'... That's quite sad, since these sensors are mainly used for the Home Assistent Energy View...

Right now there is no quick fix to this situation - but since also data for 'DISPLAY' is missing my guess is, that SENEC is just building so called Banana-Software. Without "DISPLAY" information from the lala.cgi the /Display.html does not work at all - so I have hope, that the empty response parts have slipped through QA/QC

Current Solutions to access your statistics data

Please read the complete post/Issue text - since at the bottom you will find important information for you Energy Dashboard

'mein-senec.de' WebAPI

When you update this Integration (ha-senec-v3) to the latest version you have the option to add an additional integration entry. When you start the process you have to select 'WEB.API: mein-senec.de Portal (usable with all SENEC.Home variants)' from the selection list and then provide your mein-senec.de credentials. This will add 13 additional Sensors to your Home Assistant (currently wallbox data is missing - cause I don't have one (yet)) - when you know the fields in the web-api - just let me know. Thanks to @mstuettgen!

Mapping lala.cgi to web-api Sensor names

Please find below a table comparing the sensor names between the classic integration and the web-api based integration - this comparison assume, that you did not change the names ['senec' & 'senec_wabapi'] since they will be used as prefixes

Name	lala.cgi based	web-api
House consumed	sensor.senec_house_total_consumption	sensor.senec_wabapi_consumption_total
Solar generated	sensor.senec_solar_total_generated	sensor.senec_wabapi_powergenerated_total
Battery discharged	sensor.senec_battery_total_discharged	sensor.senec_wabapi_accuimport_total
Battery charged	sensor.senec_battery_total_charged	sensor.senec_wabapi_accuexport_total
Grid Imported	sensor.senec_grid_total_import	sensor.senec_wabapi_gridimport_total
Grid Exported	sensor.senec_grid_total_export	sensor.senec_wabapi_gridexport_total
Wallbox charged	sensor.senec_wallbox_energy	-not-implemented-yet-

Riemann Sum Integal Platform

AS WORKARROUND you can add the following section to your ha configuration.yaml (if you have already a sensor section obviously you just need to add the sensor's itself) - unfortunatly the the integration platform does not allow to provide icon's - so you have to edit them manually later via the GUI (if you need/want them)...

Please note, that the source sensor names based on the default integration name 'senec' - if you modified the name during the initial setup you need to adjust the source sensors.

sensor:
  - platform: integration
    source: sensor.senec_house_power
    name: senec_house_total_consumption_fb
    unique_id: senec_house_total_consumption_fb
    #icon: mdi:home-import-outline
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_solar_generated_power
    name: senec_solar_total_generated_fb
    unique_id: senec_solar_total_generated_fb
    #icon: mdi:solar-power
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_battery_charge_power
    name: senec_battery_total_charged_fb
    unique_id: senec_battery_total_charged_fb
    #icon: mdi:home-battery
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_battery_discharge_power
    name: senec_battery_total_discharged_fb
    unique_id: senec_battery_total_discharged_fb
    #icon: mdi:home-battery-outline
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_grid_imported_power
    name: senec_grid_total_import_fb
    unique_id: senec_grid_total_import_fb
    #icon: mdi:transmission-tower-import
    unit_prefix: k
    round: 2

  - platform: integration
    source: sensor.senec_grid_exported_power
    name: senec_grid_total_export_fb
    unique_id: senec_grid_total_export_fb
    #icon: mdi:transmission-tower-export
    unit_prefix: k
    round: 2

#  - platform: integration
#    source: sensor.senec_???
#    name: senec_wallbox_energy_fb
#    unique_id: senec_wallbox_energy_fb
#    icon: "mdi:ev-station"
#    unit_prefix: k
#    round: 2    

Handling the change of the EntityIDs in the Energy Dashboard

Since we have the situation, that some SENEC.Home Systems still return STATISIC data (since they are not updated) and that this data might come back with another Update - who knows? - the EntityIDs of the new Sensors (that have been added in both cases (WebAPI or Riemann)) are different by purpose.

When you want to keep your historic data in the Energy Dashboard (and I guess you want) then the current known best way is described here in an additional issue: How to configure Energie Dashboards (after SENEC.Home have been updated to 0825+)

[electricdelicate]

Oh no ....

[neubi4]

If i got is correctly, these are all "daily usage" statistics data, e.g. how many kWh House consumed today.

This should be replaceable by using an Home Assistant Riemann sum integral (https://www.home-assistant.io/integrations/integration/#energy) which simply can summarize House Power (W) to kWh like in the linked example.

Since this counts infinitely higher, to get an daily used you can use Utility Meter (https://www.home-assistant.io/integrations/utility_meter/) to get an sensor on the integral which is resetted daily/monthly value.

These Utility Meter sensors then could be used in Home Assistents Energy View.

We cloud provice YAML examples for this in the readme of this repo.

[neubi4]

After looking at the old data, it seems that the Utility Metern is not needes since the old data also only counts upwards infinitely.

So it should be fine only using the Riemann sum integral sensors. I will try this.

[marq24]

FUNNY - while I am tyring - you already provide the info...

the values (that are still available in my environment) are constantly growing - so there is no need to reset them on a monthly/weekly/daily/hourly basis - so looks like using the suggested integration from @neubi4 sounds like a decent replacement...

Once the http-v3 build is confirmed as "working" I can spend some time working on example yaml parts that you can copy&paste in your ha configuration.yaml...

[electricdelicate]

Sounds great, thank you for your support!

[marq24]

added workaround information in the initial post

[egonkab]

hello marq24,

thanks for your hard work. It works very well.
I use Powercalc to calculate the total sensors, this also works well.

[sigcarunchio]

Hello marq24, just discovered your fork after firmware update, great work!

1. About missing entities, i also redo them using custom sum integration templates, but instead using yaml code, i use helpers; do you think it's the same?

2. My problem is that i have a master and a slave senec v3; i think that missing statistics gave already total values, now (but i confirm this later) with custom sensor i have only data about master senec (the one that i register in the integration); do i need to register also slave battery and do total custom sensors?

3. Worst thing when i change sensors in energy panel with custom ones, I LOST all past data of about one year!!!!! sigh!!!!

anyway again great work, thank you!

[neubi4]

Hi @sigcarunchio

1.: yes, using Helpers and the yaml code is the same.

3.: i left the old sensors with their current VARIABLE_NOT_FOUND value in the energy dashboard and only added the new helper sensors. This way i still have the old stats data in my energy dashboard.
I dont know exactly how the dashboard works internally, it is possible that readding the old sensors will not work for you.

2.: This is an interesting one since not many users have multiple systems. Does the slave senec also has an seperate IP and the webinterface? If yes, then i would try to add the slave additionally to the master.
To get the total stat values of both systems, you could try to calculate the sum of the values from both senecs and calculate the Riemann sum integral on this value.

[sigcarunchio]

2. Yes, it has a separate IP with the same web interface, try this way!

3. Ok, i replaced them, try to reinsert also old sensors

[sigcarunchio]

Readded also old sensors, past stats availables again!
As soon as i will add also slave senec, I will report if it works

[electricdelicate]

added workaround information in the initial post

Good job, work also like a charm.
One question, is the "Riemann sum" exact enough for a proper calculation?
I will compare that to the Senec App.

[bobthebuilder4711]

I wrote a cautioning comment over on the old integration about Riemann sums: mchwalisz#47 (comment)

So here just a short summation: Riemann sums will start to differ from what the battery reports with time and a better solution would be to simply get that data from mein-senec.de.

mein-senec.de only updates every 5 minutes, but it's only for the totals, so that should be fine. The other values can still be queried from the lala.cgi if possible.

[bobthebuilder4711]

It also seems like the update reset the settings for our Home V2.1 to poll our three SMA inverters. Solar Power just stays at 0W. Anyone else has this problem?

Don't have the installer's password for the senec battery, so I guess I'll have to add the inverters to home assistant directly...

Edit: Okay, Solar Power is back. Seems like the battery needed an hour to work properly again.

[Kuechenfenster]

https://github.com/mchwalisz/home-assistant-senec/issues/47#issuecomment-1701014858

[marq24]

Please note, there was a typo in my provided configuration.yaml

in the senec_grid_total_export_fb the source has to be: source: sensor.senec_grid_exported_power (and not source: sensor.senec_senec_grid_exported_power) - sorry...

[marq24]

Since here (locally) I still have the http version - I am able to compare the values [created via the integration platform vs data from the senec statistics json data]

[Kuechenfenster]

@Kuechenfenster

mchwalisz#47 (comment)

could you let us know, what you wanted to say?

my apology.. please ignore it.

[electricdelicate]

added workaround information in the initial post

Good job, work also like a charm. One question, is the "Riemann sum" exact enough for a proper calculation? I will compare that to the Senec App.

As far as I can tell at my system, the Riemann sum is only correct at my charging value, the discharging value is very deviating to MySenec/official App ....

[marq24]

here is a comparison of the 'discharging value' here @ my home... YES the numbers are different...

Just as Update (working on a better replacement) - I have spend already some hours trying to access mein-senec.de - but failed with the aiohttp python classes to get an appropriate login (with the request-lib this works smooth like silk) - one of the many redirects is failing

[egonkab]

Hello,
the powercalc result from 09/01/2023

[marq24]

Passt über den dicken Daumen - oder? [It just took me a while to realize, that top part (of the image) is from SENEC-App and the lower part is from HA-Energy-Tab (I assume feeded by the 'integration' sensors...)

[egonkab]

Ich schwenke mal auf deutsch um. Die Werte aus dem HA Energy Tab sind die der PowerCalc Berechnung.
Ich habe am zum Vergleich folgende Sensoren aufgesetzt (senec_grid_exported_power), Ergebnis:

PowerCalc: 14,53kWh - RMS Trapez: 14,12kWh - RMS Left: 14,19kWh - Senec App: 14,5kWh

PowerCalc ist am nächsten dran, warum kann ich aber nicht sagen. Abfrage Intervall vom Speicher sind 2 Sekunden.

[BMeyendriesch]

Könntest Du mal bitte die powercalc.yaml für obige Ausgabe mitteilen.

[BMeyendriesch]

Hat sich erledigt. Ich hatte mir mit gleichzeitigen Konfigurationsversuchen per GUI und yaml selbst auf dem Fuß gestanden . . .

[marq24]

hier teilen wäre 'trotzdem' schön ;-) - Gute Morgen btw..

[marq24]

updated initial post -> added a solution alternative: mein-senec.de WebPortal Integration Entry

[BMeyendriesch]

Ich habe hier einen Haufen von Helpern und darauf aufbauenden Sankey-Diagrammen, die auf den ursprünglichen Entities der v2.1-Integration

sensor.senec_solar_total_generated
sensor.senec_house_total_consumption
sensor.senec_grid_total_import
sensor.senec_grid_total_export
sensor.senec_battery_total_charged
sensor.senec_battery_total_discharged

basieren und würde nur sehr ungern sämtliche Abhängigkeiten von diesen Entities anfassen und modifizieren. Gibt es nicht eine schlanke Methode, die Ausgangswerte von powercalc in diese Entities umzuleiten? Bisher habe ich powercalc nicht dazu überreden können. Möglicherweise habe ich auch das gesamte Konzept der Entities in Homeassistant nicht hinreichend verstanden . . .

[marq24]

For my V2.1 the portal updates ever 5 minutes. But I've been told that you can poll the portal much quicker if you have a V4. 30 seconds seems to still be okay. But I can't say if the portal updates even quicker than that.

I can confirm that you can poll data from mein-senec.de at least every 30 seconds if you have a V4. But I also have not tried shorter polling intervals, as 30s is absolutely fine for me. Maybe the V4 transmits its data more often "back home"? I might investigate that when I return from my vacation.

thanks! - I actually have already (this morning) adjusted the polling interval to 60sec if the system identifies itself as 'Home V4'... but this only takes effect, when you re-install the integration entry [since the interval is currently not part of the configuration]

[Thomk-63]

HI, I am now using the WebAPI for getting the statistics values. This seems to work and I can take these WebAPI sensors for the energy dashboard. BUT: All the energy dashboard history is then gone, as this is saved under the original senec entity IDs. Is there any way to write the WebAPI values to the senec entity IDs. Ideally this would be done directly in the integration by mapping the WebAPI to the unavailable statistics entity IDs. Unfortunately, I am not a software programmer. Any idea?

[bobthebuilder4711]

Seems like you can just delete the old sensors and then rename the new sensors to have the entity_ids of the old ones: https://community.home-assistant.io/t/migrate-lts-history-to-replacement-sensor/460115/3

And if one sensor is in kWh and the other is in Wh then use a template sensor for that as I described earlier in German or here in English: mchwalisz#47 (comment)

But definitely create a backup before you try any of this.

Edit:

I just tried this with the house consumption sensor because that is calculated by home assistant anyway and thus not important for me if it broke. If you rename the new sensor's entity_id to the entity_id of the old sensor then you do in fact have the old historical data on that new sensor.
But: If you've already gathered data points in the last few days then those aren't transferred over.

All the energy dashboard history is then gone, as this is saved under the original senec entity IDs.

To combat that just leave the old entities in the energy dashboard. I have both the old and the new sensors in there and it looks like this on the day of the update. It's yellow instead of orange from there on but that doesn't really bother me.

In the future someone with too much time on their hands might write some SQL code to migrate the long term data from the old entity to the new one. But I don't see me doing that in the next few weeks or even months.

[marq24]

> To combat that just leave the old entities in the energy dashboard. I have both the old and the new sensors in there and it looks like this on the day of the update. It's yellow instead of orange from there on but that doesn't really bother me. !

exactly! - as documented here - #11 - thanks @bobthebuilder4711 for your help here supporting the integration!

Once more I have edited my initial text of this issue in order to also inform you about the Energy Dashboard...

[SteffenS27]

Hi, sorry for confusing that technical discussion. Bit can someone please tell what I have to do with that beta version and so on ? Sorry bit HA is new for me

[marq24]

@SteffenS27 - have you read this? #14

[SteffenS27]

@marq24 thanks that should be what I need, but I have at the moment another problem, in HACS it is not possible to install or renew a Integration, the circle is just turning, but nothing happens

[marq24]

@SteffenS27 I hope it's obvious, that I can't provide HA support in general... I would do:

0. CREATE A BACKUP
1. login via ssh to get shell access
2. rename the custom_components folder in your config directory (e.g. to xustom_components
3. restart HA

this will disable all custom_components you might have installed - and you can re-enable them one by one to find the one causing the loop... [also checking the ha system logs might be already a way simpler start to find the root of your problem]...

IF it's the SENEC.Home integration - double check, if you have specified the correct credentials (WEBAPI) - If your system was http only before (and now https is required) - make sure that you remove the LaLa.cgi based integration - and reinstall it.

[Cyrelion]

@marq24,

Heute hab ich zufällig in einem der Facebook Posts gelesen dass eine neue Firmware ausgerollt wird welche die Probleme mit der aktuellen NPU-Error etc. lösen soll.

Dabei gibt es ebenfalls ein Absatz über unser Vars.html Problem, welches aber leider eher weniger zur Freude einlädt.

Anbei ein Screenshot der Nachricht aus der ich gerne mal Zitire:

Bisher konnten Endkunden über /display.html bestimmte Werte des Speichers auslesen, diese Werte finden sich in der App bzw. in mein-senec.de. Im Zuge der Optimierung des Speicherverbrauchs der Firmware setzen wir diese Funktion nicht fort.
Dies gilt auch für den direkten Zugriff auf Entwicklungsvarieablen der Systeme, die zuvor durch Endkunden und Fachpartner selbst ausgelesen werden konnten (/Vars.html). Im Zuge einer allgemeinen Erhöhung der Sicherheitsstandards der Systeme wird der Zugriff auf diese Entwicklungsinformationen auf SENEC selbst beschränkt."

Heißt im Klartext: Weil die Mainboards anscheinend zu wenig RAM haben und vermutlich die Firmware jetzt durch die SSL Implementierung mehr Speicher benötigt haben sie solche wichtigen Informationen wie Gesamtverbrauch etc. einfach wegoptimiert? Das kann doch nicht deren Ernst sein.

Evtl. kannst du ja zumindest die Funktionen der WebAPI und des Speicherzugriffs dann zusammenfassen und die WebAPI Daten dann in die fehlenden Speicherentities packen?

Gruß,
Andree

[bobthebuilder4711]

Evtl. kannst du ja zumindest die Funktionen der WebAPI und des Speicherzugriffs dann zusammenfassen und die WebAPI Daten dann in die fehlenden Speicherentities packen?

Mit einem V4 geht ja sowieso nur noch die Abfrage über mein-senec.de oder die App. Die reine WebAPI hat also auf jeden Fall ihre ganz eigenständige Daseinsberechtigung.

Und wenn mal eins der beiden nicht geht, ist auch die Fehlersuche einfacher.

[marq24]

Evtl. kannst du ja zumindest die Funktionen der WebAPI und des Speicherzugriffs dann zusammenfassen und die WebAPI Daten dann in die fehlenden Speicherentities packen?

In der Ankündigung von SENEC steht, das die Vars.html und die display.html abgekündigt wird [was natürlich an sich schon erbährmlich ist - aber so ist es eben - beides sind aber Funktionen, die ich persönlich bislang nicht verwendet habe]... Wie es mit den STATISTICS-Daten aussieht, werden wir dann alle merken, wenn das Update tatsächlich mal ausgerollt worden ist...

Und vor allem, was noch alles so zusätzlich mt dem Arsch umgeworfen wurde - bis dahin will ich aber erstmal abwarten...

Die WEB-API Felder aber in die Local-lala.cgi Implementierung zu integrieren halte ich im Augenblick aber noch nicht so zielführend - schon allein wegen der unterschiedlichen "polling-intervallen" und der Tatsache, das das eine mit dem anderen nicht wirklich viel gemeinsam haben...

... im Zuge einer allgemeinen Erhöhung der Sicherheitsstandards der Systeme...

Humor haben'se wenigstens!

[loeffelpan]

Hab auch gerade die Meldung in der App bekommen.
Klingt nach als kämen die Statistikdaten wieder. Aber mal sehen.

[Cyrelion]

Dann hoffen wir mal das beste das die Statistiken in die lala.cgi wieder kommen. Ansonsten hab ich jetzt alles erstmal auf die WebAPI umgestellt in meinen Dasbboads und für die Automatisierungen ob und wieviel Überschuss gerade da ist habe ich ja gott-sei-dank mit Discovergy direkten Zugriff auf die Daten des Stromzählers sodass ich hier sogar genauer und direkter als mit Senec arbeiten kann.

[bobthebuilder4711]

direkten Zugriff auf die Daten des Stromzählers

Dann noch einen Shelly 3EM (oder einen Emporia Vue Gen 2) an das Stromkabel vom Senec Speicher klemmen und Du brauchst die Daten theoretisch gar nicht mehr vom Speicher selber abzurufen.

[marq24]

... direkten Zugriff auf die Daten des Stromzählers ...

Seit ein paar Wochen habe ich einen Tibber Pulse im Keller - und heute habe ich mich mal hingesetzt und damit auseinandergesetzt, wie ich direkt an die Daten komme...

-> https://github.com/marq24/ha-tibber-pulse-local

[marq24]

Also after the update to 0826 - no STATISTIC data in the lala.cgi response... thanks for nothing SENEC!
-> so the two current known approached to address the issue are still valid

2023-10-05 07:31:00 [I|System Update|Firmware] Performing firmware update...
...
2023-10-05 07:33:19 [I|Settings] MCU hardware version: 1
2023-10-05 07:33:19 [I|Settings] MCU firmware version: 0826, GUI version 0966

[Cyrelion]

I still don’t think that it will come back at all 😭

[marq24]

Till today it did not came to my mind, that the data displayed in the mein-senec.de portal could be inaccurate - but please note, that at least in my setup/environment the grid-import data is not-correct. There is a difference of ~9% between the physically electricity meter reading and the data in the mein-senec.de portal - Details can be found here in this discussion I just have started #37 -> let's discuss details there... TIA

[marq24]

as little update - even with the rollout of v0827 - no local STATISTIC data... but //vars is still working - so wegen' der Sicherheit & so...