Pow-P1: Day History Missing

[nossaile]

Day history missing, only current day shown, se image below:

Hardware information:

• Country: Sweden
• AMS reader: Pow-P1

Relevant firmware information:

• Version: 2.2.28 (Ithink the problem was intruduced in v2.2.27)

[ArnieO]

So you lost the history during upgrade?

[nossaile]

No, I updated to v2.2.28 just after it was released and as you see above only yesterday's day value is presented.

[ArnieO]

OK, I see.
v2.2.28 was released in December, so my understanding is then that you upgraded to v2.2.28 some time in December.

I'm trying to understand what has happened:
Has the "Monthly" graph shown only previous day since you upgraded, or was it today you lost the historical values?

[nossaile]

Sorry, for not being able to be more precise in my observations. My Pow-P1 is connected to my Home Assistant system and I only look at its web-interface occasionally. The communication with the meter and my HA-system works perfectly.

As I wrote above I think it was introduced with the 2.2.27 update. My reason for not reporting it earlier was that it’s such an obvious fault, but yesterday when I again observed it and I read through all error reports without finding anything about it, I thought it was about time.

To answer your question: I have observed that part of the history is missing several times and sometimes the hourly values too. Today it looks like follows:

The value for January 17th is not correct as you can see in the HA-presentation below:

[nossaile]

Correct value is 99,47kWh and not 61kWh as presented by Pow-P1s web-ifc

[ArnieO]

Interesting.

There is a known issue with a 1-hour time shift to HA from smart meters used in Norway and Sweden that follow the HAN-NVE standard for payloads. The reason for this is that those payloads only send accumulated energy (kWh) once per hour, a few seconds after whole hour. So when that measurement arrives in HA, it will be time stamped with the incoming time, which is the hour after the measurement was done. This has proven to be quite difficult to circumvent for us, as it is HA that time stamps the message when it arrives. You will find other threads here on the subject.

Since your device is a Pow-P1, you should not be affected by this, as P1 meters report accumulated energy (kWh) in each payload.

So your observation is something quite different, and I am really not sure what is going on here.
It could look as if the accumulated value for 17th Jan is registered in HA on the next day (except that in that case it should have been 99 in amsreader GUI, not 98; rounding issue?)
Maybe @gskjold has an idea?

[nossaile]

No no, sorry, that was a typo from me in my last comment. Corrected: "Correct value is 98,47kWh and not 61kWh as presented by Pow-P1s web-ifc"

With that comment I wanted to highlight that in Pow-P1s several hour values were missing when the day value was calculated.

I confirm what you stated: I have no 1-hour time shift problem. HA is in sync with Pow-P1 there and the history i HA is correct and complete.

[ArnieO]

What I was referring to is this value, indicated for 18th Jan:

It is almost the correct value for 17th Jan according to HA.

What value does HA give for 16th Jan? Is it around 61 kWh (shown in GUI for 17th Jan)?

[nossaile]

Aha... No, reading are as follows:
16/1: Pow-P1 = 00.00kWh and HA = 160.57kWh
17/1: Pow-P1 = 61,27kWh and HA = 98.47kWh
18/1: Pow-P1 = 99.23kWh and HA = 99.23kWh

The 16th was very cold for our part of Sweden, below -10 degrees. Could the problem be temperature related or related to the huge energy value?

[nossaile]

As long as I have had Pow-P1 together with HA the recorded monthly energy value has been exactly the same as on my monthly energy bill.

[ArnieO]

The 16th was very cold for our part of Sweden, below -10 degrees. Could the problem be temperature related or related to the huge energy value?

No, that sounds unlikely. This is digital; it either works or it does not. We have had reported a Pow-K in Sweden that stopped working below -25°C. It worked at -23, did not work at -29. We don't know what caused that, and do not have access to test facilities that could help us find out. The suspects are the supercap or timing issues in the ESP module (although both are specified to -40°C).

Your issue sounds more like something related to missing readings - for some reason. But that is more likely to be an issue with meters configured as HAN-NVE, as they only send kWh once per hour. The P1 meters send kWh in each payload.

Can you check in on the GUI from time to time and see if you get any error messages (parity error or checksum error flagged in red on the top of the screen)? And also please follow-up how that Month graph evolves, compared to what you see in HA.

[gskjold]

Interesting stuff. How is your energy dashboard configured in HA?

[nossaile]

This is how my energy dashboard is configured in HA:

and

[ArnieO]

Interesting stuff. How is your energy dashboard configured in HA?

My understanding is that things are fine in HA, it is the firmware GUI that is wrong (or did I misunderstand, @nossaile ?).

[nossaile]

That is correct, everything looks fine in HA, but the history shown by the GUI from Pow-P1 loose values both in day- and month- history.

When I entered the GUI yesterday an error message shown i red appeared for a second or two. I think it was something like "HAN FOOTER CHECKSUM ERROR"

[ArnieO]

When I entered the GUI yesterday an error message shown i red appeared for a second or two. I think it was something like "HAN FOOTER CHECKSUM ERROR"

OK, so there seems to be an issue here (although I don't yet understand how that could give the discrepancy HA/GUI).

• How long is the cable between meter and Pow-P1? Should not be longer than 3 meters.
• What is your buffer size? If it is 256, increase it to at least 512.
  
• Please run a Telnet debug as explained here. If you see error messages, please post the result of the corresponding section.

[nossaile]

The cable is 3m. The buffer size 384:

I have now increased it to 512
I will run a telnet debug and be back with the result

[ArnieO]

If the buffer increase did not help: Can you do a test with shorter cable?

The bitrate on these meters is quite high, so your cable length might be an issue.
Aidon says maximum 3 meters in the specification for their meters, I have not seen that parameter mentioned anywhere for other meters. And there are probably some variations in the hardware various manufacturers use - so it is worth while to test with a short cable to see if it makes a difference.

[nossaile]

Maybe this is a clue:

As you can se Pow-P1 restarted (by it self) 3minutes ago and I happened to look at GUI. The value showing "Consumption hour" was reset by the restart! An other observation is that the "Cost Month" is way too high (I think it has been too high at least since the history was lost: Either 331kWh0.89SEK=295SEK or with correct consumption 2666kWh0.89SEK=2383SEK. I do not use any price calculations in Pow-P1 more than the spot price.

This time no history was lost:

[nossaile]

"Consumption Last Month" above is also wrong. Correct value for December is 2353kWh.

[ArnieO]

@gskjold : What do you get out of this?

[nossaile]

I need to correct myself on my statement “This time no history was lost” Jan26th above: As you can see day values from 16th to 20th are missing and the value for the 21st is wrong (should be 81kWh). This could of cause be the result of an additional reboot the 21st also.

You speculated above that the problem could be “something related to missing readings”. I have not observed any missed readings from the power meter in HA’s presentation of online or history values, My Pow-P1 does an excellent job there. It’s only Pow-P1’s own history calculations and storage that is affected. I think it is important to note that the problem is not only limited to history values being missing: some values are wrong both too low and too high as I reported above.

I’m not sure but I think the problem is connected to spontaneous reboot of Pow-P1. This happens from time to time but the history is not affected each time. I have not observed any faulty history values after planned reboot, like after a firmware upgrade.

I do not understand why my Pow-P1 reboots spontaneously. Does this give any clue?

[gskjold]

"Consumption Last Month" above is also wrong. Correct value for December is 2353kWh.

If you loose data, this is to be expected. Consumption for last month is stored when passing into a new month, and is based on consumption from the graph data. When the graph is not complete, the number will be wrong.

I’m not sure but I think the problem is connected to spontaneous reboot of Pow-P1.

This sounds plausible, and in any case we should identify why this happens before drawing any conclusions.

• How is your RSSI? (Found in header)
• What is your power saving settings? (Found on config page)
• Photo of how you device is placed. Also, is it inside a metal cabinet or outside the house or far otherwise far away from accesspoint

[nossaile]

RSSI:

Power saving settings:

I have run with Power Savings = Off and Power =19.5dBm until two weeks ago.

Before it was removed I had "Auto reboot on connection problem" unchecked, but it rebooted spontaneous anyway and everytime I rebooted the wifi router.

My Pow-P1 unit is placed about 1m above the meter cabinet in a water resistant plastic cabinet about 5m and two walls from the closest mesh node. It is placed outside about 2m above ground under a roof where it is protected from wind, rain, and snow.

[nossaile]

Pow-P1 reboots for three reasons:

1. Planned: After an upgrade or changed setting.
2. When wifi connection is lost. (My wifi router is setup to reboot every Monday at 04:44)
3. Spontaneous (unknown reason)
   It is only after spontaneous reboot I have observed the history values to be affected.

[ArnieO]

RSSI around -60 dBm should not be an issue, and your placement seems fine.

3. Spontaneous (unknown reason)
It is only after spontaneous reboot I have observed the history values to be affected.

This is new and interesting information. I can't remember having user report something like this before.
So I'm starting to lean towards this being due to a weirdly faulty ESP32 module on your board.

Please contact me on post@amsleser.no to arrange shipment of a replacement board to you.

[nossaile]

Just want to explain my silence. It’s because I test my Pow-P1 with the 30cm RJ12-cable provided by amsreader.no. Unfortunately, this forces it to be placed inside the meter cabinet (made of steel) and as a result the RSSI drops to around -80dBm. So far it looks stabile. I will keep you updated.

In the meantime I have a small request of improvement connected to this: Add information in the payload about last boot or just a running number increased by one every boot to enable HA to log it and take actions like set an alarm.

[ArnieO]

OK, I see. Please take care to position it on the front of the meter, using the enclosed ziplock pad.
Especially, do not tuck it in to a corner of the cabinet where the antenna could be pointing into a corner.

The signal strength is surprisingly little attenuated by most indoor steel cabinets. Outdoor cabinets is a different story, they give far more damping. -80 dBm usually works, but is "limit". Maybe you can buy a longer cable? It is a standard 1:1 RJ12 cable. Cable should not be longer than 3 meters.

[stigvi]

In the meantime I have a small request of improvement connected to this: Add information in the payload about last boot or just a running number increased by one every boot to enable HA to log it and take actions like set an alarm.

You could make a binary template sensor and check for uptime less than 120. Then you will have something indicating a boot.

{{ int(states('sensor.ams_uptime')) < 120 }}

[nossaile]

@ArnieO: It’s an outdoor cabinet but inside our bicycle storage. I will try to align my Pow-P1 for best possible RSSI inside the cabinet.

@stigvi: Good idea, but ams_uptime is not part of the payload, I’m afraid.

[stigvi]

@ArnieO: It’s an outdoor cabinet but inside our bicycle storage. I will try to align my Pow-P1 for best possible RSSI inside the cabinet.

@stigvi: Good idea, but ams_uptime is not part of the payload, I’m afraid.

Ams_uptime is my sensor name. Uptime is available in raw.

[nossaile]

@stigvi: Thanks, I found a variable called "up" with uptime in seconds. For some reason not automatically visible in HA GUI.

[andreas1974]

Hi!

I've lost the contents of the Energy use last 31 days (kWh) graph a few times too, lately. At least two times the last two weeks. I had some version below 2.2.27 when I upgraded to 2.2.28, and don't recall I've seen this behaviour before.

The disappearance of the 31 days history seems to coincide with the last reboot. The reason for reboot is also unknown to me.
Last boot: 26.02.2024 06:18
Reason: Vbat power on reset (1/0)

Device information
Chip: esp32s2 (160MHz)
Device: [Pow-P1]
Last boot: 26.02.2024 06:18
Reason: Vbat power on reset (1/0)
Meter
Manufacturer: Unknown
Model: U18SEH13F10030
Firmware
Installed version: v2.2.28
Latest version: v2.2.28

[andreas1974]

I try to read the previous messages, but don't quite understand what your thoughts are about this issue.

My energy use for the preceding month ("29 days") was reset again at March 3rd.
February 26 (as seen above on my previous message) until March 3rd was lost.

[nossaile]

@andreas1974: Don’t know if your question was for me, but it looks like we have the same kind of problem. As I wrote on Jan29 my Pow-P1 reboots for three reasons:

1. Planned: After an upgrade or changed setting.
2. When wifi connection is lost. (My wifi router is setup to reboot every Monday at 04:44)
3. Spontaneous (unknown reason)

It’s only after spontaneous reboots that I have observed the history values to be affected (month history partly reset and some calculated values are faulty (see my Jan26 comment above)). My Pow-P1 reboots spontaneous about 3-10 times per week, but the history is only affected once every two to three months (last time was in mid January). It’s no big deal to me because it’s connected to my Home Assistant system and the payloads are not affected.

I have an unverified feeling that the problem is somehow temperature related. It looks to happen only when the outside temperature is below +5ͦC. Additionally; I think the trigger for spontaneous reboot could be due to poor connection in RJ12-connector on the board.

[ArnieO]

We have been trying to find a cause for this, but are really struggling. As far as we can see there are two reports, both are Pow-P1 users - but that can be a coincidence. And you are "only" 2-3 persons that have reported this, out of around 2500 users. So it is not common.

Our plan now is to first release v2.3 - and then wait for your updates whether the issue is still there, as there are quite a number of changes vs v2.2.x.

If the problem then persists for you 2-3 persons, we want to replace the hardware for the one that first reports the issue with v2.3.

[SVH-Powel]

And you are "only" 2-3 persons that have reported this, out of around 2500 users. So it is not common.

You have to open the ams web page daily if you are going to see this. The reported values on {publish topic}/realtime/import/hour and {publish topic}/realtime/import/day are not affected much (or in a obvious way) about this.

So out of 2500 users, my guess a lot more than 2-3 persons have this issue without knowing it.

[SVH-Powel]

The code below .....

float EnergyAccounting::getUseToday() {
if(tz == NULL) return 0.0;
float ret = 0.0;
time_t now = time(nullptr);
if(now < FirmwareVersion::BuildEpoch) return 0.0;
tmElements_t utc, local;
breakTime(tz->toLocal(now), local);
for(uint8_t i = 0; i < this->realtimeData->currentHour; i++) {
breakTime(now - ((local.Hour - i) * 3600), utc);
ret += ds->getHourImport(utc.Hour) / 1000.0;
}
return ret + getUseThisHour();
}

.... should instead had a return like "return ret + getUseSinceLastHourImport();"

As it is working in the current firmware, /realtime/import/day decrement a little when an import is missing for an hour. A kwh counter for the day should never be less than it was during that day.

[ArnieO]

So out of 2500 users, my guess a lot more than 2-3 persons have this issue without knowing it.

You are probably right, and I can assure you we would very much like to get to the bottom of this!

The code below .....

Thank you for the concrete proposition, @gskjold will look into it and consider testing an adjustment.

[gskjold]

You would still experience a decrement in that sensor, as "use this hour" is a calculated value and can deviate from the actual value. Creating a method for "getUseSinceLastHourImport" would still be calculated, and will differ from actual use over time.

[SVH-Powel]

Yes, I agree on that. The calculated/estimated value can be higher than the actual value.

A decreasing value is in fact a little troublesome for HA users:

https://developers.home-assistant.io/blog/2021/08/16/state_class_total/

"For sensors with state_class STATE_CLASS_TOTAL_INCREASING, a decreasing value is interpreted as the start of a new meter cycle or the replacement of the meter. It is important that the integration ensures that the value cannot erroneously decrease in the case of calculating a value from a sensor with measurement noise present. This state class is useful for gas meters, electricity meters, water meters etc."

Decreasing values mess things up, but I am not sure at the moment how large impact this has. Anyway, it will only be a problem for those users which is using energy accounting in HA.

[SVH-Powel]

And I am not using that value in my HA because of this. I have implemented my own stuff. Which is not what I wanted, because amsreader is so close of getting this right.

Edit: My solution is to ignore values that is less than previous value (except when it is expected on a new cycle for a new day). The integration error is so small that it only takes a few seconds or a minute before the increasing value as become larger than the integration error.

[gskjold]

Some thoughts on code to deal with this comes to mind now, I will take notes of this for a future change

[gskjold]

I have a new development build in #738 that may fix this. If anyone could try it, that would be great.

[gskjold]

For anyone who have upgraded to 2.3, have you experience this since?

[nossaile]

Okay, so far. But the real time plot zeroed values before reboot, see below. No values missing in the day and month history, and pay loads to HA.

[ArnieO]

the real time plot zeroed values before reboot, see below.

Values in the real time plot are not stored, only held in working memory, so this is the expected behavior.

[nossaile]

Okay, as expected then.

[nossaile]

From my point of view: This issue can be closed. I have not experienced any history failure since January and my PowP1 is working more stabile (a lot fewer reboots) since the v2.3ff updates.