Resubscribe real-time and cumulative usage after connection outage

[JacobWasFramed]

Currently running the beta with real-time energy usage. I notice that after an internet connection time out, the real-time usage doesn’t resubscribe when connection is restored. The cumulative usage only seems to fail if the outage happens during a reading due to an internet or Duke API outage longer than the 15 minute interval. Seems like any outage requires a restart of HA to restore. I’m unsure if this can be addressed here or should be address in the Duke Python repository.

I think a good solution would be to poll every 1-5 minutes after a connection is out until a successful response is received and at that point resubscribe or reinitialize the integration.

Potentially related, being unable to restart the integration without having to restart HA.

[mjmeli]

Thanks for bringing this up. When this happened, did you see the integration show a failure in the Integrations page? (it would show with a red outline and an error message).

[JacobWasFramed]

Honestly can’t recall. I “introduced” a network failure just now and the plug-in did not crash. The cumulative usage was able to resume, but real-time MQTT did not resubscribe.

The errors that I have seen are different than the self-introduced network failure.

The original errors I saw when posting this were as follows:

Error: (update_coordinator.py)
Error fetching homeassistant data: Error
communicating with Duke Energy Usage API: Request
failed with unexpected error [https://cust-api.duke-
energy.com/gep/v2/auth/oauth2/token]: 400,
message='Bad Request', ur|=URL('https: //cust-
api.duke-energy.com/gep/v2/auth/oauth2/token')

Warning: (realtime.py)
Error requesting smartmeter auth, will retry after 5
seconds.

Warning: (realtime.py)
Unexpected message:
<paho.mqtt.client. MQTTMessage object at
0x7fe757d69c80>
Unexpected message:
<paho.mqtt.client. MQTTMessage object at
0x7fe74f6274a0>

What I’ve just experienced from a self-introduced network failure is as follows:

Error: (update_coordinator.py)
Error fetching homeassistant data: Error communicating with Duke Energy Usage API: Request failed with unexpected error [https://app-core1.de-iot.io/rest/cloud/smartmeter/usageByHour]: 504, message='Gateway Time-out', url=URL('https://app-core1.de-iot.io/rest/cloud/smartmeter/usageByHour?startHourDt=2021-11-20T05:00&endHourDt=2021-11-21T05:00')
Error fetching homeassistant data: Error communicating with Duke Energy Usage API: Request failed with unexpected error [https://app-core1.de-iot.io/rest/cloud/smartmeter/usageByHour]: Cannot connect to host app-core1.de-iot.io:443 ssl:default [Network unreachable]

Error: (realtime.py)
MQTT disconnect error, result code: Out of memory. (This may not be accurate)

Both ways though, real-time MQTT doesn’t resubscribe, whether api or internet outage and I have had it where the cumulative was not updating either before restarting HA. I know Duke had an 1-2week outage at the end of October/beginning of November, but this was after that time. Thanks for looking into this; I appreciate your time. If I see an outage again, I’ll make sure to update here with logs.

[mjmeli]

I had a network outage over the weekend so I did experience this myself. The MQTT stuff does have reconnect logic, but we have to make a separate API request to Duke to initiate that MQTT streams, and I think that API request is the one causing the issues as it does not re-attempt after some period of time. I'll look into it when I have some time.

[mjmeli]

I believe this should be fixed by the updated pyduke-energy version in v0.1.0b9 (#53). in the event of a connection loss, the error will be caught and we will attempt to reconnect indefinitely with an exponential back-off strategy (starts at 1 minute, maxes out at 60 minutes). the times can be adjusted if wanted later.

a warning log will be logged with the back-off time when this happens.

I tested this by cutting network to the docker container and it worked for that case. however, depending on where and when the internet drops, a different exception may be thrown. I believe I've caught all of them but there is a chance I missed some. any that I missed will be logged as an error log and we will not retry. please open a issue on that instance.