Communication error with E3DC: Max retries reached #124

Phoenix-DH · 2024-03-20T05:13:39Z

System Health details

System Information

version	core-2024.3.1
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.12.2
os_name	Linux
os_version	6.1.73-haos-raspi
arch	aarch64
timezone	Europe/Berlin
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.34.0
Stage	running
Available Repositories	1409
Downloaded Repositories	35

Home Assistant Cloud

logged_in	false
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 12.1
update_channel	stable
supervisor_version	supervisor-2024.03.0
agent_version	1.6.0
docker_version	24.0.7
disk_total	439.4 GB
disk_used	45.8 GB
healthy	true
supported	true
board	rpi4-64
supervisor_api	ok
version_api	ok
installed_addons	deCONZ (6.23.0), UniFi Network Application (3.0.3), Terminal & SSH (9.10.0), Studio Code Server (5.15.0), Samba share (12.3.1), InfluxDB (5.0.0), Grafana (9.2.0), Glances (0.21.0), OneDrive Backup (2.3.1), AdGuard Home (5.0.4), Let's Encrypt (5.0.15)

Dashboards

dashboards	4
resources	23
views	14
mode	storage

Recorder

oldest_recorder_run	March 13, 2024 at 5:59 AM
current_recorder_run	March 18, 2024 at 1:13 PM
estimated_db_size	1099.65 MiB
database_engine	sqlite
database_version	3.44.2

Sonoff

version	3.6.0 (8dd8af9)
cloud_online	2 / 2
local_online	2 / 2

Checklist

I have enabled debug logging for my installation.
I have filled out the issue template to the best of my ability.
This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
This issue is not a duplicate issue of currently previous issues..

Describe the issue

I have againt the following message on my system:

This error originated from a custom integration.

Logger: custom_components.e3dc_rscp.e3dc_proxy
Source: custom_components/e3dc_rscp/e3dc_proxy.py:29
integration: E3DC Remote Storage Control Protocol (Git) (documentation, issues)
First occurred: 12:00:49 AM (20 occurrences)
Last logged: 12:04:00 AM

Communication error with E3DC: Max retries reached
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc.py", line 226, in sendRequest
result = self.rscp.sendRequest(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc_rscp_local.py", line 117, in sendRequest
raise CommunicationError(receive[2])
e3dc._e3dc_rscp_local.CommunicationError: UNKNOWN

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/config/custom_components/e3dc_rscp/e3dc_proxy.py", line 29, in wrapper_handle_e3dc_ex
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/e3dc_rscp/e3dc_proxy.py", line 116, in get_db_data
return self.e3dc.get_db_data_timestamp(timestamp, timespan_seconds, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc.py", line 710, in get_db_data_timestamp
response = self.sendRequest(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc.py", line 237, in sendRequest
raise SendError("Max retries reached")
e3dc._e3dc.SendError: Max retries reached

and

This error originated from a custom integration.

Logger: custom_components.e3dc_rscp.coordinator
Source: helpers/update_coordinator.py:318
integration: E3DC Remote Storage Control Protocol (Git) (documentation, issues)
First occurred: 12:00:49 AM (20 occurrences)
Last logged: 12:04:00 AM

Unexpected error fetching e3dc_rscp data: Communication Failure: Failed to send data
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc.py", line 226, in sendRequest
result = self.rscp.sendRequest(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc_rscp_local.py", line 117, in sendRequest
raise CommunicationError(receive[2])
e3dc._e3dc_rscp_local.CommunicationError: UNKNOWN

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/config/custom_components/e3dc_rscp/e3dc_proxy.py", line 29, in wrapper_handle_e3dc_ex
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/e3dc_rscp/e3dc_proxy.py", line 116, in get_db_data
return self.e3dc.get_db_data_timestamp(timestamp, timespan_seconds, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc.py", line 710, in get_db_data_timestamp
response = self.sendRequest(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/e3dc/_e3dc.py", line 237, in sendRequest
raise SendError("Max retries reached")
e3dc._e3dc.SendError: Max retries reached

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 318, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/e3dc_rscp/coordinator.py", line 169, in _async_update_data
db_data_today: dict[str, Any] = await self.hass.async_add_executor_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/e3dc_rscp/e3dc_proxy.py", line 37, in wrapper_handle_e3dc_ex
raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Communication Failure: Failed to send data

I am not pretty sure if we are talking about the same than earlier in #75

Wan't to double check if this is a new one. To be honest, debugging is currently still off.

Reproduction steps

Having a look to the logs

Debug logs

Currently still off

Diagnostics dump

No response

The text was updated successfully, but these errors were encountered:

torbennehmer · 2024-03-22T16:34:57Z

I'll have to forward this to https://github.com/fsantini/python-e3dc, unfortunately, I can't say how long a resolution will take as this seems rather strange. I'll cc you there.

Waiting for fsantini/python-e3dc#115

Phoenix-DH · 2024-03-22T16:43:56Z

@torbennehmer Great!

torbennehmer · 2024-03-22T16:44:02Z

@Phoenix-DH How often do you see this error? What side-effects does it have? Can you download the diagnostics dump from e3dc? Can you please enable debug logging and send me the next error that occures along with the information immediately before it? hacs-e3dc did pull the daily stats at that point and I don't understand what's going wrong.

Phoenix-DH · 2024-03-22T18:14:45Z

Bad thing is that my HA has just been restarted. Log gone. But if I got it right in my mind, both only one or twice a day. As it seems all 12 hours short past 12, always the same.

Will try to gather the debug logs. Enabling only for this integration right?

And the data directly from my e3dc? Is that save? Had to take a look how.

thijsdewitt · 2024-03-22T20:27:35Z

I'm seeing the same error on my system, about once every minute. I'm not sure what the side effects are.
Here is my log:

home-assistant_e3dc_rscp_2024-03-17T20-50-27.085Z.log

Pe2023Ro · 2024-03-23T09:20:12Z

Good Morning,
Same at my System.
But only at S10x (Master of farm). The additional Inverter S10SE works well.
I‘m not in the latest state of HA Core (2023.11.1) and also E3DC Integration is not the latest (3.4.3)
Could that be a reason?
No Side effects so far.

EDIT:

HA updated to Core 2024.3.3
E3DC integration updated to 3.6.2
Restart

Now it works again

Phoenix-DH · 2024-03-24T09:57:05Z

I still have the issue.
This tine 00:04 with no other issues or side effects.
During the week I will enable debugging if still needed with that already uploaded obe.

Harden e3dc data polling in coordinator to execptions: - Log ex only as warning, but continoue as good as possible. This will avoid breaking the entire update loop if one update fails (db_data does that right now). - Listen only on HomeAssistant-Type errors, not on ConfigEntryAuthFailed, let the latter propagate. - Reduce proxy error log level to debug, let upstream do the logging. Affects the logging errors we see in #113, #124

Phoenix-DH · 2024-03-29T09:09:34Z

Just installed, thanks.
Will provide feedback.

torbennehmer · 2024-03-29T09:18:13Z

@thijsdewitt

I'm seeing the same error on my system, about once every minute. I'm not sure what the side effects are. Here is my log:

home-assistant_e3dc_rscp_2024-03-17T20-50-27.085Z.log

As far as I understand the log, the E3DC was temporarily unreachable. I too see this from time to time, probably when E3DC is updating (or whatever). Also, there seem to be problems directly at midnight with DB Data Polling (see also #113). 3.6.3 beta-1 introduces a better error handling for testing (if you're willing to). It should no longer propagate communication error exceptions out of the update loop, instead you'll see simpler warnings stating that we can't talk to E3DC at the moment.

Be aware, that this is related to E3DC being unreachable, not the integration misbehaving. At least as far as I understand it at the moment.

The Tag Errors we've seen (unknown tag in manual charging data polling) something else, which I track on #114.

Please check if 3.6.3 solves your problem given these provisions - either use the beta or wait for stable.

@Pe2023Ro

E3DC integration updated to 3.6.2

You've probably been affected by #114 as well, 3.6.1 handles this gracefully until we get the communication working again.

@Phoenix-DH

This tine 00:04 with no other issues or side effects.

I believe you're seeing the same nightly outage in DB Data telemetry, so 3.6.3 should take care of the hard errors as well.

For all of you, this currently means, that on midnight the statistics are a few minutes off, you will still see warnings, but that is to be expected on an E3DC outage. If you still experience funny things beyond that on midnight, please follow up on #130. Right now, We can't do much about it, as E3DC isn't answering our queries at that time for whatever reason. Nothing I can fix, just work around.

Thus, I'll close this issue for now. Please reopen if the graceful error handling doesn't work for you.

ubarni · 2024-04-02T11:34:56Z

I just got the same error and found this thread.
In my case (after checking the last answer) i found out that my Unifi Flex Switch went offline and thats why the network connection was broken to my E3DC.
So if you get this error? Check if the world symbol on the E3DC display is GREEN and not RED.

Phoenix-DH · 2024-04-02T12:12:33Z

That would mean your switch will loose connection each night?
Can‘t believe that. But I have also a Unify switch.

Will see if there is something visible on the controller.

edit:
Just had a look tonthe audit log where I see the reconnects. Nothing …

ubarni · 2024-04-02T12:30:20Z

That would mean your switch will loose connection each night? Can‘t believe that. But I have also a Unify switch.

Will see if there is something visible on the controller.

edit: Just had a look tonthe audit log where I see the reconnects. Nothing …

No @Phoenix-DH, in my case it was a random outage but with the same log entries.
And @torbennehmer is right saying: "the E3DC was temporarily unreachable"

When is saw Communication error with E3DC: Max retries reached i first thought i had send to many requests to my E3DC?
But glad to see that it is a "simple" connection problem.

Phoenix-DH added the bug Something isn't working label Mar 20, 2024

torbennehmer added the dependencies Pull requests that update a dependency file label Mar 22, 2024

torbennehmer self-assigned this Mar 22, 2024

torbennehmer mentioned this issue Mar 22, 2024

e3dc._e3dc_rscp_local.CommunicationError: UNKNOWN fsantini/python-e3dc#115

Open

torbennehmer added the e3dc E3DC Bug label Mar 22, 2024

This comment was marked as off-topic.

Sign in to view

torbennehmer mentioned this issue Mar 29, 2024

Harden data polling against exceptions #129

Merged

torbennehmer mentioned this issue Mar 29, 2024

History data polling (today metrics) throw errors at midnight #130

Open

4 tasks

torbennehmer closed this as completed Mar 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Communication error with E3DC: Max retries reached #124

Communication error with E3DC: Max retries reached #124

Phoenix-DH commented Mar 20, 2024 •

edited

torbennehmer commented Mar 22, 2024

Phoenix-DH commented Mar 22, 2024

torbennehmer commented Mar 22, 2024 •

edited

Phoenix-DH commented Mar 22, 2024

thijsdewitt commented Mar 22, 2024

Pe2023Ro commented Mar 23, 2024 •

edited

Phoenix-DH commented Mar 24, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

Phoenix-DH commented Mar 29, 2024

torbennehmer commented Mar 29, 2024

ubarni commented Apr 2, 2024

Phoenix-DH commented Apr 2, 2024 •

edited

ubarni commented Apr 2, 2024

Communication error with E3DC: Max retries reached #124

Communication error with E3DC: Max retries reached #124

Comments

Phoenix-DH commented Mar 20, 2024 • edited

System Health details

System Information

Checklist

Describe the issue

Reproduction steps

Debug logs

Diagnostics dump

torbennehmer commented Mar 22, 2024

Phoenix-DH commented Mar 22, 2024

torbennehmer commented Mar 22, 2024 • edited

Phoenix-DH commented Mar 22, 2024

thijsdewitt commented Mar 22, 2024

Pe2023Ro commented Mar 23, 2024 • edited

Phoenix-DH commented Mar 24, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

Phoenix-DH commented Mar 29, 2024

torbennehmer commented Mar 29, 2024

ubarni commented Apr 2, 2024

Phoenix-DH commented Apr 2, 2024 • edited

ubarni commented Apr 2, 2024

Phoenix-DH commented Mar 20, 2024 •

edited

torbennehmer commented Mar 22, 2024 •

edited

Pe2023Ro commented Mar 23, 2024 •

edited

Phoenix-DH commented Apr 2, 2024 •

edited