-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wemo plugs keep reconnecting to the network #275
Comments
I've seen similar issue described here: https://github.com/home-assistant/core/issues/47965
My wemo plugs don't try to connect to my dockerized HA on port 8989. I've made a connectivity test from other device in the same VLAN as wemos, and connection can be established. Maybe passed CALLBACK URL is incorrect? |
Thanks for troubleshooting this further! I agree. Maybe the CALLBACK URL is incorrect for the subnet, or possibly the WeMo devices will not initiate those subscription requests when the CALLBACK URL is on a different subnet from the device. |
What can be done then? Is there any easy way for me to check the passed url? I can live without subscriptions, simple local polling is enough. But those frequent disconnections disrupt my automations. It also makes voice controlling via Google Assistant very unpredictable. |
Another example of errors I get:
|
In Home Assistant, the subscriptions impact the frequency of the local polling. As long as the subscription is active then local polling doesn't happen. This was a feature added in 2021.3.0 to help improve connectivity for devices that don't have a solid WiFI connection (home-assistant/core#45904). I don't believe there is anything more we can do in pyWeMo to improve the WiFi connection. I did notice when debugging the above issues that home-assistant/core#13106 made the polling interval more aggressive (the default is 30 sec and that PR made it 10 sec). home-assistant/core#45904 (comment) I haven't tried this personally, but it may be possible to configure the scan/poll interval. See https://www.home-assistant.io/docs/configuration/platform_options/ Assuming you don't have any automations that would need to see the state of the device when it was controlled locally (on the device itself), I'd try bumping the scan interval up above 2 minutes. (Just saw your latest post. Those HTTPNotOkException messages are odd. Especially the 404 status.) |
When did these error messages begin (all of them, not just the HTTPNotOkException)? Was there a particular version of Home Assistant where they started happening and a version where they weren't happening? |
Just as a debugging measure, can you ping the devices for several minutes. Is there any packet loss? |
As I said at the beginning my HA makes all wemos plugs disconnecting frequently from the network. I ping them every minute using Zabbix and I can see plenty of packet loss. This doesn't happen when HA is off. Those issues started around month ago. I do not upgrade my HA frequently, it's possible I've upgraded quite old version that time. Unfortunately I cannot be more precise because similar time I've also upgraded the fw on wemos. |
Ah! Thanks for clarifying. I think the increased network traffic from Home Assistant probably makes the connection worse. I can look more after work tonight. For now, try to adjust the scan interval. And also configure the devices statically and disable discovery. Discovery causes a lot of network traffic and happens every 5 minutes (more frequently at startup). Let me know if either of those help. |
All those plugs are defined as static because of their separate vlan subnet. Discovery on wemo platform is already disabled. Just started playing with scan interval, I'll let you know about results soon. Thx for your help. |
I've added
because scan_interval was not accepted directly in wemo section. It didn't help. Honestly I doubt it has changed anything because state of wemos when triggered from the official app are reflected in the HA in just 10 seconds. |
I've taken quite old 1.0.0b6 version of wemo and installed it for testing as a custom component. It didn't solve my issues. Errors in the log are not exactly the same but quite similar and my wemos still disconnect from the network when HA is running. As a last resort I've enabled wemo integration on very old HA 0.96.5 installed on Pi in the same vlan as wemos. This time everything was working as a charm with immediate wemo status updates and without any disconnections. I guess something has changed in the latest firmware which makes current way of communication from HA to wemos not possible, and forces those devices to reconnect a network. Maybe this is due to different subnets. Probably there is no way to downgrade the fw so I'm doomed now. |
I've made some other test and now I'm pretty sure that my wemos will not accept any subscription request with the address from another subnet, even if it's a valid private network. They always disconnect on such try. I guess it's related to CallStranger vulnerability. I think the only way to make it working with different subnets is to avoid any subscriptions at all and use polling only. |
@esev, don't you run your wemo's on a separate vlan too? |
I do, but my Home Assistant (Docker) has interfaces on both my |
Maybe it would be worth considering to add a parameter which avoids any subscription requests across subnets and forces local pulls only. I can see it works such way with Tasmota's devices running with wemo emulation enabled, which don't subsrcibe properly and return 404 on subscription requests. Local pulls work fine though even between different subnets. |
I like this idea. We can probably expose something like a |
Yes. I think it would be still wise to add the per device parameter in HA in case of devices which do not follow current Belkin's behavior and allow subscriptions between subnets (I just realized we do not discuss on HA's github...). |
Adding per-device parameters in Home Assistant seems like the best route. We can set the parameter default values based on what works for most Belkin devices. And folks can override those settings when it makes sense for their specific device. I tried to break out individual devices so they each have their own config entry before, but was told that wasn't recommended (home-assistant/core#44192 (comment)). At the time there weren't any per-device settings though. I can bring this up again and see if the situation would be different if we wanted per-device options/settings. |
Just following up to see if there has been any updates or solutions to this? I am having the same problems that OP was describing. |
Hi @joshuakohut77, Are you also using Home Assistant? Are you also seeing a lot of packet loss when you ping the device? For @onlyoneme and anyone else who is interested in this bug: If I were to add per-device configuration options for wemo devices in Home Assistant, what options would you like to see added? What would help to resolve this issue for you? Let's collect a list of options here, and I'll try to make the changes after the 2021.9.0 release. |
Hi @esev, As far as suggestions about resolutions, I'm not qualified to give any of value. I'm just a normal user who found this bug thread trying to resolve my log messages. However I am more than happy to do any testing and provide feedback with my home experience if that will be of use to you. |
@joshuakohut77 There are a few other ports that could be used, but the ones you listed are indeed the most common. pywemo/pywemo/ouimeaux_device/__init__.py Line 28 in 2ef78e0
I think one Home Assistant per-device config option that would help you is an option to disable the subscriptions, since we now know they won't work across subnets. |
I truly appreciate all you do so I don't want to seem impatient, but has there been any updates to the progress on this issue? |
I haven't had a chance to work on per-device configuration options yet. If you'd like to disable the subscriptions, and see if that improves things; comment out, or remove, these three lines in the wemo component within Home Assistant. Having feedback about whether this improves things would be very helpful! |
Okay I have been running this since yesterday afternoon. I no longer get consistent error logs as before. The device appear more responsive and turn on/off quicker but that could just be me wanting to see success. I have not visibly seen any devices become unavailable in home-assistant however when it happened before it would only last for a few seconds so unless I was watching, I wouldn't notice. These may or may not any significance as everything appears to be working as expected with only a sample size of 1 day to analyze. I'll keep an eye on devices becoming unresponsive in home-assistant throughout the week as I'll be at my desk working and it will be easier to monitor rather than the weekend where I am out and about. Let me know if you have any questions or anything more I can help with. |
Thanks for testing. This sounds very promising. The first set of logs look related to the number of network interfaces. Looks like you have quite a few. pyWeMo is trying to listen on all of them in order to respond to WeMo devices on the network. It's probably okay to ignore those logs. However, assuming this is Linux, you can try increasing the max multicast group memberships to make those logs go away. # List current value
sysctl net.ipv4.igmp_max_memberships
# Increase value (where X is something larger)
sysctl -w net.ipv4.igmp_max_memberships=X The second set of logs is indicative of pyWeMo not being able to connect to a device, or having an unreliable connection. Seeing that Maybe per-device configuration isn't needed here. It's a lot easier for me just to add a single setting to disable subscriptions. I'll try to put a change together by next week. |
Day two and everything is still great. Have not noticed any drop outs and the switches still seem much quicker to toggle. I think a single setting to disable subscriptions would be sufficient for this problem. |
What about NAT the request back via the firewall interface on the same subnet? I was about to try this out now... would need the callback URL to reference the firewall interface of the subnet rather than the address of the HA docker container because looking at a quick pcap the wemo doesn't seem to try connect if it's on another subnet. With this approach then the firewall can NAT it to the real address of the container on the other subnet. |
Did you have any luck with this? Sounds like a lot of effort when disabling the subscriptions seems to be fine for me at least. I’m not sure what benefit this would add in the end other than letting the pyWemo to work without any changes. |
@joshuakohut77 I've put together a PR for home assistant to add a configuration option to disable the subscriptions for all devices. home-assistant/core#56972 @tknz I think that would work. You'd need to:
Personally I think it's simpler to add the container to the same subnet as the WeMo (assuming that's possible). That way all the HA discovery logic works (for Wemo and all other integrations too). My docker host has a bridge interface on my IoT VLAN. Then the HA docker container has an interface on that bridge too. /etc/network/interfaces (Ubuntu host)
docker-compose.yaml services:
hass:
...
networks:
2_dmz:
4_iot:
ipv4_address: 192.168.100.15
1_internal:
ipv4_address: 192.168.1.15
networks:
4_iot:
enable_ipv6: true
ipam:
driver: default
config:
- subnet: 192.168.100.0/24
ip_range: 192.168.100.16/28
gateway: 192.168.100.1
driver_opts:
com.docker.network.bridge.name: br100
com.docker.network.bridge.host_binding_ipv4: 192.168.100.1 The key is to match the ipmaipam & driver_opts up with the same interface & settings as the bridge interface on the host. |
Yes got it going. Only real benefit is instant update when the button is pressed on the device. HA is delayed with other devices that are similar like Shelly unless COIOT (I use this) update or MQTT is used e.t.c |
There are a few other side-effects too. The WeMo motion sensor, and sensor on the WeMo Maker don't work well with HA when the subscriptions are disabled. The only time these devices will work properly, when subscriptions are disabled, is when HA polls at the exact same time as the motion occurs. The WeMo Insight energy monitors also don't display the current energy values as quickly when subscriptions are not used. The long-press feature on the wall Switches/Dimmers also will not be detected in HA. This isn't related to subscriptions, but rather relies on HA and the switches being able to see multicast messages on the local subnet. |
This thread is worthy of another wiki page that describes options for WeMo's on a different subnet. Could document the option to disable subscriptions (when/if added to pywemo) and what the caveats are to that. Of if you want to keep everything working well with subscriotions, the other two options are the extra interface like @esev does with Docker or the idea @tknz has implemented with DNAT. We can add the details on setting that up that we can send future people with the similar issues. |
Thanks, guys. It was driving me crazy. |
Thanks for this, I didn't realize how easy/simple this would be and it works great too. Would have saved me a lot of time configuring dozens of firewall rules to allow comms across vlans. |
Thanks for the feedback @joshuakohut77. It's great to hear that was helpful for you too.
Good call @Spectre5! I started https://github.com/pywemo/pywemo/wiki/Networking. I'll start a Home Assistant docs PR to have the wemo page link to this also. Anyone with other tips/tricks for getting pyWeMo to work across subnets, please feel free to contribute your solutions also! |
Thanks everyone for sorting this out, it has been bugging me for a while. I am kind of noob with networking, but if anyone is interested to guide me I could try to make it work with HAOS running as a VM on a proxmox node with 2 NICs. HAOS is on main LAN, wemos are on IoT LAN. I'd help documenting the steps and everything. |
Can you help me out with this? I've:
but i'm still seeing resubscribe errors in the HA logs If i can get this working, i'll update the guide at https://github.com/pywemo/pywemo/wiki/Networking. Would really like the button presses to update HA instantly, and also clear out these error messages in the logs. Commenting out the 3 lines doesn't fix either of these, even if the devices stop crashing. |
@jherby2k |
@jherby2k |
So, something like this?
also instead of editing subscribe.py, i set `pywemo_callback_address = '192.168.2.1' on line 75 in util.py. |
@jherby2k --to-destination in iptables should point to 192.168.1.2 |
Right, sorry - that was a typo. So, Python neophyte here - where can i define PYWEMO_CALLBACK_ADDRESS? in const.py? or can i define it somewhere else and not have to hack the wemo integration at all via custom_components? |
@jherby2k callback_address = get_callback_address( in subscribe.py with callback_address = '192.168.2.1:8989' |
FYI The |
Oh! It was actually released yesterday too. That was fast. https://github.com/home-assistant/core/releases/tag/2022.6.4 |
OK, so this is where i'm at: Running 2022.6.4 already. I hardcoded pywemo_callback_address = '192.168.2.1:8989' on line 75 of util.py, rather than setting PYWEMO_CALLBACK_ADDRESS somehow inside the HA container. Either way i need to make changes inside the container, right? This seems like a better spot than subscribe.py. then set this on my router:
(not specifying protocol/port means all the traffic intended for Wemo cloud gets redirected too) HA logs are clear of errors, but i still don't get instant updates in the UI when pressing the switch. I also don't see any connections to port 8989 in the FW logs. Thanks for your help - i really will document the heck out of this once i have it working. |
@jherby2k |
Had a typo in my iptables command (the above is fine)... working great now! I've added the following section: https://github.com/pywemo/pywemo/wiki/Networking#tips-for-running-on-a-different-lan it would be great if we could set PYWEMO_CALLBACK_ADDRESS via the Home Assistant integration. |
@jherby2k |
If anyone stumbles across this thread dealing with a separate iot vlan for wemos and is running HA in k8s, here is how I was able to configure PYWEMO_CALLBACK_ADDRESS. Pretty much the same solution as iptables, except k8s handles the iptables rules. My iot vlan is First add an available ip in the 192.168.4.x subnet to metallb config. I chose 192.168.4.200. Then set in HA container env:
Then configure a load balancer to expose 8989 on the iot subnet:
|
Are there any plans to add better logging or debugging to the subscription errors? I attempted to follow this thread and the wiki guide but my HA instance continues to receive a 400 error, "400 Client Error: Bad Request for url". Ive checked the env variable is set by using the HA terminal & ssh addon to verify its set with the IP and port yet it continues to error. I also disabled my vlan blocking connected another device and tested to confirm going to the callback address and port gives a 404 error which is the same thing I get on my main vlan |
I'm not sure if my issue is related to the library or to the home assistant integration. When home assistant is running with wemo integration enabled all my wemo smart plugs keep reconnecting to the network constantly every few minutes. This doesn't happen when my HA is off. Logs don't seem to be helpful for me:
The text was updated successfully, but these errors were encountered: