WiFi reliability testing with Ubiquity APs

[emontnemery]

Background

In my environment with Ubiquity APs, the only stable core version with sleep (50, dynamic) enabled, is Core 2.3.0

There are frequent reconnects to WIFI and MQTT server when sleep (50, dynamic) is enabled for the following core versions:

• Core 2.4.2
• Core 2.5.0
• Staging core + SDK 2.2.1 (Extremely unstable)
• Staging core + SDK 2.2.2

I'm testing on 26 devices (Sonoff T1 EU 1CH + Sonoff T1 EU 2CH + Sonoff S26)
WiFi APs are 2 x Ubiquitu. All Sonoffs are connected to same virtual WLAN where the clients may be handed over between the two APs.

Test 1 Core 2.4.2 - 46 hours:

In this test, sleep was disabled on most devices, purpose is to get a baseline for stability with sleep disabled.

No of devices	Core	Sleep	MQTT reconnects	WIFI reconnects
24	2.4.2	0	7 (0.006 / hour)	0 (0.00 / hour)
2	2.4.2	50/dynamic	6 (0.07 / hour)	4 (0.04 / hour)

Test 2 Core 2.4.2 - 37 hours:

In this test, sleep is enabled on around half of the devices, purpose is to get a baseline for stability with sleep enabled.

No of devices	Core	Sleep	MQTT reconnects	WIFI reconnects
13	2.4.2	0	18 (0.04 / hour)	0 (0.00 / hour)
11	2.4.2	50/dynamic	62 (0.15 / hour)	39 (0.10 / hour)

Test 3 Staging core - 7 hours:

Around half of the devices are upgraded to stage core with SDK 2.2.1 with sleep enabled.
Keep sleep disabled on half of devices to make sure no issue with WiFi.
This is clearly performing much worse than 2.4.2.
Maybe related to WiFi sleep always being disabled in 2.4.2?

No of devices	Core	Sleep	MQTT reconnects	WIFI reconnects
13	2.4.2	0	0 (0.00 / hour)	0 (0.00 / hour)
11	stage (sdk2.2.1)	50/dynamic	525 (6.82 / hour)	397 (5.16 / hour)

Test 4 Staging Core + SDK 2.2.2 - 7 hours:

Around half of the devices are upgraded to stage core with SDK 2.2.2 with sleep enabled.
Keep sleep disabled on half of devices to make sure no issue with WiFi.

No of devices	Core	Sleep	MQTT reconnects	WIFI reconnects
13	2.4.2	0	0 (0.00 / hour)	0 (0.00 / hour)
11	stage (sdk2.2.2)	50/dynamic	7 (0.09 / hour)	5 (0.06 / hour)

Test 5 Core 2.3.0 + SDK 1.5.3 - 9.5 hours:

Around half of the devices are downgraded to Core 2.3.0 with sleep enabled.
Keep sleep disabled on half of devices to make sure no issue with WiFi.

No of devices	Core	Sleep	MQTT reconnects	WIFI reconnects
13	2.4.2	0	0 (0.00 / hour)	0 (0.00 / hour)
11	2.3.0	50/dynamic	0 (0.00 / hour)	0 (0.00 / hour)

Test 6 Core 2.3.0 + SDK 1.5.3 - 240 hours (10 days):

All devices downgraded to Core 2.3.0 with sleep enabled.

No of devices	Core	Sleep	MQTT reconnects	WIFI reconnects
27	2.3.0	50/dynamic	47 (0.01 / hour)	0 (0.00 / hour)

SW & HW configuration:

• Device used (i.e. Sonoff Basic) : Sonoff S26 / Sonoff T1 EU 1CH / Sonoff T1 EU 2CH
• Tasmota binary firmware version number used : Commit 58d075d
• Development IDE - Compiler / Upload tools used : PlatformIO / OTA

[ascillato]

Hi,

Arduino core issues are outside the control of Tasmota. Searching in issues, other users with ubiquity have solved the unstable wifi using core 2.3.0.

Please, try the precompiled bin with core 2.3.0

http://thehackbox.org/tasmota/020300/sonoff.bin

The guys from arduino are migrating the sdk.

Some other Tasmota users also have solved wifi issues using the stage core. For compiling with latest version of the stage core you have compile by youself.

[emontnemery]

@ascillato I'm not expecting Tasmota to solve potential bugs in Arduino core, but I think the default settings with default binary should have working settings.

As you can see from my test above:

• 0 WiFi reconnects during over 1400+ tested hours with sleep disabled
• WiFi reconnects roughly every 10 hour with core 2.4.2 with dynamic sleep 50 - This is Tasmota default settings

[ascillato]

Yes, that is explained in the wiki in troubleshooting.

In some brands of routers core 2.4.2 and 2.5.0 have wifi issues, and core 2.3.0 have wifi issues with other brands of routers. So, we can't offer one version that works for everyone. That is why we offer Tasmota precompiled with the 3 available cores.

In the troubleshooting article of the core we put also that sleep 0 in core 2.4.2 may solve this wifi issues. But, sleep 0 can render some sonoff basic of a known bad batch without wifi.

So, for example Ubiquity and Fritzbox works better with core 2.3.0 and some mesh with latest stage. Tplink and mikrotik work fine with core 2.4.2
And the list goes and goes.
So, please, try the precompiled bin with core 2.3.0

[emontnemery]

OK, but could Sonoff try to be a bit intelligent about the sleep setting, e.g. toggle the sleep setting if there are reconnects?

Also, if the reason for making sleep default is a bad batch of Sonoff Basic, wouldn't it make sense to enable sleep by default only for Sonoff Basic?

Edit:
Just to make sure it's clear: I have solved my stability problem by disabling sleep. I'm just thinking if there's something that can be done to make Tasmota more stable "out of the box", without doing the long-term multi day testing that I've been doing.

[ascillato]

I totally understand you and your issue. But there isn't a core stable for everyone. What works in your case don't work for example in mine. Sleep 0 don't solve wifi issues for TP Link.

[ascillato]

The solution we have is offering the same last version of Tasmota compiled under the 3 cores so you can test which one works better for you

[DavinKD]

@emontnemery
You could also do a fork and make any changes you want. This isn't a commercial product, it's a labor of love for these folks.

[emontnemery]

@ascillato ah, so with the tplink APs you refer to it doesn't matter what sleep setting is used?
Then clearly my idea to implement some more intelligent sleep control won't work.

I can do some tests with staging core and core 2.3.0 if you're interested in the statistics, if not interesting this issue can be closed.

@DavinKD ?

[ascillato]

Yes, we are interested 👍 That is super useful.

The actual stage core (with the new toolchain) works better than 2.5.0 but you have to copy all files and download the full toolchain as they explain in the core readme. You need to delete all files before copying because if you left some older files, it will not compile.

We think that the next core (may be 2.5.1) will be much more stable due all the changes the arduino guys are doing. They are working really hard on that.

In that case, we might have just one core for everyone. But for now, we have 3.

From the statistics of the downloads, about the 5% of Tasmota users have issues with 2.4.2 and they move to core 2.3.0 or to core 2.5.0 in order to solve their wifi/mqtt issues.

[ascillato2]

@emontnemery

As a this wifi and mqtt reconnections issue is known by the arduino core guys and as they are working on that, and as depending on the brand of the router every user need to test which of the 3 cores works better (because depending on the brand, some works better with 2.3.0, others with 2.4.2 and others with 2.5.0), what is the best way you think this information/workaround can be transmitted to the users?

Now we have this explanation in the wiki at https://github.com/arendst/Sonoff-Tasmota/wiki/Troubleshooting#Wi-Fi-issues-arduino-core-versions-and-expressif-sdk and also we have this link posted in the wifi/mqtt old github issues.

[ascillato2]

Closing this issue as there isn't much we can do from Tasmota Software side as it is a known issue of the 2.4.2 core with ubiquity routers.

Anyway, please, let's continue with the testings so as to add this information to the wiki. Which router brands works with different core versions.

Thanks for all the testings. It is very appreciated. If any other idea or workaround shows up, please also share it.

[emontnemery]

OK, I'll do tests with core 2.3.0 and the staging core during the next few days and update here.
I renamed this issue to reflect it's now about collecting statistics with Ubiquiti routers.

[Jason2866]

@emontnemery @ascillato
Please test with nonos-sdk 2.2.2 (19.03.13) too

[emontnemery]

@Jason2866 Tested now, it seems to be just as bad as 2.4.2, but I'll let it run a few more hours. Staging core with SDK 2.2.1 is absolutely terrible though. I updated the stats in the issue description.
Let me know if I should test something else, maybe core 2.3.0?

[Jason2866]

Core 2.3.0 will be interesting because the upcoming release 6.5.0 will be again build by default with.
SDK 2.2.x isnt a big hit all. It is slower than 2.2.1. Hope was it solves wifi issues. So no reason to use...

EDIT (26.03): With Compiler flag option -O2 SDK 2.2.2 works as fast as 2.2.1

[emontnemery]

OK, I interrupted the staging+2.2.2 test and changed to 2.3.0.

[emontnemery]

@Jason2866 Core 2.3.0 seems to work fine with sleep enabled, I updated issue description.
I'll flash all devices with Core 2.3.0 + sleep enabled now to get some more statistics.

[digiblur]

Two Ubiquiti AC LR access points here, single SSID of 2.4ghz for Tasmota devices, I also have setoption56 and 57 turned on to let them roam around like they are supposed to. I use core 2.3.0 on all my 40+ devices as well because 2.4.2 and 2.5.0 just had constant MQTT disconnect issues. 2.3.0 is super solid and I won't see any LWT disconnects at all. Default dynamic sleep of 50 is enabled on all of them.

[emontnemery]

@digiblur Also for me, core 2.3.0 is the only stable version.
Just curious, what does setoption56 and 57 do if you have a single SSID?

[digiblur]

From my understanding it will select the better AP eventually say from a rolling firmware AP upgrade that shakes things up. #4817 (comment)

[emontnemery]

That's if you have defined two APs in Tasmota settings though, aren't you only defining a single one?

[Jason2866]

Mhh, if i remember correct it switches AP too, if you use same SSID for all your APs (one AP in Tasmota configured). This is the usual setup in a WLAN with more AP. Devices can do a handover or are forced to (wlan ap steering).

[Jason2866]

Thinking about AP Steering, maybe this is the reason for the behaviour with ubiquiti.
Ubiquiti has a function called Fast Roaming / Zero Handoff (a implementation of AP steering).
With this function client does only see ONE AP (in real there are more...).
Maybe this handling (background switching APs) generates reconnects.
@emontnemery can you test to switch all this nice features off?
Found this links https://help.ubnt.com/hc/en-us/articles/115004662107 https://help.ubnt.com/hc/en-us/articles/205144590-UniFi-What-is-Zero-Handoff

[emontnemery]

@JSON yeah, that's what I'm using, and I think @digiblur is using it too so I'm curious if setoption56 and 57 are needed.
I've tested connecting to a virtual WLAN tied to just one of the APs, it improves stability a bit, but not to the same level of 2.3.0.

[Jason2866]

@emontnemery SetOption 56 and 57 shouldnt be needed. Controller based steering of clients is much better than client side one... We have to fight with the buggy or incomplete wifi implementation from espressif (closed source)
Weired that it doesent work reliable with just one AP!

[meingraham]

I have a Google WiFi mesh with two WiFi Points (i.e., two AP). Both broadcast the same SSID. My Tasmota devices connected to the first AP with the configured SSID that they found. I was having to play lots of games to get each device to connect to the AP with the best signal). That was, until SetOption56 and SetOption57 were introduced. Now, if a device starts up, those options cause Tasmota to scan for all APs with the configured SSID and connect to the best signal... and to repeat that process every 44 (?) minutes. I have found that signal levels from my AP fluctuate (even before I had a mesh). I'm sure that many things regarding just our regular "living" in the house causes this, even using the microwave oven. For my house, SetOption56 and SetOption57 really "stabilized" and improved my Tasmota device connections.

[digiblur]

Not using fast roaming myself, I can see the multiple MAC IDs when I do scan without issue on Tasmota.

[emontnemery]

I updated the stats for core 2.3.0 after 10 days of testing (a total of 6500 hours of testing across the 27 devices), there are still 0 WIFI reconnects.

[sislakd]

Do know why there are MQTT reconnects? I'm experimenting with WiFi and MQTT stability on Asus AC68U where I have 2.4GHz band used only by 24 ESP8266 devices (mix of Shelly 1, Shelly 2, ESP12F,Nodemcu,Sonoff Basic, S26, 4CHPRO R2, POW R2 and RF Bridge). However, I observe reconnects even with core 2.3.0 and Dynamic 50ms sleep. The setup where I have no WiFi reconnects is Normal 0ms sleep. It is working for me with old core 2.3.0 but also with stage core (2.6.0-dev) using SDK 2.2.1 and SDK 2.2.2-dev. I've experimented with frequency 160MHz CPU/80MHz flash and 80MHz CPU/40MHz flash where there isn't any substantial effect. The only case where I need 160MHz is ESP8266 connected to MQTT over TLS (at the moment AxTLS) with software serial for GSM module, hardware serial for Paradox system and some DS18x20 sensors. With 80MHz clock I don't have stability on software serial on longer AT messages exchanged with GSM module.

[Jason2866]

@sislakd The issues have dependencies with the used wifi hardware/software.
For example i have devices with core 2.4.2/2.2.1 and core stage /2.2.1 2.2.2 with 0 reconnects over days.
All devices are using Tasmota dynamic sleep 50. I dont use core 2.3.0 anymore because no need for...

Example

18:21:41 MQT: tele/sonoff-13D92F/STATE = {"Time":"2019-03-26T18:21:41","Uptime":"6T22:22:42","Vcc":3.423,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi {"AP":1,"SSId":"Jason_Home_WLAN","BSSId":"00:A0:57:2A:BD:19","Channel":13,"RSSI":100,"LinkCount":1,"Downtime":"0T00:00:05"}}
18:31:23 MQT: stat/sonoff-13D92F/STATUS2 = {"StatusFWR":{"Version":"6.5.0.1(sonoff)","BuildDateTime":"2019-03-19T19:34:51","Boot":31,"Core":"STAGE","SDK":"2.2.1(cfd48f3)"}}

[majherek]

Hi,

I have similar issue. I use two Meraki MR33 with the same SSIDs. I use client ip assigment as bridge (not L3 Roaming). I don't user band steering. I do not user webserver, I use BearSSL + MQTT. I do not use SetOption56 and SetOption57.

And every tasmota device (6.4.0.14 with Core 2.5.0) is reconnecting.

This is event log from dashboard meraki for one of my sonoff basic.

Time(CET)       | Access point | SSID     | Client                           | Event type | Details
Mar 21 06:36:26 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | WPA authentication | ""
Mar 21 06:36:26 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | 802.11 association | "channel: 11, rssi: 42"
Mar 21 06:36:26 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | 802.11 disassociation | "client was deauthenticated"
Mar 21 06:36:26 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:72:0B:55"
Mar 14 22:40:58 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | WPA authentication | ""
Mar 14 22:40:58 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:72:0B:55"
Mar 14 22:40:58 | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | 802.11 association | "channel: 11, rssi: 43"
Mar 6 17:10:27  | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | WPA authentication | ""
Mar 6 17:10:27  | GARDEROBA | atomix | sonoff14 KOTŁOWNIA | 802.11 association | "channel: 11, rssi: 42"

And for sonoff SV

Time(CET)     | Access point | SSID | Client | Event type | Details
Mar 20 19:08:08 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 20 19:08:08 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 20 19:08:08 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 28"
Mar 20 19:08:08 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 disassociation | "client was deauthenticated"
Mar 15 01:19:18 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 15 01:19:18 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 15 01:19:18 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 29"
Mar 15 01:19:18 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 disassociation | "client was deauthenticated"
Mar 15 01:19:18 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 14 22:40:52 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 14 22:40:52 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 30"
Mar 14 22:20:59 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 14 22:20:59 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 14 22:20:59 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 30"
Mar 14 22:20:59 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 disassociation | "client was deauthenticated"
Mar 12 16:37:38 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 12 16:37:38 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 12 16:37:38 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 27"
Mar 12 16:37:38 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 disassociation | "client was deauthenticated"
Mar 12 16:37:38 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 10 20:18:50 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 10 20:18:50 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 31"
Mar 10 20:18:50 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 disassociation | "client was deauthenticated"
Mar 10 20:18:50 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA deauthentication | "radio: 0, vap: 2, client_mac: 5C:CF:7F:1C:06:2E"
Mar 6 17:10:27 | GARDEROBA | atomix | sonoff6 ODKURZACZ | WPA authentication | ""
Mar 6 17:10:27 | GARDEROBA | atomix | sonoff6 ODKURZACZ | 802.11 association | "channel: 11, rssi: 33"

[Jason2866]

Core 2.5.0 and which SDK ? 3.0.0. dev is known to be faulty.
Arduino crew has stopped using it, back to SDK 2.2.1

[gajotnt]

Also have Ubiquiti unifi AC-LR, been using tasmota with pre 2.6 core since its the recommended, had some problems that i cannot tell if its the tasmota or the unifi AP (since i only had them a couple of days).
For the unifi users any recomendations? Already have them in a seperate SSID (hidden)

[Jason2866]

For Tasmota do NOT hide SSID. Disable channel auto select in APs

[gajotnt]

Awww mannnn :( lol was really enjoying not having SSID showing that no one has to know about :P

[tobox]

For a long time, I have had exactly the same problems as the OP with all core versions except for 2.3.0.

But a few weeks ago I started updating some of my Tasmota-Devices to "STAGE" instead of "2.3.0" and everything seems to be running as stable as with 2.3.0. Definitely much better than 2.4.2 and 2.5.0.

Maybe the OP can run some more tests with the new versions?

[Jason2866]

In core pre 2.6 are two big fixes since the tests made. Until now there are NO negative feedbacks.
The new release v.6.7.0 (launched on 25.10.2019) Tasmota will use core pre 2.6.
All other core support will be removed on the next release 6.8.0

[gajotnt]

Not that ubiquiti wifi experience is a reliable feature but:

Most of my sonoffs are runing great with stage, but these two dont seam to like it very much.

[meingraham]

@gajotnt

Wondering if this might help - https://github.com/arendst/Sonoff-Tasmota/wiki/FAQ#Weaker-Wi-Fi-signal-after-upgrade

[gajotnt]

Thanks, will try it when i get home :)

[gajotnt]

worked on one of them, the other stayed the same. will try another sonoff

[Jason2866]

Sometimes, wifi issues are just solved when doing a full erase flash (recommended esptool.py)
and flashing Tasmota (esptool.py) again.

[gajotnt]

Ok, will try a another sonoff and do a full erase flash before uploading the FW. Thanks for the input :)