Wemos D1 mini bricked (no WiFi) after OTA upload - with new wifi credentials

[MatejsLab]

Basic Infos

• [y] This issue complies with the issue POLICY doc.
• [y] I have read the documentation at readthedocs and the issue is not addressed there.
• [August 2022, but I can't repair the device with the latest] I have tested that the issue is present in the current master branch (aka latest git).
• [ y] I have searched the issue tracker for a similar issue.
• [just serial debug message] If there is a stack dump, I have decoded it.
• [ y] I have filled out all fields below.

Platform

• Hardware: [ESP-12|ESP-01|ESP-07|ESP8285 device|other]
• Core Version: esp8266 core 3.0.2 [latest git hash or date]
• Development Env: [Arduino IDE|Platformio|Make|other]
• Operating System: [Windows|Ubuntu|MacOS]

Settings in IDE

• Module: [Generic ESP8266 Module|Wemos D1 mini r2|Nodemcu|other]
• Flash Mode: [DOUT|qio|dio|other]
• Flash Size: [4MB/1MB]
• lwip Variant: [v1.4|v2 Lower Memory|Higher Bandwidth]
• Reset Method: [ck|nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz|160MHz]
• Upload Using: [OTA|SERIAL]
• Upload Speed: [115200|other] (serial upload only)

Problem Description

Wemos D1 mini gets bricked aka unable to connect to WiFi after OTA upload with new SSID and password.
By bricked I really mean, no help to it, by loading new different sketch to it via serial still no WiFi connection, if I take new Wemos it runs the same sketch that bricked the other ESP no problem, it runs Wifi and MQTT fine, but I don't want to brick another one just to prove it.
I tried to Erase all Flash contents but it was no help to the bricked ESP.
I faced the same issue previously about 4 times over the last few years, that changing wifi credentials bricks the ESP after reboot, still, I was always able to recover by serial upload, but not this time.

Why is this an issue, and how should I avoid it, maybe with Wifi Manager Library, I've read about it but don't yet know how to use it. The goal is can change the wifi credentials on ESPs without messing things up, and using serial, why do we have OTA? And yes I do stage updates on test devices first.

How can I get bricked ESP back to life what should I upload to it as it is not dead it talks on serial, receives uploads normally, and runs blink sketch just fine, but not Wifi. The madness is that my Wifi MQTT code runs on new ESP but not on bricked one, is it possible to damage internal ESP wifi hardware with code?

The code below should prove the issue, my code has also MQTT which has nothing to do with the issue.
To prove it just follow these steps:

1. upload sketch with first wifi credentials via serial
2. change wifi credentials
3. upload via OTA while still connected to the first wifi
4. esp is unable to connect to wifi and shows the error below, by enabling serial debug Wifi

The issue may not emerge if the code running ESP OTA is on the latest core(aka updating over the latest core version), but that does not mean it is fine, as it still breaks old devices by updating via OTA.

Am I the first poking the thin ice on Arduino OTA, as I have not found much related to the issue on google.

Looking forward to finding the solution.

Cheers Matej

MCVE Sketch

#include <ESP8266WiFi.h>
#include <ESP8266mDNS.h>
#include <WiFiUdp.h>
#include <ArduinoOTA.h>

#ifndef STASSID
#define STASSID "first ssid"  // than change to another ssid
#define STAPSK  "first passwd" // than change to another passwd and do OTA
#endif

const char* ssid = STASSID;
const char* password = STAPSK;

void setup() {
  Serial.begin(115200);
  Serial.println("Booting");
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.waitForConnectResult() != WL_CONNECTED) {
    Serial.println("Connection Failed! Rebooting...");
    delay(5000);
    ESP.restart();
  }

  ArduinoOTA.begin();
  Serial.println("Ready");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
  ArduinoOTA.handle();
}

Debug Messages

SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635
Booting
fpm close 1 
mode : sta(80:7d:3a:6e:4c:d4)
add if0
wifi evt: 8
wifi evt: 2
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
reconnect
wifi evt: 1
STA disconnect: 2
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
reconnect
wifi evt: 1
STA disconnect: 2
scandone
state: 0 -> 2 (b0)
state: 2 -> 0 (2)
reconnect
wifi evt: 1
STA disconnect: 2
Connection Failed! Rebooting...

[mcspr]

Do you mean the issue happens after update from firmware using Core 2.7.4 to 3.0.2?
After erasing flash contents, does anything change after adding WiFi.persistent(false); before the WiFi.begin(...);?

[MatejsLab]

Hm, WiFi.persistent(false); sadly does not help.
Indeed, the previous core version ought to be 2.7.4 as I uploaded my (water level measuring) sketch on 5. February 2021. That is before 3.0.0.

I tested this bricked ESP with basic WiFi AP sketch and it works fine, so WiFi hardware is still operational. It is madness that no matter what I do, in WiFi STA mode, the bricked ESP is unable to connect, but the new one runs any WiFi STA sketch fine including my sketch, the bricked one can't run any STA sketch successfully.

I have no clue what exactly OTA did to mess with Wifi STA connectivity, and how can i fix it, it is more about other 2.7.4 devices that I don't want to brick in the future with OTA.

[MatejsLab]

That 2.7.4 version in combination with OTA is a nuisance. I found out that I still have 2.7.4 core active on my weekend loghouse Linux box. Didn't know that before I wanted to remotely update some other ESP with some sensors in the loghouse, but updating 2.7.4 over 3.0.2 also causes issues, now ESP is not accessible via OTA anymore but luckily WiFi and my code still work as I haven't messed with credentials in that case.

3.0.2 is awesome and stable for OTA, but getting rid of all that 2.7.4 is a pain with all embedded devices.

[mcspr]

I may have foggy memory about which settings are simply ignoring that flag... Perhaps, one also needs to forcibly disable 'auto{re}connect'; at least one of those causes your previously entered credentials to be stored on flash, which in turn causes a connection attempt to be initiated by SDK with those credentials and without any user interaction required.

We do have some preventative actions like disconnect(), but may be simply canceling the previous attempt of connection (or, going through with it during that time) without initiating the new one.
I'd suggest to check re-connect flags

[MatejsLab]

I browsed the forums and GitHub and I found this below, apparently, it tries to force new credentials that are written in code, but it does not repair the poor bricked esp.

const char* ssid = "my_ssid";
const char* password = "my_pass";

void setup()
{
WiFi.persistent(false);
WiFi.setAutoConnect(false);
WiFi.setAutoReconnect(false);
WiFi.mode(WIFI_OFF);
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, password);
}

I really have no clue how to check what is wrong with the esp wifi_sta hardware that it does not connect on that bricked esp, because it does work with wifi_ap, so the wifi hardware is not broken right? I assume that It messed something up in the flash when it failed the OTA and that register for wifi_sta is broken and can't be set?

Or that I messed it up by setting erase all flash and then it stopped working like described in this post>> #3100 Wifi only works once after erasing flash. I did that erase after failing OTA, not understanding the consequences.

Possibly I wish too hard for that esp to work again as I have some working spares, but is there still hope, at least I would like to confirm the root cause forESP in sta mode not connecting.

And thanks for all your effort, I appreciate it.

[JAndrassy]

try to select in Tools menu "Erase flash" to "All flash content" for one upload

[TD-er]

I've also had the same here on some module.
The only way to get it to connect to WiFi again was flashing it again via the IDE.
N.B. I did not need to erase the flash for it.

I got it somewhat "reproducible" when messing around with the WiFi reconnect code.
But not sure yet what was happening. It seems a timing issue when the WiFi radio was turned off quite fast during connecting to WiFi. (acting on an event which got apparently somewhat delayed)