ESPhttpUpdate seems to corrupt WiFiManager after a OTA http update

[domingosl]

Basic Infos

Hardware

WiFimanager Branch/Release:

• Master
• Development

Esp8266/Esp32:

• ESP8266
• ESP32

Hardware: ESP-12e, esp01, esp25

• ESP01
• ESP12 E/F/S (nodemcu, wemos, feather)
• Other

ESP Core Version: 2.4.0, staging

• 2.3.0
• 2.4.0
• staging (master/dev)

Description

I recently added OTA updates using HTTP to a program I've been testing for the past month with no issues. Something strange happens tho after an OTA update, the BIN downloads and install correctly, the ESP restarts, the WiFi credentials are wiped so the ESP goes into configuration mode, but as soon it gets there it reboots constantly with this error stack:

Exception (28):
epc1=0x40240429 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000038 depc=0x00000000

stack>>>

ctx: cont
sp: 3fff0270 end: 3fff0630 offset: 01a0
3fff0410: 3fff0b34 3fff1914 3fff1914 4022cc8b
3fff0420: 4024403d 00000002 3ffee618 00000000
3fff0430: 40244055 00000002 00000002 00000002
3fff0440: 402441b9 00000002 00000002 4024416d
3fff0450: 40243fee 00000001 00000000 3fff0b34
3fff0460: 40244292 00000002 00000002 00000001
3fff0470: 40204027 402044cc 3ffeeee4 3fff04f0
3fff0480: 40215618 3ffeeee4 3fff0540 3ffe86c2
3fff0490: 00000000 3ffeeee4 3fff0540 40209c62
3fff04a0: 3ffef1e8 000001a3 000001a3 401009e8
3fff04b0: 00000000 3ffeeee4 3fff04fc 40100e2c
3fff04c0: 00000000 3ffeeee4 3fff04f0 4020fbd0
3fff04d0: 3ffe8778 00000000 3fff0540 3ffe86c2
3fff04e0: 00000000 3ffeeee4 3fff0540 40209e6d
3fff04f0: 00000000 00000000 00000000 00000000
3fff0500: 00000000 00000000 00000028 40100c38
3fff0510: 00000001 3ffe8778 3fff0540 40208350
3fff0520: 00000001 00000000 3ffeef80 00000001
3fff0530: 3ffeed00 3ffeee00 3ffeef80 40202cbd
3fff0540: 00000000 00000000 3ffe8c24 00000000
3fff0550: 3fff1414 0000000f 00000000 3fff142c
3fff0560: 0000000f 00000000 00000000 00000000
3fff0570: 00000000 40215eb0 00000000 40215eb0
3fff0580: 00000000 40215eb0 00000000 40215eb0
3fff0590: 00000000 40215eb0 00000000 40215eb0
3fff05a0: 00000000 00000000 ffffffff fe000001
3fff05b0: 3ffe8778 00000000 fe00ef35 40202e3c
3fff05c0: 00000000 0000000a 3fff1444 feefeffe
3fff05d0: feefeffe feefeffe feefeffe feefeffe
3fff05e0: feefeffe feefeffe feefeffe feefeffe
3fff05f0: 0000000a feefeffe feefeffe feefeffe
3fff0600: feefeffe feefeffe feefeffe 3ffef098
3fff0610: 3fffdad0 00000000 3ffef064 402109f4
3fff0620: feefeffe feefeffe 3ffe863c 40100e89
<<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,7)

What its weird to me is that the downloaded BIN and the running BIN are exactly the same, so when the installed one runs it does not crash, ever, but when you download the copy from my http local server and install it something in memory gets corrupted that affects WiFiManager.

Does anyone experience something similar? If not, I'm not very good at debugging using the stack, maybe you can point me in the right direction.

BTW, the code I'm using to update my program is the basic one:

 t_httpUpdate_return ret = ESPhttpUpdate.update("192.168.1.53", 80, "/", firmwareVersion);
  
  switch(ret) {
      case HTTP_UPDATE_FAILED:
          Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
          break;
      case HTTP_UPDATE_NO_UPDATES:
          Serial.println("[update] Update no Update.");
          break;
      case HTTP_UPDATE_OK:
          Serial.println("[update] Update ok.");
          break;
  }

Im using a ESP12F, 4Mb no SPFSS.

Somebody had what it seems a similar issue #236 but is not very clear.

[domingosl]

Quick follow up, after the OTA update, the crash is so bad that reflashing the chip just maintains the corrupted stuff and keeps crashing. I have to change the "Erase flash" option in the Arduino IDE from "Only Sckect" to ALL FLASH CONTENTS so it clears everything

[tablatronix]

You have corrupt flash have to full erase, you probably updated from an older sdk or something.

[tablatronix]

See if iy happens again

[domingosl]

It happens every single time. Is not an older SDK, this is how I reproduce the issue every time:

1. Flash the program to the ESP using the Arduino IDE
2. Run and test, it goes into config mode, I set up the WiFi credentials, it connects to the network and checks for updates in my local web server that is down in this step, then the program continues with no problems.
3. I take the binary generated in step 1, and added to my local web server and lunch it.
4. Reset the ESP, it connects to WiFi using the credentials from step 2, connects to my web server, downloads the BIN, reset itself after a successful install
5. The ESP goes into WiFi config mode as expected but then it resets with the stack output from my first message and it keeps resetting every time
6. From here I need to completely clear the flash storage to fix the issue

As you can see the BIN I'm using during the serial programming is exactly the same BIN I'm using during the OTA update. Unless there is something wrong with the ESPhttpUpdate library, it almost looks to me that during the OTA flash some parameter store by WiFiManager in flash gets corrupted

[tablatronix]

I have not tested ota on master branch, only development, and have not had an issue with it.

when you say

The ESP goes into WiFi config mode as expected

Should not be expected, credentials should not be affected from an ota update.

Is ota code doing anything with wifi , like disconnect() ?

[tablatronix]

only thing I can think of is to decode the exception, and check the stack trace.

or eliminate pieces and see what happens, maybe remove wm, and do manual begin()

[domingosl]

Should not be expected, credentials should not be affected from an ota update.

I just assume that OTA update clears all the flash memory, if not, you're right, the WiFi credentials should still be there. This just still makes me think that something weird happens after the OTA. Again, the BIN is correct but something is being modified in flash that breaks WiFIManager.

One of my theories was a bad board parameter, for example, having a board with 2Mb of flash but setting it as 4Mb could mean that a BIN flashed on it could overwrite data in places of the flash memory used by other things, like the WiFi credentials. But I'm using an ESP12F that are supposed to be 4Mb flash.

I need an expert of low-level stack to understand what is missing from the stack trace of my first message.

[tablatronix]

yes wrong flash will cause issues like this, maybe the ota method you are using is wrong, or requires a missing argument for flash size?

you can just use the espexception decoder ( its a ide pliugin) and post it

[domingosl]

yes wrong flash will cause issues like this, maybe the ota method you are using is wrong, or requires a missing argument for flash size?

So far I think I'm doing everything correctly

you can just use the espexception decoder ( its a ide pliugin) and post it

Decoding stack results
0x402178d0: _Unwind_RaiseException at ../../../../dl/gcc-xtensa/libgcc/unwind.inc line 83
0x4021f4b4: __cxxabiv1::__terminate(void ()()) at ../../../../../dl/gcc-xtensa/libstdc++-v3/libsupc++/eh_terminate.cc line 39
0x4021fbf4: __gxx_exception_cleanup(_Unwind_Reason_Code, _Unwind_Exception) at ../../../../../dl/gcc-xtensa/libstdc++-v3/libsupc++/eh_throw.cc line 33
0x4021fdf5: operator new(unsigned int) at ../../../../../dl/gcc-xtensa/libstdc++-v3/libsupc++/new_op.cc line 57
0x40209832: WiFiManager::setupConfigPortal() at C:\Users\Domingo\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 109
0x40210950: esp_yield() at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_main.cpp line 91
0x402016aa: delay at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_wiring.c line 54
0x4020169f: delay at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_wiring.c line 51
0x40209ca0: WiFiManager::startConfigPortal(char const*, char const*) at C:\Users\Domingo\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 240
0x401009e8: _umm_free at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\umm_malloc\umm_malloc.c line 1300
0x40100e2c: free at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\umm_malloc\umm_malloc.c line 1760
0x4020fbd0: String::invalidate() at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\WString.cpp line 141
0x40209e6d: WiFiManager::autoConnect(char const*, char const*) at C:\Users\Domingo\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 183
0x40100c38: malloc at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\umm_malloc\umm_malloc.c line 1673
0x40208350: WiFiManager::WiFiManager() at C:\Users\Domingo\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 72
0x40202cbd: setup() at C:\Users\Domingo\dev\ESP\gw-ticker-2/gw-ticker-2.ino line 93
0x40202e3c: configModeCallback(WiFiManager*) at C:\Users\Domingo\dev\ESP\gw-ticker-2/gw-ticker-2.ino line 135
0x402109f4: loop_wrapper() at C:\Users\Domingo\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.0-beta2\cores\esp8266\core_esp8266_main.cpp line 122

[tablatronix]

do you have serial logs before the exception ?

[tablatronix]

curious how far into setupConfigPortal this is failing, it is not clear

[domingosl]

This might be related with esp8266/Arduino#5522

[mignacio]

Hi, I'm having exactly the same issue. ESP 12F, core version 2.5.0. Also in 4Mb no SPIFFS.

After trying an OTA update, trying to setup the ESP as AP would throw Exception(28). I'm no an expert in reading the stack, but just wanted to let you know that I was able to recreate the same issue.

I'll look into #5522 for more info.

[Daemach]

This is an aside, but by any chance did you do a firmware update using serial before testing OTA? There is a known issue that you need to fully power down the device after a serial firmware update before doing OTA or it will get into a crash loop like this. Just disconnect everything then power it up and try again.

…

On Wed, Mar 20, 2019 at 8:22 AM Ignacio Moya ***@***.***> wrote: Hi, I'm having exactly the same issue. ESP 12F, core version 2.5.0. Also in 4Mb no SPIFFS. After trying an OTA update, trying to setup the ESP as AP would throw Exception(28). I'm no an expert in reading the stack, but just wanted to let you know that I was able to recreate the same issue. I'll look into #5522 for more info. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#808 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AL-O4eS9K4xzE040OK_Z234cSCa19EiFks5vYlIegaJpZM4aD-4m> .

[mignacio]

Yes. First I updated the firmware through serial. Then did a reset but with the reset button. I will try fully powering down and then up. And see if the problem persists.

[mignacio]

Ok, I just confirmed fully powering it down and the up has the same results.

[Daemach]

So you unplugged the device from all power, then plugged it back in, waited for the OTA and it got into a death loop after the update? That's unfortunate. Cycling power solved my problem. See this thread for more info: esp8266/Arduino#2478 Maybe you can post your code here.

…

On Thu, Mar 21, 2019 at 5:58 AM Ignacio Moya ***@***.***> wrote: Ok, I just confirmed fully powering it down and the up has the same results. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#808 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AL-O4Zo5hrVC72qJIfsAFPEJIEh5-m51ks5vY4HggaJpZM4aD-4m> .

[tablatronix]

What exactly should I do to reproduce this?

[smarta1980]

Arduino version 1.8.10
ESP Board version 2.5.2

I havae a very serious problem and not find any solution for it.
I upload bin file to update ESP-12E from server using OTA
then all times I have issue for wdt reset.
on the other hand, I tried to program same code via usb no have any problem so the problem just with OTA wdt reset all times
could you please guide me what to do?

this is settings I used in Arduino IDE

[tablatronix]

Do you have a log? Are you reseting esp after programming it via usb ( in case you are testing, you must reset for ota after serial upload )

[Daemach]

You need to unplug power after programming via USB/Serial and before doing OTA.

…

On Tue, Sep 17, 2019 at 12:11 PM Shawn A ***@***.***> wrote: Do you have a log? Are you reseting esp after programming it via usb ( in case you are testing, you must reset for ota after serial upload ) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#808?email_source=notifications&email_token=AC7Y5YIPNFPCPKTXFSRM4LDQKETV3A5CNFSM4GQP5YTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65S3KI#issuecomment-532360617>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC7Y5YM3WCZBQUV2V3UIVM3QKETV3ANCNFSM4GQP5YTA> .

[smarta1980]

you can check the log after sending 108 messages each 10 seconds to firebase cloud messaging then wdt rest.
Log.txt

[smarta1980]

You need to unplug power after programming via USB/Serial and before doing OTA.

The steps that I do as following :
1- program OTA sketch via usb.
2- then after upload it I disconnect usb port to esp board
3- connect usb again to esp board
4- wait until OTA update

[smarta1980]

I can not find solution all times wdt reset

[tablatronix]

What does this have to do with ota ? You are having WDT resets in your code..

[smarta1980]

after ota update the code wdt reset
if use same code without ota it is working without any problem.

[tablatronix]

So you can reboot and it will run fine, do an ota, it resets, runs for a bit then wdts?

[marcolettieri]

same issue here.
Firmware works perfectly, when ota is launched, after some seconds i receive exception stack and after reboot the esp8266 'forget' all Wifi credentials.

this is decoded exception

Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
Decoding 29 results
0x40236031: netif_set_down at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c line 616
0x402477b0: node_remove_from_list at ?? line ?
0x40236054: netif_remove at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c line 537
0x40239bde: wifi_get_opmode at ?? line ?
0x402381d2: eagle_lwip_if_free at ?? line ?
0x4023ad7a: wifi_station_disconnect at ?? line ?
0x40203abc: ESP8266WiFiSTAClass::disconnect(bool) at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/ESP8266WiFiSTA.cpp line 373
0x402189c8: tcpip_tcp_timer at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c line 145
0x4021fe80: mem_malloc at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/mem.c line 210
0x40218928: lwip_cyclic_timer at /home/gauchard/dev/esp8266/esp8266/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c line 233
0x401001fc: millis at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/core_esp8266_wiring.cpp line 186
0x4020b21a: String::reserve(unsigned int) at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/WString.cpp line 148
0x4020b36c: String::move(String&) at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/WString.cpp line 230
0x4020107f: std::_Function_handler ::_M_invoke(std::_Any_data const&, WiFiEventStationModeDisconnected const&) at c:\users\marcoletteri\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-3-20ed2b9\xtensa-lx106-elf\include\c++\4.8.2/functional line 2073
0x402034e8: std::_Function_handler  )::{lambda(_esp_event*)#1}>::_M_invoke(std::_Any_data const&, _esp_event*) at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/ESP8266WiFiType.h line 112
:  (inlined by) operator() at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/ESP8266WiFiGeneric.cpp line 122
:  (inlined by) _M_invoke at c:\users\marcoletteri\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-3-20ed2b9\xtensa-lx106-elf\include\c++\4.8.2/functional line 2071
0x40100aca: ppEnqueueRxq at ?? line ?
0x40204cf5: WiFiClient::stopAll() at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/WiFiClient.cpp line 386 (discriminator 2)
0x4020371b: ESP8266WiFiGenericClass::_eventCallback(void*) at c:\users\marcoletteri\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-3-20ed2b9\xtensa-lx106-elf\include\c++\4.8.2\bits/stl_list.h line 157
:  (inlined by) ESP8266WiFiGenericClass::_eventCallback(void*) at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/ESP8266WiFiGeneric.cpp line 234
0x401008d2: ppProcessTxQ at ?? line ?
0x4023c2e4: wifi_set_status_led_output_level at ?? line ?
0x40104438: call_user_start_local at ?? line ?
0x4010443e: call_user_start_local at ?? line ?
0x4010000d: call_user_start at ?? line ?
0x40100600: cont_ret at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 142
0x401005ad: cont_continue at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\cores\esp8266/cont.S line 51

[tablatronix]

Erase flash

[tablatronix]

What esp versions are you all using ?

[marcolettieri]

@tablatronix as you can see in logs 2.5.2
Flash has been erase already

[tablatronix]

Ok, are you using arduino ota, or http update ?
Can I reproduce this by just adding ota to one of the basic WM examples?

[marcolettieri]

ESPhttpUpdate

[tablatronix]

Just tested using ota branch (dev + ota form)

https://github.com/tzapu/WiFiManager/blob/ota/examples/DEV/OnDemandConfigPortal/OnDemandConfigPortal.ino

Works fine, it uses form upload ota.

Could be a bug in master branch, are you using 0.15 beta?

[tablatronix]

Got min sketch code ?

[tablatronix]

Your code is causing a disconnect(), Not sure what you are calling, but that is where your creds go, bye buye

0x40203abc: ESP8266WiFiSTAClass::disconnect(bool) at C:\Users\MarcoLetteri\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.5.2\libraries\ESP8266WiFi\src/ESP8266WiFiSTA.cpp line 373

It sure does look like that is a out of bounds memory access in lwip

[tablatronix]

Do you have any event callbacks ?

[marcolettieri]

yes, no one is called, it starts doing update and then rise the exception

        t_httpUpdate_return ret = ESPhttpUpdate.update(REMOTE_SERVER, 80, SKETCH_BIN);

        switch(ret) {
            case HTTP_UPDATE_FAILED:
                Serial.println(ESPhttpUpdate.getLastErrorString().c_str());
                break;

            case HTTP_UPDATE_NO_UPDATES:
                Serial.println("HTTP_UPDATE_NO_UPDATES");
                break;
        }

[tablatronix]

Ok so maybe it has something to do with wificlient , ill try a ota fetch.
Wonder if it has sonething to do with wm not stopping webserver and dns after using it

[tablatronix]

Make sure you are trying the 0.15 beta there was a loss of creds bug fixed I think

[marcolettieri]

this happens also without WiFiManager opened issue on esp8266/Arduino#6555