-
Notifications
You must be signed in to change notification settings - Fork 766
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: T-Lora v2 1.6 goes into boot-loop #2084
Comments
I suspect this has to do with the heap leak. |
This one stays in loop until RESET is pressed |
i saw the same on my t-beam but running 2.0.10 fixed it for me. |
OK, I will give 2.0.10 a try. |
Sorry to say, but it ended up in boot-loop again with Meshtastic hwvendor=3, swver=2.0.10.e09b12c-d |
Does removing the BME sensor have any affect? |
Thought about that as well as it loops after I2-init. I have normally "power.isPowerSaving true" and switching this off seems to help, but it will of course shorten batterytime. I just soldered an elko over VCC-GND next to BME680 - did not help. Next now is to disconnect BME680. Let's see |
@karamo Earlier versions used to print out heap-usage when there was a change, but I have not seen this on console output since versions. I saw that the heap grew, but then it also released space so I never saw running out of heap. So, I am still concentrating on finding out how that externally connected BME680 might cause the boot-loop. The system remains in this boot-loop until RESET is pressed and I would expect a boot to clean up a heap-leak like you have implemented with WDT |
@thebentern It ended up in boot-loop also without BME680. |
@thebentern So, this time it happened faster than I hoped: |
@rpsainio the heap status is off on release builds. if you grab any build artefact from our CI that is not a release you will have it back. |
this is most likely a stuck core. "Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1)." 2 seconds into the boot suggests so. Because from a cold boot we only use CPU0. The second core is only enabled when threading starts. That would be the codec2 thread (if enabled) or the webserver. To my knowledge the rest of the firmware does not use FreeRTOS threads. |
It started by asserting esp_sleep_wakeup_cause_t doLightSleep(uint64_t) sleep.cpp:308 (esp_light_sleep_start() == ESP_OK) |
This is a standalone node and should not have codec2 or webserver enabled. Besides Core1 panics only after reboot, so is there some missing init for CPU1 ? |
i have decoded the coredumps... First one is straightforward...
|
Second is way more elusive.
|
and third one is actually not the same. Guru Meditation Error: Core 1 panic'ed (Interrupt wdt timeout on CPU1).
|
this really looks if CPU1 is not initialized after the assert/reboot. But we don't do that, the IDF does that. Crashes/Dumps are random and show whatever is on the call stack when the WDT triggers. |
going back to the original assert. Building a firmware now that will tell us the actual error code from ESP32. |
can you check https://github.com/meshtastic/artifacts/blob/device/pr/pr2183-firmware-2.0.14.2e0c4a2.zip?raw=true please? - if it asserts again, please post the logfile. |
@caveman99 The logfile is 40M, but here is most likely the interesting part; the return code from esp_light_sleep_start assert failed: esp_sleep_wakeup_cause_t doLightSleep(uint64_t) sleep.cpp:317 (res == ESP_OK) Backtrace: 0x4008448d:0x3ffd88f0 0x400966a1:0x3ffd8910 0x4009c821:0x3ffd8930 0x400f7b8d:0x3ffd8a60 0x400d44c5:0x3ffd8a80 0x400fa7cd:0x3ffd8aa0 0x400de239:0x3ffd8ac0 0x400d58df:0x3ffd8ae0 0x400fb121:0x3 ELF file SHA256: 8842e7ec291e84ff E (24421) esp_core_dump_flash: Core dump flash config is corrupted! CRC=0x7bd5c66f instead of 0x0 rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) |
@karamo in fact there was also a reboot caused by heap-leak (seems to me), but the original problem lies still in entering LS |
@caveman99 I was not able to find out how you implemented the printing of return code as the value 1 looks something I cannot verify from documentation. Looking into esp_sleep.h tells that the function is supposed to return ESP_ERR_INVALID_STATE (is defined as 0x0103 ), OTOH online documentation claims that it will return other values starting with ESP_ERR_SLEEP... So, if you print only first byte then it might match or mask by 0xff. Supposing ESP_ERR_INVALID_STATE as value, then it is caused by BT or WIFI not being shutdown. |
@caveman99 UPDATE: I am now testing a way to solve this problem, but I need to let it run several days - So, I let you know then |
@caveman99
Then if there is nothing else going the timer returns and here starts the funny part
So, we flash the LED for 1ms.... I tested this by commenting out lines below. // setLed(true); // briefly turn on led I also replaced the original assert with code that prints out returnvalue of esp_light_sleep_start() |
This issue has been mentioned on Meshtastic. There might be relevant details there: https://meshtastic.discourse.group/t/clarification-on-router-client-mode-app-visibility/7208/2 |
@rpsainio good catch. Interesting part is, that code is 3 years old (!) Maybe a regression or behaviour change from the newer ESP-IDF I'll change that to a 100ms flash for testing. |
Could you test the firmware from that Pull Request? |
@caveman99 just flashed and started again |
@caveman99 It has been running now 47.000 seconds and earlier it failed around 2.000 - 6.000 seconds. No assert or esp_light_sleep_start messages so time to close this now. BTW, as a side effect I can tell that heap consumption is also OK Total heap used:-19976 |
Category
Other
Hardware
T-Lora v2 1.6
Firmware Version
2.0.8.090e166-d
Description
Every now and then T-Lora v2 1.6 gets stuck in boot loop and only pressing the reset-button makes it tick again. This one has extra BME680 sensor connected so it is not just a plain board. Normally it runs for days without a hitch.
I have two similar nodes (with BME680) and they both behave the same way.
Relevant log output
The text was updated successfully, but these errors were encountered: