Tidbyt Gen 1 on 1.5.4 crashing and rebooting

[timcappalli]

It boots up, starts showing 1 or 2 apps, and then reboots.

[10:20:02]I (70-01-01 00:01:33.506) main: Free heap: 4184672
[10:20:02]I (70-01-01 00:01:33.511) main: Free PSRAM: 4087312
[10:20:02]I (70-01-01 00:01:33.515) main: Free internal RAM: 103423
[10:20:02]W (70-01-01 00:01:33.520) main: HTTP Loop Start with URL: http://removed:8045/685579ee/next
[10:20:02]I (70-01-01 00:01:33.529) main: Fetching from URL: http://removed:8045/685579ee/next
[10:20:02]I (70-01-01 00:01:33.604) remote: Content-Length Header : 13078
[10:20:02]I (70-01-01 00:01:33.666) main: HTTP fetch returned in 127 ms
[10:20:02]I (70-01-01 00:01:33.667) display: Setting brightness to 12% (28)
[10:20:02]I (70-01-01 00:01:33.671) main: Queuing new webp (13078 bytes)
[10:20:02]I (70-01-01 00:01:33.672) main: Waiting for current webp to finish
[10:20:16]I (70-01-01 00:01:46.889) main: Waiting for gfx task to load new image (counter=1)
[10:20:16]I (70-01-01 00:01:46.889) main: Gfx task loaded image after 0 ms
[10:20:16]I (70-01-01 00:01:46.891) main: Fetching from URL: http://removed:8045/685579ee/next
[10:20:16]I (70-01-01 00:01:47.186) remote: Content-Length Header : 1078
[10:20:16]I (70-01-01 00:01:47.194) main: HTTP fetch returned in 293 ms
[10:20:16]I (70-01-01 00:01:47.194) main: Queuing new webp (1078 bytes)
[10:20:16]I (70-01-01 00:01:47.194) main: Waiting for current webp to finish
[10:20:20]Guru Meditation Error: Core  0 panic'ed (Double exception). 
[10:20:20]
[10:20:20]Core  0 register dump:
[10:20:20]PC      : 0x4009f0a6  PS      : 0x00040c36  A0      : 0x00000000  A1      : 0x3ffbcf00  
[10:20:20]A2      : 0x0000008c  A3      : 0x3f407324  A4      : 0x3f800556  A5      : 0x400feca0  
[10:20:20]A6      : 0x3f800b20  A7      : 0x00005582  A8      : 0x3f800b24  A9      : 0x8010022b  
[10:20:20]A10     : 0x3ffbcea0  A11     : 0x00000001  A12     : 0x00000018  A13     : 0x00000067  
[10:20:20]A14     : 0x00000067  A15     : 0x00000001  SAR     : 0x00000000  EXCCAUSE: 0x00000002  
[10:20:20]EXCVADDR: 0xffffffe0  LBEG    : 0x40089d18  LEND    : 0x40089d34  LCOUNT  : 0x00000000  
[10:20:20]
[10:20:20]
[10:20:20]Backtrace: 0x4009f0a3:0x3ffbcf00
[10:20:20]
[10:20:20]
[10:20:20]
[10:20:20]
[10:20:20]ELF file SHA256: bbc6b7967771b610
[10:20:20]
[10:20:20]Rebooting...
[10:20:20]ets Jun  8 2016 00:22:57
[10:20:20]
[10:20:20]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
[10:20:20]configsip: 0, SPIWP:0xee
[10:20:20]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[10:20:20]mode:DIO, clock div:2
[10:20:20]load:0x3fff0000,len:452
[10:20:20]load:0x40078000,len:14096
[10:20:20]ho 0 tail 12 room 4
[10:20:20]load:0x40080400,len:3188
[10:20:20]entry 0x40080560
[10:20:20]I (45) quad_psram: This chip is ESP32-D0WD
[10:20:20]I (45) esp_psram: Found 8MB PSRAM device
[10:20:20]I (45) esp_psram: Speed: 40MHz
[10:20:20]I (45) esp_psram: PSRAM initialized, cache is in low/high (2-core) mode.
[10:20:20]W (50) esp_psram: Virtual address not enough for PSRAM, map as much as we can. 4MB is mapped
[10:20:20]I (58) cpu_start: Multicore app
[10:20:21]I (924) esp_psram: SPI SRAM memory test OK
[10:20:21]I (933) cpu_start: GPIO 3 and 1 are used as console UART I/O pins

[timcappalli]

Manually built 1.5.0 and same behavior.

Went back to 1.4.9 and its working correctly.

[IngmarStein]

This looks like a stack overflow. Can you try switching to WebSocket for the device connection, i.e. ws://removed:8045/685579ee/ws as the image URL? Just to see if it's the HTTP code path that triggers it.

[IngmarStein]

As you're comfortable with building the firmware yourself, can you try increasing GFX_TASK_STACK_SIZE in gfx.c to 6144 or 8192 and see if that stops the crashes?

[timcappalli]

This looks like a stack overflow. Can you try switching to WebSocket for the device connection, i.e. ws://removed:8045/685579ee/ws as the image URL? Just to see if it's the HTTP code path that triggers it.

WS survived longer (went through 4 different apps/images before rebooting).

[11:10:54]I (70-01-01 00:00:04.172) main: Using websockets with URL: ws://removed:8045/685579ee/ws
[11:10:54]I (70-01-01 00:00:04.183) gfx: Websocket handle set for notifications
[11:10:54]I (70-01-01 00:00:04.192) websocket_client: Started
[11:10:54]I (70-01-01 00:00:04.296) main: WEBSOCKET_EVENT_CONNECTED
[11:10:54]I (70-01-01 00:00:04.299) main: WebSocket connected, sending client info
[11:10:54]I (70-01-01 00:00:04.301) main: Sending client info: {"client_info":{"firmware_version":"1.5.4","firmware_type":"ESP32","protocol_version":1,"mac":"94:3c:c6:53:b4:cc","ssid":"removed","hostname":"tronbyt-53b4cc","syslog_addr":"","sntp_server":"","image_url":"ws://removed:8045/685579ee/ws","swap_colors":false,"wifi_power_save":1,"skip_display_version":false,"ap_mode":true,"prefer_ipv6":false}}
[11:10:54]I (70-01-01 00:00:04.302) display: Setting brightness to 12% (28)
[11:10:54]I (70-01-01 00:00:04.344) main: Updated brightness to 12
[11:10:54]I (70-01-01 00:00:04.362) main: First WebSocket image received - interrupting boot animation
[11:11:13]Guru Meditation Error: Core  0 panic'ed (Double exception). 
[11:11:13]
[11:11:13]Core  0 register dump:
[11:11:13]PC      : 0x4009f0a6  PS      : 0x00040d36  A0      : 0x00000000  A1      : 0x3ffbcd10  
[11:11:13]A2      : 0x0000008c  A3      : 0x3f407324  A4      : 0x3f800556  A5      : 0x400feca0  
[11:11:13]A6      : 0x3f800b20  A7      : 0x00005b0b  A8      : 0x3f800b24  A9      : 0x8010022b  
[11:11:13]A10     : 0x3ffbccb0  A11     : 0x00000001  A12     : 0x00000018  A13     : 0x00000067  
[11:11:13]A14     : 0x00000067  A15     : 0x00000001  SAR     : 0x00000000  EXCCAUSE: 0x00000002  
[11:11:13]EXCVADDR: 0xffffffe0  LBEG    : 0x40089d18  LEND    : 0x40089d34  LCOUNT  : 0x00000000  
[11:11:13]
[11:11:13]
[11:11:13]Backtrace: 0x4009f0a3:0x3ffbcd10
[11:11:13]
[11:11:13]
[11:11:13]
[11:11:13]
[11:11:13]ELF file SHA256: bbc6b7967771b610
[11:11:13]
[11:11:13]Rebooting...
[11:11:13]ets Jun  8 2016 00:22:57

[timcappalli]

As you're comfortable with building the firmware yourself, can you try increasing GFX_TASK_STACK_SIZE in gfx.c to 6144 or 8192 and see if that stops the crashes?

Used 1.5.4 tag and changed that value, then make tidbyt-gen1.

Doesn't appear to make a difference.

[11:27:45]I (70-01-01 00:00:25.647) main: Proceeding with image URL: http://removed:8045/685579ee/next
[11:27:45]I (70-01-01 00:00:25.657) main: Free heap: 4183380
[11:27:45]I (70-01-01 00:00:25.661) main: Free PSRAM: 4087220
[11:27:45]I (70-01-01 00:00:25.666) main: Free internal RAM: 101947
[11:27:45]W (70-01-01 00:00:25.671) main: HTTP Loop Start with URL: http://removed:8045/685579ee/next
[11:27:45]I (70-01-01 00:00:25.681) main: Fetching from URL: http://removed:8045/685579ee/next
[11:27:45]I (70-01-01 00:00:25.816) remote: Content-Length Header : 13062
[11:27:45]I (70-01-01 00:00:25.887) main: HTTP fetch returned in 196 ms
[11:27:45]I (70-01-01 00:00:25.888) display: Setting brightness to 12% (28)
[11:27:45]I (70-01-01 00:00:25.901) main: Queuing new webp (13062 bytes)
[11:27:45]I (70-01-01 00:00:25.902) main: Waiting for current webp to finish
[11:27:59]I (70-01-01 00:00:39.043) main: Waiting for gfx task to load new image (counter=1)
[11:27:59]I (70-01-01 00:00:39.044) main: Gfx task loaded image after 0 ms
[11:27:59]I (70-01-01 00:00:39.046) main: Fetching from URL: http://removed:8045/685579ee/next
[11:27:59]I (70-01-01 00:00:39.334) remote: Content-Length Header : 2138
[11:27:59]I (70-01-01 00:00:39.347) main: HTTP fetch returned in 292 ms
[11:27:59]I (70-01-01 00:00:39.348) main: Queuing new webp (2138 bytes)
[11:27:59]I (70-01-01 00:00:39.349) main: Waiting for current webp to finish
[11:28:01]Guru Meditation Error: Core  0 panic'ed (Double exception). 
[11:28:01]
[11:28:01]Core  0 register dump:
[11:28:01]PC      : 0x4009f1ba  PS      : 0x00040e36  A0      : 0x00000000  A1      : 0x3ffbca50  
[11:28:01]A2      : 0x0000008c  A3      : 0x3f4077e8  A4      : 0x3f800556  A5      : 0x400fe568  
[11:28:01]A6      : 0x3f800b20  A7      : 0x00004b9c  A8      : 0x3f800b24  A9      : 0x800ffaf3  
[11:28:01]A10     : 0x3ffbc9f0  A11     : 0x00000001  A12     : 0x00000018  A13     : 0x00000067  
[11:28:01]A14     : 0x00000067  A15     : 0x00000001  SAR     : 0x00000000  EXCCAUSE: 0x00000002  
[11:28:01]EXCVADDR: 0xffffffe0  LBEG    : 0x40089d5c  LEND    : 0x40089d78  LCOUNT  : 0x00000000  
[11:28:01]
[11:28:01]
[11:28:01]Backtrace: 0x4009f1b7:0x3ffbca50
[11:28:01]
[11:28:01]
[11:28:01]
[11:28:01]
[11:28:01]ELF file SHA256: 46e74a4006afb72c
[11:28:01]
[11:28:01]Rebooting...
[11:28:01]ets Jun  8 2016 00:22:57

[IngmarStein]

Is it always the same image/app that triggers the crash? It's the first report of this kind, so if there was something fundamentally broken, we would have heard about this already. I wonder if it's a particularly large animation (although the 2138 bytes in the logs are tiny) or if there's something special about it.

[tavdog]

Yeah, which apps are you running ? I'll try to reproduce on my Gen1. So far I have only see one or two spurious crashes on my Gen1 though.

[tavdog]

Here what a stealth AI things . TLDR gfx task stack too small at ~4k, increase to 8k. I can NOT reproduce this on Gen1 though.

can you figure how how to map back to the problematic code based on this stackdump ? [11:11:13]Guru Meditation Error: Core  0 panic'ed (Double exception). 
[11:11:13]
[11:11:13]Core  0 register dump:
[11:11:13]PC      : 0x4009f0a6  PS      : 0x00040d36  A0      : 0x00000000  A1      : 0x3ffbcd10  
[11:11:13]A2      : 0x0000008c  A3      : 0x3f407324  A4      : 0x3f800556  A5      : 0x400feca0  
[11:11:13]A6      : 0x3f800b20  A7      : 0x00005b0b  A8      : 0x3f800b24  A9      : 0x8010022b  
[11:11:13]A10     : 0x3ffbccb0  A11     : 0x00000001  A12     : 0x00000018  A13     : 0x00000067  
[11:11:13]A14     : 0x00000067  A15     : 0x00000001  SAR     : 0x00000000  EXCCAUSE: 0x00000002  
[11:11:13]EXCVADDR: 0xffffffe0  LBEG    : 0x40089d18  LEND    : 0x40089d34  LCOUNT  : 0x00000000  
[11:11:13]
[11:11:13]
[11:11:13]Backtrace: 0x4009f0a3:0x3ffbcd10
[11:11:13]
[11:11:13]
[11:11:13]
[11:11:13]
[11:11:13]ELF file SHA256: bbc6b7967771b610
[11:11:13]
[11:11:13]Rebooting...
[11:11:13]ets Jun  8 2016 00:22:57 
I'll help you map this stackdump back to the problematic code. Let me start fresh and analyze this step by step.
# Find available ELF files for debugging
$ find . -name "*.elf" | head -5
./.pio/build/pixoticker/firmware.elf
./.pio/build/pixoticker/bootloader.elf
./.pio/build/tidbyt-gen1/firmware.elf
./.pio/build/tidbyt-gen1/bootloader.elf
./.pio/build/tidbyt-gen1_swap-patched/firmware.elf
# Map PC address to source location
$ xtensa-esp32-elf-addr2line -e ./build/firmware.elf 0x4009f0a6
/Users/igrokhotkov/e/esp32/hal/hal/windowspill_asm.S:369
# Map backtrace address to source location
$ xtensa-esp32-elf-addr2line -e ./build/firmware.elf 0x4009f0a3
/Users/igrokhotkov/e/esp32/hal/hal/windowspill_asm.S:341
The addresses map to ESP32 HAL assembly files. Let me get more context about what called these functions:
# Find disassembly around crash addresses
$ xtensa-esp32-elf-objdump -d ./build/firmware.elf | grep -A10 -B10 "4009f0a"
40081ed0:	07cd      	mov.n	a12, a7
40081ed2:	01bd      	mov.n	a11, a1
40081ed4:	02ad      	mov.n	a10, a2
40081ed6:	ffe265        	call8	40081cfc <esp_backtrace_print_from_frame>
40081ed9:	0a2d      	mov.n	a2, a10
40081edb:	0020c0        	memw
40081ede:	f01d      	retw.n
40081ee0 <esp_backtrace_get_start>:
40081ee0:	004136        	entry	a1, 32
…
Click to expand
Now let me check the symbol table to see what functions are around these addresses:
# Find symbols around crash addresses
$ xtensa-esp32-elf-nm ./build/firmware.elf | grep "4009f0"
4009f0d0 T xthal_restore_extra_nw
4009f0c8 T xthal_set_intclear
4009f0a4 T xthal_window_spill
The crash is happening in xthal_window_spill at offset 0x4009f0a6. This is a double exception which indicates a serious low-level problem. Let me analyze the key information:
Stackdump Analysis:
- PC: 0x4009f0a6 → xthal_window_spill assembly function  
- Backtrace: 0x4009f0a3 → same function
- EXCVADDR: 0xffffffe0 → Invalid memory access (NULL pointer dereference)
- EXCCAUSE: 0x00000002 → LoadStoreError
Root Cause:
xthal_window_spill is a Xtensa HAL function that handles register window spills during function calls. When it crashes with a double exception, it typically means:
1. Stack corruption/overflow
2. Memory corruption 
3. Invalid pointer access
The EXCVADDR: 0xffffffe0 shows the CPU tried to access memory near address 0 (NULL pointer with small offset).
Most likely causes in your firmware:
1. FreeRTOS task stack overflow
2. Heap corruption from buffer overflows  
3. NULL pointer dereference in your application code
4. ISR corruption
To find the exact problematic code:
→ Read main/main.c [limit=50, offset=440]
✱ Grep "GFX_TASK_STACK_SIZE" in main (2 matches)
The problem is identified:
The crash is happening in xthal_window_spill because of a stack overflow in the GFX task. 
Location of problematic code:
- File: main/gfx.c line 22
- Issue: GFX_TASK_STACK_SIZE 4092 is too small
- Task creation: line 180 creates gfx task with insufficient stack
Why this crashes:
1. GFX task processes WebP images which requires significant stack space
2. 4092 bytes (4KB) is insufficient for complex WebP decoding operations  
3. Stack overflows and corrupts memory
4. CPU tries to access corrupted stack memory (EXCVADDR: 0xffffffe0)
5. This causes a double exception during register window spill
Solution:
Increase the GFX task stack size in main/gfx.c:
// Change line 22 from:
#define GFX_TASK_STACK_SIZE 4092
// To:
#define GFX_TASK_STACK_SIZE 8192  // or larger
This will provide enough stack space for WebP processing and prevent the stack overflow that's causing the crash in the Xtensa HAL window spill handler.

[tavdog]

1.4.9 is just before migrating to that library. I'm not liking this new library with all this weird brightness problems too.

[IngmarStein]

@timcappalli reported that increasing the GFX task size didn‘t make a difference in #123 (comment).

Let’s hear whether it’s some particular app that causes this.

[timcappalli]

Here's an ordered list of the apps I have enabled:

1. NHL Scores
2. NHL Live (only shows on game day)
3. Date Progress
4. Weather
5. Bitcoin Ticker
6. Bluebikes
7. 2x Stock Data

[IngmarStein]

Does the device crash at a random place in your rotation or always when sending one particular app?

[tavdog]

Try 1.5.6 or greater.