Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Toolchain built for 64bit wide time_t fails to build working hello-world-example (IDFGH-6318) #7980

Closed
sven-hm opened this issue Nov 29, 2021 · 6 comments
Labels
Resolution: Done Issue is done internally Status: Resolved Issue is done internally

Comments

@sven-hm
Copy link

sven-hm commented Nov 29, 2021

Environment

  • Development kit: ESP32 NodeMCU Development Board
  • Module or chip used: ESP32-WROOM-32
  • IDF version: v4.3.1-448-g48997e82c0
  • Build System: Make/Cmake
  • Compiler version: xtensa-esp32-elf-gcc (crosstool-NG esp-2021r2.2-abb32e6-dirty) 8.4.0
  • Operating System: Linux
  • Using an IDE?: No
  • Power Supply: USB

Problem Description

I want to use 64bit wide time_t in an application. Therefore i followed the instructions to compile the Toolchain from Source and removed --enable-newlib-long-time_t from crosstool-NG/samples/xtensa-esp32-elf/crosstool.config leading to the following diff (and a "dirty" compiler version, see above):

diff --git a/samples/xtensa-esp32-elf/crosstool.config b/samples/xtensa-esp32-elf/crosstool.config
index 5fa14eaf..fea33731 100644
--- a/samples/xtensa-esp32-elf/crosstool.config
+++ b/samples/xtensa-esp32-elf/crosstool.config
@@ -30,7 +30,7 @@ CT_NEWLIB_DEVEL_REVISION=""
 CT_NEWLIB_DEVEL_SUBDIR=""
 CT_NEWLIB_DEVEL_BOOTSTRAP=""
 CT_LIBC_NEWLIB_TARGET_CFLAGS=""
-CT_LIBC_NEWLIB_EXTRA_CONFIG_ARRAY="--enable-newlib-long-time_t --enable-newlib-nano-malloc --enable-newlib-retargetable-locking --enable-newlib-iconv"
+CT_LIBC_NEWLIB_EXTRA_CONFIG_ARRAY="--enable-newlib-nano-malloc --enable-newlib-retargetable-locking --enable-newlib-iconv"
 CT_LIBC_NEWLIB_DISABLE_SUPPLIED_SYSCALLS=y
 CT_LIBC_NEWLIB_REENT_SMALL=y
 CT_LIBC_NEWLIB_IO_C99FMT=y
@@ -41,7 +41,7 @@ CT_LIBC_NEWLIB_WIDE_ORIENT=n

 CT_LIBC_NEWLIB_AUX_BUILD=y
 CT_LIBC_NEWLIB_AUX_BUILD_NAME="nano"
-CT_LIBC_NEWLIB_AUX_BUILD_EXTRA_CONFIG_ARRAY="--with-newlib --enable-multilib --disable-newlib-io-c99-formats --disable-newlib-supplied-syscalls --enable-newlib-nano-formatted-io --enable-newlib-reent-small --enable-target-optspace --enable-newlib-long-time_t --enable-newlib-nano-malloc --enable-newlib-retargetable-locking --disable-newlib_wide_orient --enable-newlib-iconv"
+CT_LIBC_NEWLIB_AUX_BUILD_EXTRA_CONFIG_ARRAY="--with-newlib --enable-multilib --disable-newlib-io-c99-formats --disable-newlib-supplied-syscalls --enable-newlib-nano-formatted-io --enable-newlib-reent-small --enable-target-optspace --enable-newlib-nano-malloc --enable-newlib-retargetable-locking --disable-newlib_wide_orient --enable-newlib-iconv"
 CT_LIBC_NEWLIB_AUX_BUILD_TARGET_CFLAGS=""

 # For compatibility with GCC 8, change it when update

When compiling the hello world example with CONFIG_SDK_TOOLCHAIN_SUPPORTS_TIME_WIDE_64_BITS=y in the sdkconfig everything looks fine.

Expected Behavior

I would expect to have the hello world example running as usual.

Actual Behavior

After flashing the application gets stuck in a restart loop:

[...]
[100%] Built target flash
--- Miniterm on /dev/ttyUSB0  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
I (168) esp_image: segment 4: paddr=00033a78 vaddr=40086910 size=049c4h ( 18884) load
I (176) esp_image: segment 5: paddr=00038444 vaddr=50000000 size=00010h (    16) load
I (182) boot: Loaded app from partition at offset 0x10000
I (182) boot: Disabling RNG early entropy source...
I (197) cpu_start: Pro cpu up.
I (197) cpu_start: Starting app cpu, entry point is 0x40081044
I (0) cpu_start: App cpu up.
I (211) cpu_start: Pro cpu start user code
I (211) cpu_start: cpu freq: 160000000
I (211) cpu_start: Application information:
I (216) cpu_start: Project name:     hello-world
I (221) cpu_start: App version:      v4.3.1-448-g48997e82c0
I (227) cpu_start: Compile time:     Nov 29 2021 17:58:51
I (233) cpu_start: ELF file SHA256:  49bbf3c4f1676a0c...
I (239) cpu_start: ESP-IDF:          v4.3.1-448-g48997e82c0
I (246) heap_init: Initializing. RAM available for dynamic allocation:
I (253) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (259) heap_init: At 3FFB3208 len 0002CDF8 (179 KiB): DRAM
I (265) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (271) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (278) heap_init: At 4008B2D4 len 00014D2C (83 KiB): IRAM
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:7360
load:0x40078000,len:14912
load:0x40080400,len:3688
entry 0x4008067c
I (27) boot: ESP-IDF v4.3.1-448-g48997e82c0 2nd stage bootloader
I (27) boot: compile time 17:58:35
I (28) boot: chip revision: 1
I (32) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (39) boot.esp32: SPI Speed      : 40MHz
I (43) boot.esp32: SPI Mode       : DIO
I (48) boot.esp32: SPI Flash Size : 2MB
I (52) boot: Enabling RNG early entropy source...
I (58) boot: Partition Table:
I (61) boot: ## Label            Usage          Type ST Offset   Length
I (69) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (76) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (84) boot:  2 factory          factory app      00 00 00010000 00100000
I (91) boot: End of partition table
I (95) boot_comm: chip revision: 1, min. application chip revision: 0
I (102) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=06da8h ( 28072) map
I (121) esp_image: segment 1: paddr=00016dd0 vaddr=3ffb0000 size=02930h ( 10544) load
I (126) esp_image: segment 2: paddr=00019708 vaddr=40080000 size=06910h ( 26896) load
I (139) esp_image: segment 3: paddr=00020020 vaddr=400d0020 size=13a50h ( 80464) map
I (168) esp_image: segment 4: paddr=00033a78 vaddr=40086910 size=049c4h ( 18884) load
I (176) esp_image: segment 5: paddr=00038444 vaddr=50000000 size=00010h (    16) load
I (182) boot: Loaded app from partition at offset 0x10000
I (182) boot: Disabling RNG early entropy source...
I (197) cpu_start: Pro cpu up.
I (197) cpu_start: Starting app cpu, entry point is 0x40081044
I (0) cpu_start: App cpu up.
I (211) cpu_start: Pro cpu start user code
I (211) cpu_start: cpu freq: 160000000
I (211) cpu_start: Application information:
I (216) cpu_start: Project name:     hello-world
I (221) cpu_start: App version:      v4.3.1-448-g48997e82c0
I (227) cpu_start: Compile time:     Nov 29 2021 17:58:51
I (233) cpu_start: ELF file SHA256:  49bbf3c4f1676a0c...
I (239) cpu_start: ESP-IDF:          v4.3.1-448-g48997e82c0
I (246) heap_init: Initializing. RAM available for dynamic allocation:
I (253) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (259) heap_init: At 3FFB3208 len 0002CDF8 (179 KiB): DRAM
I (265) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (272) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (278) heap_init: At 4008B2D4 len 00014D2C (83 KiB): IRAM
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
[...]

Steps to reproduce

  1. Setup Linux Toolchain from Scratch using using crosstool-NG on branch esp-1.24.x (abb32e6) with the above mentioned patch (or remove --enable-newlib-long-time_t as described in the docs).
  2. Checkout latest "hello-world" example, set CONFIG_SDK_TOOLCHAIN_SUPPORTS_TIME_WIDE_64_BITS=y, compile with toolchain from step 1 and flash.

Additional notes

  • when i build the toolchain without removing --enable-newlib-long-time_t and use it for building the hello world example, everything works as expected

Thanks in advance!

@espressif-bot espressif-bot added the Status: Opened Issue is new label Nov 29, 2021
@github-actions github-actions bot changed the title Toolchain built for 64bit wide time_t fails to build working hello-world-example Toolchain built for 64bit wide time_t fails to build working hello-world-example (IDFGH-6318) Nov 29, 2021
@itadori-1
Copy link

I tried "https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/get-started/linux-setup-scratch.html"
it same problem.when it set to "64bit time_t",but it cloudn't start the hello_world application.reboot every 9 second.
please see my log:
ets Jun 8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:6756
load:0x40078000,len:14796
load:0x40080400,len:3792
0x40080400: _init at ??:?
entry 0x40080694
I (27) boot: ESP-IDF v5.0-dev-957-g6cb6087dd0-dirty 2nd stage bootloader
I (28) boot: compile time 15:57:29
I (29) boot: chip revision: 1
I (32) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (39) boot.esp32: SPI Speed : 40MHz
I (44) boot.esp32: SPI Mode : DIO
I (49) boot.esp32: SPI Flash Size : 2MB
I (53) boot: Enabling RNG early entropy source...
I (59) boot: Partition Table:
I (62) boot: ## Label Usage Type ST Offset Length
I (69) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (77) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (84) boot: 2 factory factory app 00 00 00010000 00100000
I (92) boot: End of partition table

I (69) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (77) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (84) boot: 2 factory factory app 00 00 00010000 00100000
I (92) boot: End of partition table
I (96) boot_comm: chip revision: 1, min. application chip revision: 0
I (103) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=0796ch ( 31084) map
I (123) esp_image: segment 1: paddr=00017994 vaddr=3ffb0000 size=0243ch ( 9276) load
I (127) esp_image: segment 2: paddr=00019dd8 vaddr=40080000 size=06240h ( 25152) load
I (141) esp_image: segment 3: paddr=00020020 vaddr=400d0020 size=149fch ( 84476) map
I (172) esp_image: segment 4: paddr=00034a24 vaddr=40086240 size=0500ch ( 20492) load
I (181) esp_image: segment 5: paddr=00039a38 vaddr=50000000 size=00010h ( 16) load
I (187) boot: Disabling RNG early entropy source...
I (203) cpu_start: Pro cpu up.
I (203) cpu_start: Starting app cpu, entry point is 0x40081004
0x40081004: call_start_cpu1 at /home/npikachu/esp/esp-idf/components/esp_system/port/cpu_start.c:152

I (0) cpu_start: App cpu up.
I (217) cpu_start: Pro cpu start user code
I (217) cpu_start: cpu freq: 160000000 Hz
I (217) cpu_start: Application information:
I (222) cpu_start: Project name: hello_world
I (227) cpu_start: App version: 1
I (232) cpu_start: Compile time: Jan 5 2022 15:57:15
I (238) cpu_start: ELF file SHA256: 7d70f879a8b69dfa...
I (244) cpu_start: ESP-IDF: v5.0-dev-957-g6cb6087dd0-dirty
I (251) heap_init: Initializing. RAM available for dynamic allocation:
I (258) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (264) heap_init: At 3FFB2D30 len 0002D2D0 (180 KiB): DRAM
I (270) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (277) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (283) heap_init: At 4008B24C len 00014DB4 (83 KiB): IRAM
[after 9second repeat forever]
how to resolve?

@espressif-bot espressif-bot added Status: In Progress Work is in progress Status: Reviewing Issue is being reviewed and removed Status: Opened Issue is new Status: In Progress Work is in progress labels Jan 6, 2022
@igrr
Copy link
Member

igrr commented Jan 6, 2022

Thank you for reporting @sven-hm @itadori-1. We found the issue and will soon fix it in master and release branches.

The issue is caused by mismatch of struct stat size between IDF code and the ROM. After switching to 64-bit time_t, IDF code expects larger size of struct stat, but some parts of ROM code assume the original size of struct stat which was valid for 32-bit time_t. This mismatch causes stack corruption which (luckily) caused a crash in this situation.

Here is the patch which can be applied to master branch.
7980.zip

Instead of applying the patch, you may also manually remove the following lines from components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld:

_isatty_r = 0x40000ea0;
__sfvwrite_r = 0x4005893c;
__smakebuf_r = 0x40059108;
__srefill_r = 0x400593d4;
__swsetup_r = 0x40058cc8;

@sven-hm
Copy link
Author

sven-hm commented Jan 11, 2022

Thanks @igrr for the fix! I can confirm that the suggested fix works with the example code.

dskulina pushed a commit to playable-tech/esp-idf that referenced this issue Feb 4, 2022
...and all their callers.

With the upcoming switch from sizeof(time_t)==4 to sizeof(time_t)==8,
sizeof(struct stat) is also increasing.

A few newlib functions present in ROM allocate 'struct stat' on the
stack and call _fstat_r on this structure. The implementation of
fstat is provided in ESP-IDF. This implementation will often do
memset(st, 0, sizeof(*st)), where st is 'struct stat*', before setting
some fields of this structure. If IDF is built with sizeof(st)
different from sizeof(st) which ROM was built with, this will lead
to an out-of-bounds write and a stack corruption.

This commit removes problematic ROM functions from the linker script.
Here are the functions which allocate 'struct stat':
* _isatty_r (in ROM)
* __swhatbuf_r, called by __smakebuf_r, called by __swsetup_r and
  __srefill_r (in ROM)
* _fseeko_r (not in ROM)
* glob2 (not in ROM)
* _gettemp (not in ROM)

As a result, these functions are used from libc.a, and use correct
size of 'stat' structure.

Closes espressif#7980
@espressif-bot espressif-bot added Resolution: Done Issue is done internally Status: Resolved Issue is done internally and removed Status: Reviewing Issue is being reviewed labels Feb 4, 2022
dskulina pushed a commit to playable-tech/esp-idf that referenced this issue Feb 5, 2022
...and all their callers.

With the upcoming switch from sizeof(time_t)==4 to sizeof(time_t)==8,
sizeof(struct stat) is also increasing.

A few newlib functions present in ROM allocate 'struct stat' on the
stack and call _fstat_r on this structure. The implementation of
fstat is provided in ESP-IDF. This implementation will often do
memset(st, 0, sizeof(*st)), where st is 'struct stat*', before setting
some fields of this structure. If IDF is built with sizeof(st)
different from sizeof(st) which ROM was built with, this will lead
to an out-of-bounds write and a stack corruption.

This commit removes problematic ROM functions from the linker script.
Here are the functions which allocate 'struct stat':
* _isatty_r (in ROM)
* __swhatbuf_r, called by __smakebuf_r, called by __swsetup_r and
  __srefill_r (in ROM)
* _fseeko_r (not in ROM)
* glob2 (not in ROM)
* _gettemp (not in ROM)

As a result, these functions are used from libc.a, and use correct
size of 'stat' structure.

Closes espressif#7980
@sven-hm
Copy link
Author

sven-hm commented Feb 7, 2022

@igrr in which release do you plan to include the fix?

@igrr
Copy link
Member

igrr commented Feb 7, 2022

@sven-hm The fix is currently in master, so will be included in v5.0 release. v5.0 will switch to 64-bit time_t as the default option.

I can also backport this to release/v4.4 and release/v4.3 branches, in this case the fix will appear in v4.4.1 and v4.3.3, respectively.

@sven-hm
Copy link
Author

sven-hm commented Feb 7, 2022

Ok, nice! Thank you, @igrr !

espressif-bot pushed a commit that referenced this issue Feb 24, 2022
...and all their callers.

With the upcoming switch from sizeof(time_t)==4 to sizeof(time_t)==8,
sizeof(struct stat) is also increasing.

A few newlib functions present in ROM allocate 'struct stat' on the
stack and call _fstat_r on this structure. The implementation of
fstat is provided in ESP-IDF. This implementation will often do
memset(st, 0, sizeof(*st)), where st is 'struct stat*', before setting
some fields of this structure. If IDF is built with sizeof(st)
different from sizeof(st) which ROM was built with, this will lead
to an out-of-bounds write and a stack corruption.

This commit removes problematic ROM functions from the linker script.
Here are the functions which allocate 'struct stat':
* _isatty_r (in ROM)
* __swhatbuf_r, called by __smakebuf_r, called by __swsetup_r and
  __srefill_r (in ROM)
* _fseeko_r (not in ROM)
* glob2 (not in ROM)
* _gettemp (not in ROM)

As a result, these functions are used from libc.a, and use correct
size of 'stat' structure.

Closes #7980
espressif-bot pushed a commit that referenced this issue Mar 4, 2022
...and all their callers.

With the upcoming switch from sizeof(time_t)==4 to sizeof(time_t)==8,
sizeof(struct stat) is also increasing.

A few newlib functions present in ROM allocate 'struct stat' on the
stack and call _fstat_r on this structure. The implementation of
fstat is provided in ESP-IDF. This implementation will often do
memset(st, 0, sizeof(*st)), where st is 'struct stat*', before setting
some fields of this structure. If IDF is built with sizeof(st)
different from sizeof(st) which ROM was built with, this will lead
to an out-of-bounds write and a stack corruption.

This commit removes problematic ROM functions from the linker script.
Here are the functions which allocate 'struct stat':
* _isatty_r (in ROM)
* __swhatbuf_r, called by __smakebuf_r, called by __swsetup_r and
  __srefill_r (in ROM)
* _fseeko_r (not in ROM)
* glob2 (not in ROM)
* _gettemp (not in ROM)

As a result, these functions are used from libc.a, and use correct
size of 'stat' structure.

Closes #7980
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Resolution: Done Issue is done internally Status: Resolved Issue is done internally
Projects
None yet
Development

No branches or pull requests

4 participants