[Kernel][Initialization] MSP and PSP Stack Conflict in z_interrupt_stacks Causes Memory Corruption

[FixJA]

Describe the bug
During early Zephyr initialization, the MSP (Main Stack Pointer) is set to the top of z_interrupt_stacks. However, the PSP (Process Stack Pointer) used by z_cstart already consumes part of z_interrupt_stacks. When an interrupt is triggered during z_sys_init_run_level(INIT_LEVEL_PRE_KERNEL_2), the MSP stack overwrites the PSP stack content, corrupting local variables in z_sys_init_run_level. This leads to memory out-of-bounds access and a chip exception.

here is my debug infomation

bufferIndex initial value is 1

An interrupt triggers the use of an MSP stack pointer, and the MSP pointer points to something that is already in use by the PSP pointer, causing the PSP stack to be corrupted

bufferIndex value is charged

Environment (please complete the following information):

• OS: linux
• Toolchain Zephyr SDK 0.17
• Zephyr OS build v4.1.0-2729-g785d0bbfb3fc
• board: mimxrt1020_evk

Additional context
prj.conf

# Copyright (c) 2021 Nordic Semiconductor ASA
# SPDX-License-Identifier: Apache-2.0
#
# This file contains selected Kconfig options for the application.

# zephyr printf
CONFIG_SERIAL=y
CONFIG_CONSOLE=y
CONFIG_UART_CONSOLE=y
CONFIG_PRINTK=y
CONFIG_INIT_STACKS=y
CONFIG_SHELL=y
CONFIG_CONSOLE_SUBSYS=y
CONFIG_SHELL_BACKEND_SERIAL=y
CONFIG_REQUIRES_FULL_LIBC=y
CONFIG_POSIX_API=y

CONFIG_GPIO=y
CONFIG_BLINK=y

#network
CONFIG_NETWORKING=y
CONFIG_NET_IPV4=y
CONFIG_NET_ARP=y
CONFIG_NET_TCP=y
CONFIG_NET_UDP=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_DHCPV4=y
CONFIG_NET_DHCPV4_OPTION_CALLBACKS=y
CONFIG_NET_CONTEXT_RCVTIMEO=y
CONFIG_NET_CONTEXT_SNDTIMEO=y
CONFIG_DNS_RESOLVER=y
CONFIG_NET_TCP_MAX_RECV_WINDOW_SIZE=5120

CONFIG_NET_PKT_TX_COUNT=6
CONFIG_NET_BUF_RX_COUNT=64

CONFIG_NET_MGMT=y
CONFIG_NET_MGMT_EVENT=y
CONFIG_NET_CONNECTION_MANAGER=y
CONFIG_NET_CONNECTION_MANAGER_MONITOR_STACK_SIZE=2048
CONFIG_SLIP_STATISTICS=n

# shell
CONFIG_NET_SHELL=y
CONFIG_SHELL_PROMPT_UART="rt1020:~$ "
CONFIG_SHELL_BACKEND_SERIAL_TX_RING_BUFFER_SIZE=128
CONFIG_SHELL_BACKEND_SERIAL_RX_RING_BUFFER_SIZE=256

CONFIG_MAIN_STACK_SIZE=4096
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_NET_TCP_WORKQ_STACK_SIZE=4096
CONFIG_NET_RX_STACK_SIZE=4096
CONFIG_NET_SOCKETS_SERVICE_STACK_SIZE=2048
CONFIG_NET_MGMT_EVENT_STACK_SIZE=2048
CONFIG_ISR_STACK_SIZE=4096

debug.conf

# compiler
CONFIG_NO_OPTIMIZATIONS=y

# logging
CONFIG_LOG=y
CONFIG_NET_LOG=y
CONFIG_APP_LOG_LEVEL_DBG=y
CONFIG_GPIO_LOG_LEVEL_DBG=y
CONFIG_ETHERNET_LOG_LEVEL_DBG=y
#debug
CONFIG_THREAD_MONITOR=y

CONFIG_STACK_SENTINEL=y
CONFIG_ASSERT=y

CONFIG_TRACING=y
CONFIG_PERCEPIO_TRACERECORDER=y
CONFIG_PERCEPIO_TRC_START_MODE_START_FROM_HOST=y
CONFIG_PERCEPIO_TRC_CFG_STREAM_PORT_RTT=y
CONFIG_PERCEPIO_TRC_CFG_CTRL_TASK_STACK_SIZE=4096

[github-actions]

Hi @FixJA! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

[danieldegrasse]

@FixJA do you have any other platforms running Zephyr that also exhibit this issue?

[FixJA]

I only have iMXRT series boards on hand, and this is my first time using Zephyr. I’ve noticed that this issue occurs when multiple component features are enabled.

[danieldegrasse]

I only have iMXRT series boards on hand, and this is my first time using Zephyr. I’ve noticed that this issue occurs when multiple component features are enabled.

I see you're using Ozone as a debugger- can you reproduce the issue outside of Ozone? I know there are have previously been issues with the default reset strategy Ozone uses that did not initialize stack pointers properly

[FixJA]

I checked the Ozone configuration file, and the initial stack settings were problematic. I reviewed the MAP file to set the correct stack pointer, but the issue persisted. When I downloaded the program and reset the chip without connecting the debugger, the program still did not work properly.

[FixJA]

I changed the location of the interrupt stack initialization code, and the program started working properly.

[dleach02]

@andyross @peter-mitsis @danieldegrasse

I did a bit of digging on this since it was assigned to me. @FixJA has identified an issue. Here is the flow from reset:

• reset.S:
  • msp = z_main_stack
  • psp = z_interrupt_stacks
  • processor is switched to use psp for Thread mode
  • z_prep_c:
    • bss_zero/data_copy
    • z_arm_interrupt_init
    • z_cstart:
      • z_sys_init_run_level(INIT_LEVEL_EARLY)
      • arch_kernel_init()
        • z_arm_interrupt_stack_setup
          • msp = z_interrupt_stack <-- now both msp and psp are pointing to the same stack area
      • z_sys_init_run_level(INIT_LEVEL_PRE_KERNEL_1)
      • z_sys_init_run_level(INIT_LEVEL_PRE_KERNEL_2)
      • switch_to_main_thread(prepare_multithreading()) <--- sets PSP to thread stack which is actually z_main_stack

After the call to z_arm_interrupt_stack_setup we find the system in a state of msp & psp both pointing to the z_interrupt_stacks. The looping calls of z_sys_init_run_level() for PRE_KERNEL_1 and PRE_KERNEL_2 now have their psp based stack exposed to interrupt handling.

Couple of questions:

1. Is there an expectation of interrupt enable status in Zephyr's initialization sequence?

   • I say this because I see we have an explicit call to arch_irq_unlock_outlined() in arch_switch_to_main_thread() so I'm wondering if we expect no interrupts to occur in the system until we get to this point. If it is the case that we don't expect interrupts to be enabled during this phase then we need to investigate each of the initialization routines in LEVEL_EARLY/PRE_KERNEL_1/2 to see if any of them enable interrupts.

2. Couldn't the psp be initialized in reset.S to z_main_stack? The z_main_stack won't be reused until switch_to_main_thread() after the prepare_multithreading() sets it to bg_thread_main thread. Code execution is not expected to return to the code after the call to prepare_multithreading() so call stack at this point is not needed.

I changed the location of the interrupt stack initialization code, and the program started working properly.

@FixJA patch above moving the z_arm_interrupt_stack_setup() to before the call of switch_to_main_thread() works in this situation because it allowed the interrupt handling to use the pre-configured msp z_main_stack, which is unused until switch_to_main_thread(). But the system is still exposed to the issue because you have to make a call to prepare_multithreading() inside the call to switch_to_main_thread().