use of __noinit with ecc memory hangs system

[cfriedt]

Describe the bug
The __noinit attribute is used primarily for the purposes listed below:

1. to preserve a section of SRAM across reboots
2. to speedup init time by reducing the size of .bss

Some configurations may also use __noinit to skip initialization of memory regions initialized by other application-specific hardware.

There have been two identified areas where memory regions marked as __noinit are read-from before they are written-to - specifically, the shell and logging subsystems. Issuing a read before a write might be reasonable in certain scenarios, but for some systems with ECC-enabled memory controllers, a read operation before a write operation will cause the memory controller to issue a fault, hanging the system.

To Reproduce

1. obtain a system with ECC memory and an ECC-capable memory controller
2. enabling logging (with deferred mode) and / or the shell
3. log messages
4. observe hang

In the above cases, the system is completely hung before the boot banner was even printed.

Expected behavior
The system is not hung.

Impact
Showstopper

Logs and console output
There are no logs, nor is there any output on the console. It may be possible to find a stacktrace.

Environment (please complete the following information):

• OS: Zephyr v2.6.0
• Toolchain: crosstool-ng / riscv64
• Commit SHA or Version used: 79a6c07

Additional context
This problem cannot be reproduced on non-ECC systems, and specifically, the ECC memory controller must also be configured to check SRAM memory regions. It is not obvious if there are any readily-available Zephyr community boards that include an ECC memory controller. This may highlight previously unknown race conditions, but those fixes may be considerably more involved. Therefore, the linked PR is mainly for mitigation purposes.

[evgeniy-paltsev]

Hi @cfriedt

There have been two identified areas where memory regions marked as __noinit are read-from before they are written-to - specifically, the shell and logging subsystems.

Do you know exact places where it happens? I can't imagine the reason for shell / logging for reading from __noinit before writing to it.

[cfriedt]

Hi @cfriedt

There have been two identified areas where memory regions marked as __noinit are read-from before they are written-to - specifically, the shell and logging subsystems.

Do you know exact places where it happens? I can't imagine the reason for shell / logging for reading from __noinit before writing to it.

I can try to find some details (e.g. stack traces) but it may be confidential. Will try my best.

Alternatively, it may be possible to try and trigger faults using MMU / MPU regions.

[galak]

Look at doing something similar to z_bss_zero to just init all of memory for these kinda of systems. Changing __noinit doesn't feel like the right solution.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[cfriedt]

The solution suggested in #48172 is that the solution should be part of a specific memory controller driver implementation.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.