NEORV32 usage for radiation testing

[BEforlin]

Hello community,

Our research group has been using NEORV32 extensively for internal development, research, and education (many, many projects with NEORV).

In the CAES group at the University of Twente, we are focusing heavily on dependability research for RISC-V cores. This includes accelerated radiation beam experiments. These are our first published results with the NEORV32 core where we perform a characterization campaign with it:

Neutron Radiation Tests of the NEORV32 RISC-V SoC on Flash-Based FPGAs

The core integrates very easily with Flash FPGAs from Microsemi, so it was quick and easy to adapt to our needs. These FPGAs are generally considered immune to bit-flips, so only the user memory remains vulnerable. We also deeply appreciate the thorough documentation of the core; it made investigating issues and adapting it so much quicker.

Regarding details about the core's reliability performance, with some ECC on the IMEM and DMEM, the core is very reliable, enough to handle a couple of millennia of natural exposure without major issues. We have made further system-level improvements to include more features and peripherals, but NEORV32 is quickly becoming our work-horse, and we don't plan to stop using it.

[stnolting]

Hey @BEforlin, thank you so much for sharing your experience!

Regarding details about the core's reliability performance, with some ECC on the IMEM and DMEM, the core is very reliable, enough to handle a couple of millennia of natural exposure without major issues.

Your paper is really interesting. I did not expect the core (on rtl level) to withstand even a single radiation-induced event. 😅

I need to have a closer look at your paper (no IEEE access at home). Just one quick thought: one base concept of the core is to identify all kind of illegal conditions (like malformed instructions) during runtime. So I am curious. What is the relative ratio of "core just crashes" and "core raises an exception" when exposed to radiation?

Another quick question: did you also add ECC to the register file? If not, do you think this might be an interesting thing to do in order to improve radiation tolerance? Maybe raising an exception when reading corrupted data from the register file is something we could easily add.

We also deeply appreciate the thorough documentation of the core; it made investigating issues and adapting it so much quicker. [...] NEORV32 is quickly becoming our work-horse, and we don't plan to stop using it.

On behalf of the community: thank you very much for your nice words! 😃

[BEforlin]

Hey @stnolting,

Glad to discuss the results!

So I am curious. What is the relative ratio of "core just crashes" and "core raises an exception" when exposed to radiation?

With the data on the paper, this ratio is 31 detected exceptions to 1 unknown (timeout), however the instruction data was located in flash memory, so it was not vulnerable to upsets itself. We think that most of the exceptions were caused by corrupted pointer addresses. This is supported by the ECC results on the DMEM, the faults plummit.

There is a trade-off when evaluating a core under beam, if you don't protect the memories you will be basically characterizing the memories themselves instead of your target (as they are so much bigger). Testing NEO's recovery mechanisms is one of our current/future projects :)

Another quick question: did you also add ECC to the register file?

Edit of my previous answer: We actually did have ECC on the register file, but I can't easily tell you the error rate it detected. On following experiments we improved the monitoring of the ECC, we are looking to publish those results soon.

Maybe raising an exception when reading corrupted data from the register file is something we could easily add.

This is one of our future work directions, NEORV won't be our only target, but improving error coverage is one of our research goals. How would you figure out what data is corrupted? Would you rely on ECC?

[stnolting]

We think that most of the exceptions were caused by corrupted pointer addresses.

Did you also include the physical memory protection (PMP) ISA extension? Maybe this could be useful to catch some (a lot?) invalid pointers before they access incorrect data.

Testing NEO's recovery mechanisms is one of our current/future projects :)

I'm looking forward to it! 😉

On following experiments we improved the monitoring of the ECC, we are looking to publish those results soon.

Sounds interesting!

I was thinking about adding some custom hardware performance monitors. These might be helpful to benchmark runtime events (like register file ECC faults in this case). What do you thing about this? Would this be a handy extension?

How would you figure out what data is corrupted? Would you rely on ECC?

Relying on ECC could be the first step. Maybe even a simple parity bit for each register file entry might already provide some (sufficient) information about corrupted data (when thinking about _single_event upsets).

RISC-V is about to ratify a defined trap for "hardware errors". Maybe this is also something we could add - at least the infrastructure for that. Custom implementation could use that as a hook to inform the runtime environment that something has gone (terribly) wrong.

This is really a very interesting research project you are working on! I'm already looking forward to further results.

[DS-567]

Hey Stephan,

Apologies for interrupting, I tend to follow the Neorv32 discussions page as there is some nice information in here and some cool projects ongoing with Neorv32 .

What do you mean by: RISC-V is about to ratify a defined trap for "hardware errors".

[stnolting]

@DS-567

This is nothing too special. It is merely a new standardized exception that indicates some kind of runtime hardware error. How this exception is actually triggered or how it should be handled is up to the implementer (yet).

[BEforlin]

Hey @stnolting

Did you also include the physical memory protection (PMP) ISA extension? Maybe this could be useful to catch some (a lot?) invalid pointers before they access incorrect data.

We did not and we plan on testing it. The idea was to have an absolute baseline for the system. That PMP can detect the invalid jumps is a given, the question is what trade-offs can you make with the system.

I was thinking about adding some custom hardware performance monitors. These might be helpful to benchmark runtime events (like register file ECC faults in this case). What do you thing about this? Would this be a handy extension?

That is what we did to get the error reports. We implemented custom counters to all of our detection hardware.

RISC-V is about to ratify a defined trap for "hardware errors". Maybe this is also something we could add - at least the infrastructure for that. Custom implementation could use that as a hook to inform the runtime environment that something has gone (terribly) wrong.

Do you mean the RERI standard? Does NEO have any current effort in that direction? We actually started work on this quite recently and would be interested in contributing to what (if anything) is already available. I think the standard is quite cumbersome for a RISCV microcontroller, so we are looking in what ways we can make it more reasanoable.

This is really a very interesting research project you are working on! I'm already looking forward to further results.

I will post more results as they come!

[stnolting]

That PMP can detect the invalid jumps is a given, the question is what trade-offs can you make with the system.

PMP in its full configuration is quite heavy in terms of area resources and impact on the critical path. However, the latest PMP configuration extensions (#808) can help to tailor the PMP functionality to your actual needs.

Do you mean the RERI standard? Does NEO have any current effort in that direction?

Do you mean this ISA proposal?

I've never heard of it, but it looks interesting at first glance. I'll have a closer look at this - maybe this is something we could add in the future.

I will post more results as they come!

Looking forward to it! Thank you very much! 👍