amrex::Abort does not unconditionally abort on GPU

[BenWibking]

Describe the bug
We have been relying on amrex::Abort to unconditionally abort the code when called from GPU code. However, this only works when compiled in debug mode (or with additional compiler flags).

We need to rewrite the error handling code for iterative solves so that it correctly handles errors when -DCMAKE_BUILD_TYPE=Release is used (the default).

The Quokka documentation incorrectly describes amrex::Abort based on the old AMReX documentation (which was incorrect): https://quokka-astro.github.io/quokka/error_checking.html

To Reproduce
Steps to reproduce the behavior:

1. Compile the PopIII problem for GPU
2. It should crash with a VODE failure at timestep 58
3. Observe that it does not.

Additional context
The AMReX documentation was recently changed (October 2023) to reflect this new behavior, but this change was not communicated to the user community: AMReX-Codes/amrex#3605.

This affects both the Newton-Raphson solve used by the radiation code as well as the Microphysics chemistry integrator.
cc @psharda @chongchonghe

[chongchonghe]

This is important to know. I always reply on AMREX_ASSERT, AMREX_ALWAYS_ASSERT, or AMREX_ALWAYS_ASSERT_WITH_MESSAGE. I guess they all reply on amrex::Abort, right? Then, what is the solution? How do we "rewrite the error handling code" so that the code handles errors in release build? According to the documentation, we should set USE_ASSERTION=TRUE at compile time to enable AMREX_ALWAYS_ASSERT in GPU release build.

[BenWibking]

This is important to know. I always reply on AMREX_ASSERT, AMREX_ALWAYS_ASSERT, or AMREX_ALWAYS_ASSERT_WITH_MESSAGE. I guess they all reply on amrex::Abort, right?

Yes.

Then, what is the solution? How do we "rewrite the error handling code" so that the code handles errors in release build?

You have to rewrite the function so it returns an error code that you then check. For an example, see the cooling code:

quokka/src/CloudyCooling.hpp

Line 278 in fcde1aa

nsubsteps(i, j, k) = nsteps;

quokka/src/CloudyCooling.hpp

Line 298 in fcde1aa

int nmax = nsubstepsMF.max(0);

You can then check the error condition in host code, and call amrex::Abort there. Or, you could return the failure condition to the calling function, and have it re-do the update with a smaller timestep. This is done in the hydro update.

[BenWibking]

Another example of how to do this is how Castro handles reactions:
https://github.com/AMReX-Astro/Castro/blob/1f3bfff9739227bb61f4ac0123653061df102a03/Source/reactions/Castro_react.cpp#L408

It updates an atomic variable for each failed cell, rather than updating cells in a separate MultiFab, so it is significantly more memory efficient. This implementation should be preferred. With this kind of implementation, you have to remember to update across all MPI ranks after the atomic update. In Castro, this step is done here:
https://github.com/AMReX-Astro/Castro/blob/1f3bfff9739227bb61f4ac0123653061df102a03/Source/reactions/Castro_react.cpp#L427

[psharda]

Fixed for PopIII by #575