Recovery kernel loop

[mlange05]

This new feature enables user-level customisation of the error handling procedures in the main kernel loop through JIT and SciPy mode. The core idea is that kernels can now throw different types of errors, eg. ErrorCode.OutOfBounds and that the user can specify custom kernels to override the default behaviour to create model-specific recovery behaviour. The recovery kernels themselves are always run in Python and can be injected via the recovery={Error.MyErrorCode: MyRecoveryKernel} argument.

Caveats:

• Minor: OutOfBounds errors do not propagate the exact sampling location that created the error as SciPy mode does, but the location of the particle at the time. Adding that would require dynamically allocating memory for meta-information, which is left for another PR.
• Major: Kernel errors do not trigger rollback! That entails that a kernel that updates particle information before throwing an error results in inconsistent particle data, since time is not incremented the kernel will be applied again once error recovery succeeded. This can be prevented by performing all field sampling before updating particle data, as most built-in kernels do.

[erikvansebille]

Code looks great @mlange05, glad you got it all to work after what looks like Herculean coding!

I'm really glad we now have this error framework sorted, and I like the syntax of recovery={ErrorCode.Error: CustomKernel}, it is very intuitive and still quite powerful.

A few general comments/caveats that we may (or may not) want to think about right now, or defer for later versions

1. It is very easy for the code to get into an infinite loop, now. For example, if the recovery function for ErrorOutOfBounds results in the particle still being out of bounds, then the recovery will never exit.
2. Is there a way to tell a particle to simply be deleted when reaching the boundary? I tried in test_execution_recover_out_of_bounds, but def MoveLeft(particle): particle.delete() made the code hang (see point above). Having a way to delete a particle when it reaches a boundary is important for non-global domains.
3. For periodic boundary conditions (global domains), it will be important to have separate ErrorOutOfBounds for the different sides of the domain. Thinking about periodic boundary conditions, we want particles which exit on the right (east) to enter on the left (west). But particles that exit on the left have to enter on the right. In fact, ideally we want to simply use the modulo (%) operator to sample the fields, if the domain is with periodic boundary conditions?
4. We will also need to provide an 'OnLand' error, possibly via a grid.landmask field that the user can set. I'm happy to have a go at that, probably in a new branch once this Pull Request lands.

[mlange05]

Ok, thanks for the review. Regarding the comments:

1. That is a general problem and I'm not sure there is an easy answer for that. We can probably add some kind of fail-safe mechanism at some point, but I'd like to wait and see how often people get stuck on this before we spend time building this.
2. That was a real bug, and I believe is now fixed in the latest commit.
3. Yes, and I think will be covered in a follow-on PR. My intuition is that is requires the additional meta-data mentioned under caveats in the PR description, since we can determine the direction from the coordinate of the OutOfBoundsError, if that is provided.
4. Agreed. But like the above a follow-on feature that should have it's own PR.