Improve the semantics of asynchronous exceptions (new simpler version)

[mshinwell]

Supercedes #765. This works in bytecode as well as native now.

New rules of the game:

• Only Sys.Break or Stack_overflow may be raised from signal handlers, finalisers or memprof callbacks. (Stack_overflow is not supposed to be raised by user code.) This helps prevent exfiltration of data from these callbacks by means of asynchronous exceptions; and avoids decisions in the runtime about how to "join" more descriptive exceptions in the case where a second asynchronous exception arises during processing of another.
• Raising any other exception from any of these callbacks will cause the toplevel uncaught exception handler to be invoked.

Exceptions will no longer be raised at allocation points; this is reflected in Mach.operation_can_raise. This now matches the semantics expected by Flambda 2.

Note that this is a breaking change: as noted in sys.mli, you will likely want to use Sys.with_async_exns after using Sys.catch_break, or else Ctrl+C interrupts will cause exceptions to be sent to the toplevel uncaught exception handler.

I still need to add some more test cases, although the one I've added plus the existing testsuite seem to cover the majority of cases.

For some reason it seemed that the existing bytecode interpreter use of sigsetjmp (for implementing exceptions) was incorrect; these calls weren't being told to save the signal mask, but the siglongjmp calls were instructing restoration of such mask. This had to be fixed for this PR to work correctly; the signal masks should be restored across these jumps.

@stedolan is on the hook to review this in due course. We plan to initiate some discussion upstream too.

[mshinwell]

One hitch: there isn't a 32-bit x86 implementation provided for native code, and I don't really want to write one. I'm going to see again if we can stop building the 32-bit compiler.

[Gbury]

I'm afraid that this change would break at least a few packages, for instance:

• those that use gc alarms (e.g. to enforce a limit on memory usage)
• those that use expressions from signal handlers to implement timeouts

[mshinwell]

This will be a breaking change, but I think any fix to this problem is going to suffer from the same. We decided on a simple model where users are expected to use Break (and only Break) to exfiltrate data from signal handlers, finalisers etc. in order to avoid complications where new asynchronous exceptions are raised during processing of an existing asynchronous exception in the runtime. With this model, there is a clear and reliable way of doing this exfiltration, which does not run the risk of losing data. The user should establish a data structure which is populated by a signal handler, finaliser or whatever and then query that structure upon receiving Break from a Sys.with_async_exns point. This is a bit like the existing technique used e.g. in Async for handling signals: the only thing that runs in the signal handler itself is pushing a value onto a queue to say that there has been a signal. Those can then be dealt with in a more normal context.

This PR has now been updated with a 32-bit x86 implementation in i386.S.

[mshinwell]

@stedolan could you please review this, thanks.

[stedolan]

An earlier version of this patch (IIRC) made caml_start_program always catch async exceptions, while the current version splits it into caml_start_program and caml_start_program_async_exn. Why? Do you ever want the former? It seems dangerous to allow async exceptions to propagate past caml_callback_exn by default.

[mshinwell]

An earlier version of this patch (IIRC) made caml_start_program always catch async exceptions, while the current version splits it into caml_start_program and caml_start_program_async_exn. Why? Do you ever want the former? It seems dangerous to allow async exceptions to propagate past caml_callback_exn by default.

This was to make sure that async exns only appear at Sys.with_async_exns points, but hmm... maybe this could be problematic for a caller that really requires their frame calling caml_callback_exn to be returned to.

[mshinwell]

@stedolan and I discussed this. I'm going to change this PR so that caml_callback*_exn do return async exns; but caml_callback* will raise async exns as async exns.

[stedolan]

I am basically happy with this patch, but I'm concerned with the size of the diff to the runtime. (Especially as I'm about to try to merge 4.14 into this branch!).

I've made a patch that simplifies some of the mechanisms here with a view to making a smaller change, which should also aid in upstreaming this. The patch is at stedolan/flambda-backend/tree/async-exns-only-break, let me know what you think of it. Commit message copied below:

Simplify the implementation of new asynchronous exception semantics

  - Use a flag in Caml_state to indicate whether an async exception is
    being raised, removing the need for two versions of caml_start_program

  - Use the same flag mechanism to distinguish async exceptions on both
    bytecode and native code

  - Check that only Break may be raised from async callbacks at the point
    of *raising*, rather than at the point of running the callback

  - No longer propagate Stack_overflow from async callbacks to the main
    program

  - Simplify raising logic by removing prepare_for_raise, as there is no
    longer a need to carefully avoid caml_raise recursion (Now, caml_raise
    may trigger callbacks, which may trigger caml_raise_async, which can
    trigger no further work)

  - Revert a change to setsigmask behaviour in interp.c

  - caml_raise_out_of_memory_fatal renamed to caml_fatal_out_of_memory,
    and is always a fatal error on both bytecode and native

[gretay-js]

For files owned by me