It is rather easy to see message loss in the face of async exceptions

[hyperthunk]

This is one for discussion rather than a bug... Consider the following code:

catchExit (receiveWait [ match (\(m :: SomeType) -> doSomething m) ])
                (\pid (e :: ProcessExitException) -> handleExFrom e pid) 

When receiveWait goes into a blocking read on the mailbox, exceptions are masked, however this is not the case when receiveWait is running a handler for which a match has succeeded. It is thus very easy to handle an exit signal, having mutated our mailbox and subsequently lost a message.

Now, this isn't necessarily wrong - though it would be nice to have ways of dealing with it more elegantly - and I think what I've realised here is why the safe-exceptions package won't let you catch async exceptions. They're meant to terminate your thread.

When we introduced catchExit and friends, the intention was to mimic Erlang's exit trapping behaviour. Specifically, how this works is that if you evaluate process_flag(trap_exit, true), then all future exit signals do not terminate the process, but instead arrive in the mailbox in the form {'EXIT', From, Reason}. I do not think we can (or should) try to emulate that behaviour in Cloud Haskell, and I understand now why @edsko was reluctant when we first asked for it. I do think we need to have a clear method of signalling requested vs. unexpected termination to processes, for proper supervision methods to work. But I feel the base primitives are easily misunderstood and will lead to complex bugs if not used carefully.

@qnikst and I were discussing some other exception handling options that we could add as optional APIs in addition to receiveWait, and these might be useful. At a minimum, I think we need to document this situation though, so that users aren't caught out by surprise.

[mboes]

Could you clarify what the issue is that you see? receiveWait doesn't open a transaction, so it's not just with catchExit that messages have the potential to be lost. Is there any behaviour here that you find surprising or unexpected?

[hyperthunk]

Hey @mboes - no I don't think it's surprising or unexpected, but I've been inside the code for a long time. For an Erlang programmer it would be very surprising, and anyone who didn't know the implementation would be left guessing as to whether the mailbox had been modified or not wouldn't they? After all, mailboxes are not a canonical Haskell idiom.

[mboes]

It may be interesting to note any difference with Erlang here. What does Erlang do? Does Erlang have asynchronous exceptions? Does interrupting a process in the middle of a message handler restore the message in the mailbox if the exception gets handled?

If you really want Erlang-like exceptions-as-messages, an approach similar to enclosed-exceptions should be possible: isolate the process from async exceptions by forking it and not publishing its ProcessId, then forwarding any async exceptions to it as a message. It sounds like an interesting pattern to me, but I'm skeptical that this should go in distributed-process proper.

[hyperthunk]

Erlang does not have asynchronous exceptions, only synchronous ones. For terminating another process, it provides exit and kill signals (where the latter is an exit signal with 'kill' as the reason. These immediately terminate the recipient unless you've set the trap_exit flag, in which case they're delivered as a message.

Since erlang has no concept of a handler per se, I don't see that comparison holding up. If I receive a message in erlang (selectively or otherwise) then no exception will be raised during waiting and no asynchronous exceptions can or will arrive. Note that kill signals are not trappable.

Term = receive %% won't fail 
{A, B} = receive %% won't fail

We have this issue because we're (a) not dynamically typed, and therefore need to match to a handler function based on types, and (b) have asynchronous exceptions to think about.

I'm not saying we /should/ follow erlang's approach here, just that we've tried to and it's incomplete, semantically different and unintuitive, and poorly documented. Since it was I that introduced the features, I think it's reasonable for me to judge then thus too, so this isn't a criticism of anyone else's code btw.

What we need, fundamentally, for supervisor to work, is a consistent way to differentiate between exit and kill signals, and to ensure that if we do handle an exit signal that the scope in which that works is clear and the message handling/loss semantics are clear and documented.

[hyperthunk]

If you really want Erlang-like exceptions-as-messages, an approach similar to enclosed-exceptions should be possible: isolate the process from async exceptions by forking it and not publishing its ProcessId, then forwarding any async exceptions to it as a message. It sounds like an interesting pattern to me, but I'm skeptical that this should go in distributed-process proper.

Neither am I. As I mentioned above, I'm not sure whether /exit handling/ was ever a good idea for cloud haskell, although we introduced that to distributed-process too.

The motivation for this issue is that supervisors need a way to (a) tell children to exit, and (b) determine whether or not a child died normally or not. Currently we have DiedNormal for processes that exit normally, and DiedException for those that didn't. Now we take advantage of Cloud Haskell's asynchronous exceptions to some extent here, since if we enqueue a message saying 'please exit' that might be stuck behinds zillions of other messages if a process has a busy/full mailbox, or is slow to respond. Supervisor handles that by placing a time limit on seeing the monitor signal, and if a child takes too long to respond to exit then it will evaluate kill and terminate the child without regard for it running shutdown/cleanup handlers.

Differentiating between exit signals is quite important here though. A child that receives ExitShutdown knows it is being asked politely to stop, whereas ExitOther reason where reason :: Message is another matter. Providing exit at all means that we're giving people a concrete API to use when they want to ask a process to stop. This is a structured exit handling mechanism, rather than a structured exception handling mechanism. We expect people to handle exit signals, since we provide kill :: ProcessId -> String -> Process () to terminate them without allowing the signal to be handled. Given that we're providing a mechanism for dealing with exit signals, I think it's a bit remiss that you can handle the signal but lose a message in the process. At best it's a leaky abstraction - you need to know that we're using asynchronous exceptions to deal with exit and kill signals, otherwise you might mask_ or catch and be surprised by the consequences when ProcessKillException doesn't do what you expect. And the potential for message loss is even worse, since it requires not only to understand how exit and kill are implemented under the covers, but also how receive{Wait, Timeout}, and in turn CQueue and the Match system works. Only by reading the code can you determine that there is a point between matching a message and evaluating a handler where any async exception can interrupt us, and the consequences of that.

[hyperthunk]

Hmmm.. I wonder if we can extend the mask such that we restore only to execute the handler... Not sure I understand Haskell's asynchronous exceptions well enough here, but is there a way to ensure that if your handler is implemented in terms of catchExit that you won't lose any messages between dequeue and the handler being evaluated?

[hyperthunk]

So this issue much higher up the stack, in the Managed Process (aka d-p-client-server) API. When using prioritised mailboxes, and annotating with the safe filter. The use of mask there is a matter of policy application, which is probably how this is best solved.