Make sure the client crashes on every unhandled exception (new approach)

[jomuel]

The raiden client is supposed to crash on every unhandled exception. However, unhandled exceptions that happen in spawned gevent.Greenlets only crash their greenlet and do not bubble up on the greenlet tree by default. Therefore, every greenlet will be required to check for unhandled exceptions in its children and propagate them.

Our former approach to achieve this behavior was to patch gevent and mess with its exception handling. After several attempts and discussions involving @andrevmatos, @ulope, @hackaugusto and myself we came to the conclusion that there is no way to achieve the behavior we want by patching gevent without unmanageable side effects.

Replaces #1829 and #2123

To do:

• Revert gevent patching (Revisit gevent exception handling #2171)
• Enforce crashing on each unhandled exception within the rest API (Rest API unhandled exceptions should crash the node #2179, fixed in Crash Rest API if unhandled exception happens on request #2196)
• keep a list of long-running greenlets and watch for exceptions in them (Exit longlived greenlets if any of them fails #2208)
• avoid the spawn-and-forget pattern, parent should always keep track of children greenlets, rewrite functions where necessary

[hackaugusto]

Question Revert gevent patching (WIP #2171) will also remove the changes to the hub's variable SYSTEM_ERROR correct?

[andrevmatos]

@hackaugusto Yes, that's the plan

[LefterisJP]

raiden-network/raiden-libs#122 is supposed to be part of the fix for this issue? Not the full fix? When can we consider this issue closed?

@andrevmatos @jomuel

If we are to "avoid a pattern" that's more of a guideline than a milestone issue so I am inclined to close it.