Precise exception support

[pmccormick]

A proposal to implement precise exception support within the runtime. There are number of factors here that would make this useful:

• Advanced debugging mechanisms.
• Expanded capabilities for fault/error recovery and restartable tasks (beyond current feature set -- perhaps more localized/fine-grained with this path?)
• Controlled injection of tasks into an executing application (e.g., advanced workflows with a REPL-based mechanism for in situ data analytics, avoiding the abort-and-requeue model in HPC workflows).
• Etc... These are the top ones (in no particular order) on my mind...

[lightsighter]

Mainly what we want is the ability to pause the runtime at a point in time and have the state of the regions and the program look like it is in the middle of a sequential execution. This will allow debuggers to build around Legion that can do introspection of Legion programs in a sane way.

[magnatelee]

Legate libraries (or any Python library built on top of Legion) can benefit from precise exceptions and I'd like to put this back to our priority list. Any errors from the C++ tasks in those libraries are currently hard errors. They should instead be tossed back to the Python land without halting the runtime and caught at the point where the faulty task was launched. The latter also requires that nothing beyond that faulty task make any perceivable changes to the program state. Since this precise handling of exceptions is inherently a sequential, blocking process, which is bad for performance, this shouldn't be the only mode of execution and we will want to have another one where exceptions are checked in a deferred manner (possibly without any recourse).

[lightsighter]

Since this precise handling of exceptions is inherently a sequential, blocking process, which is bad for performance, this shouldn't be the only mode of execution and we will want to have another one where exceptions are checked in a deferred manner (possibly without any recourse).

To be clear the default will be to have non-precise exceptions and users will have to opt-in to running the runtime in a way that enables support for precise exceptions.