serialization of package-specific exceptions as non-package-specific exceptions

[benclifford]

I just did this hack for @svandenhaute as a dirty workaround for a problem I've seen a few times:

when you have workers running in one Python environment and the submit side in another Python environment, objects defined in only one of those environments cannot be passed between the two environments with Parsl's usual serialization mechanisms. This setup is discouraged, but users keep wanting it - @svandenhaute in this case, but I've repeatedly experienced funcX pushing the envelope in this direction too.

In the invocation direction, it's often OK to say "well, don't send those objects then" - for example, in a situation where a function only takes an integer or other simple Python object.

In the reverse direction, this commonly breaks when a package-specific exception is raised - although the return value might be nicely structured as an int, for example, exceptions are often a rich package-specific description of what went wrong, and a user is less able to stop those from being returned from an app execution.

The hack I made for @svandenhaute below addresses specifically that "exceptions-are-too-rich-to-send" problem: it replaces all exceptions with RuntimeExceptions (which can "always" be pickled/unpickled) containing a rendered string of hopefully interesting/useful exception information.

Considering how this might be put more seriously into Parsl, I think a user opt-in could be made in the serialization layer, so that any exception (except exceptions in builtin, for example?) are replaced by similarly constructed RuntimeExceptions - perhaps conveying the original stack trace rather than a replacement like this hack.

I think this behaviour (although not the below implementation of this behaviour) can be plugged directly into pickle by a user, because pickle allows per-type serializers to be defined, and a per-type serializer for Exception would handle this.

diff --git a/parsl/app/errors.py b/parsl/app/errors.py
index 24b941d285..162315553e 100644
--- a/parsl/app/errors.py
+++ b/parsl/app/errors.py
@@ -1,5 +1,6 @@
 """Exceptions raised by Apps."""
 import logging
+import traceback
 from functools import wraps
 from types import TracebackType
 from typing import Callable, List, Optional, TypeVar, Union
@@ -139,6 +140,9 @@ def wrap_error(func: Callable[P, R]) -> Callable[P, Union[R, RemoteExceptionWrap
         from parsl.app.errors import RemoteExceptionWrapper
         try:
             return func(*args, **kwargs)
-        except Exception:
-            return RemoteExceptionWrapper(*sys.exc_info())
+        except Exception as e:
+            try:
+                raise RuntimeError("GOT THIS EXCEPTION: "+"\n".join(traceback.format_exception(e)) + "\nEND OF GOT THIS EXCEPTION")
+            except: 
+                return RemoteExceptionWrapper(*sys.exc_info())
     return wrapper

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. E.g. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context about the feature request here.