Error propagating exceptions on-non subclassable types

[FedericoV]

I was using joblib to run 8 optimizations using nlopt in parallel, and got this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-138-0fe3676d9e18> in <module>()
     31     for i in range(iterations):
     32         pool = Parallel(n_jobs=max_jobs, verbose=5)
---> 33         out = pool(delayed(nlopt_optimization)(job_id) for job_id in range(max_jobs))
     34         solutions.extend(out)
     35 

/home/federico/anaconda/lib/python2.7/site-packages/joblib/parallel.pyc in __call__(self, iterable)
    658                 # consumption.
    659                 self._iterating = False
--> 660             self.retrieve()
    661             # Make sure that we get a last message telling us we are done
    662             elapsed_time = time.time() - self._start_time

/home/federico/anaconda/lib/python2.7/site-packages/joblib/parallel.pyc in retrieve(self)
    540                             )
    541                         # Convert this to a JoblibException
--> 542                         exception_type = _mk_exception(exception.etype)[0]
    543                         raise exception_type(report)
    544                     raise exception

/home/federico/anaconda/lib/python2.7/site-packages/joblib/my_exceptions.pyc in _mk_exception(exception, name)
     67         this_exception = type(this_name, (exception, JoblibException),
     68                     dict(__repr__=JoblibException.__repr__,
---> 69                         __str__=JoblibException.__str__),
     70                     )
     71         _exception_mapping[this_name] = this_exception

TypeError: type 'nlopt.ForcedStop' is not an acceptable base type

My understanding is because a bunch of nlopt types are actually all generated using SWIG, and cannot be naturally inherited. Should there be a check before joblib tries creating a nice custom exception that the type can indeed be inherited from?

[lesteve]

Interesting, I don't really see a way to work-around this problem cleanly off the top of my head.

I have been working on this code recently and as far as I understand the reason we want to inherit from exception is that we should be able to catch the exception in your main process with the same name that the one that actually happens on a worker.

Another way to look at it is that in the following snippet, you don't want that the exception you are catching depends on whether n_jobs == 1 or n_jobs != 1:

try:
    Parallel(n_jobs=n_jobs)(delayed(f)(i) for i in range(10))
except ValueError:
    # do something

Something that could be done in your case is that _mk_exception type returns a JoblibException and then the client code just has to do something like:

try:
    Parallel(n_jobs=n_jobs)(delayed(f)(i) for i in range(10))
except ValueError, JoblibException:
    # do something

Better suggestions welcome !