Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error propagating exceptions on-non subclassable types #269

FedericoV opened this issue Nov 9, 2015 · 1 comment

Error propagating exceptions on-non subclassable types #269

FedericoV opened this issue Nov 9, 2015 · 1 comment


Copy link

@FedericoV FedericoV commented Nov 9, 2015

I was using joblib to run 8 optimizations using nlopt in parallel, and got this error:

TypeError                                 Traceback (most recent call last)
<ipython-input-138-0fe3676d9e18> in <module>()
     31     for i in range(iterations):
     32         pool = Parallel(n_jobs=max_jobs, verbose=5)
---> 33         out = pool(delayed(nlopt_optimization)(job_id) for job_id in range(max_jobs))
     34         solutions.extend(out)

/home/federico/anaconda/lib/python2.7/site-packages/joblib/parallel.pyc in __call__(self, iterable)
    658                 # consumption.
    659                 self._iterating = False
--> 660             self.retrieve()
    661             # Make sure that we get a last message telling us we are done
    662             elapsed_time = time.time() - self._start_time

/home/federico/anaconda/lib/python2.7/site-packages/joblib/parallel.pyc in retrieve(self)
    540                             )
    541                         # Convert this to a JoblibException
--> 542                         exception_type = _mk_exception(exception.etype)[0]
    543                         raise exception_type(report)
    544                     raise exception

/home/federico/anaconda/lib/python2.7/site-packages/joblib/my_exceptions.pyc in _mk_exception(exception, name)
     67         this_exception = type(this_name, (exception, JoblibException),
     68                     dict(__repr__=JoblibException.__repr__,
---> 69                         __str__=JoblibException.__str__),
     70                     )
     71         _exception_mapping[this_name] = this_exception

TypeError: type 'nlopt.ForcedStop' is not an acceptable base type

My understanding is because a bunch of nlopt types are actually all generated using SWIG, and cannot be naturally inherited. Should there be a check before joblib tries creating a nice custom exception that the type can indeed be inherited from?


This comment has been minimized.

Copy link

@lesteve lesteve commented Nov 9, 2015

Interesting, I don't really see a way to work-around this problem cleanly off the top of my head.

I have been working on this code recently and as far as I understand the reason we want to inherit from exception is that we should be able to catch the exception in your main process with the same name that the one that actually happens on a worker.

Another way to look at it is that in the following snippet, you don't want that the exception you are catching depends on whether n_jobs == 1 or n_jobs != 1:

    Parallel(n_jobs=n_jobs)(delayed(f)(i) for i in range(10))
except ValueError:
    # do something

Something that could be done in your case is that _mk_exception type returns a JoblibException and then the client code just has to do something like:

    Parallel(n_jobs=n_jobs)(delayed(f)(i) for i in range(10))
except ValueError, JoblibException:
    # do something

Better suggestions welcome !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.