ImportError: cannot import name 'SafeFunction'

[cjacoby]

Can anyone else confirm that this is an issue? Maybe it's just an issue with my environment? But I don't think so.

The below is with a completely fresh Python 3.5.2 environment, just installing with python setup.py install. My hunch is that this is a recent JobLib change.

Python 3.5.2 (default, Jun 29 2016, 13:42:59)
Type "copyright", "credits" or "license" for more information.

IPython 5.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import pescador
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-916cecc10ef3> in <module>()
----> 1 import pescador

/Users/christopher/devel/pescador/pescador/__init__.py in <module>()
      3
      4 from .util import *
----> 5 from .zmq_stream import *
      6 from .pescador import *
      7

/Users/christopher/devel/pescador/pescador/zmq_stream.py in <module>()
     20     import json
     21
---> 22 from joblib.parallel import SafeFunction
     23
     24 __all__ = ['zmq_stream']

ImportError: cannot import name 'SafeFunction'

And also

python3 setup.py test
running test
running egg_info
writing requirements to pescador.egg-info/requires.txt
writing pescador.egg-info/PKG-INFO
writing top-level names to pescador.egg-info/top_level.txt
writing dependency_links to pescador.egg-info/dependency_links.txt
reading manifest file 'pescador.egg-info/SOURCES.txt'
writing manifest file 'pescador.egg-info/SOURCES.txt'
running build_ext
pescador (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: pescador (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: pescador
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 462, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/christopher/devel/pescador/pescador/__init__.py", line 5, in <module>
    from .zmq_stream import *
  File "/Users/christopher/devel/pescador/pescador/zmq_stream.py", line 22, in <module>
    from joblib.parallel import SafeFunction
ImportError: cannot import name 'SafeFunction'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

[ejhumphrey]

+1, this is an issue for me in Python 2.7.11, joblib-0.10.0

[ejhumphrey]

waitbutwhyyy

[bmcfee]

cause i'm busy putting out fires on jams 😢

[ejhumphrey]

okay, looks like travis is using joblib-0.9.4, from the logs...

Collecting joblib (from pescador==0.1.2)
  Downloading joblib-0.9.4-py2.py3-none-any.whl (112kB)
    100% |████████████████████████████████| 122kB 6.1MB/s 

what's the preferred approach for supporting libraries where versions are organized differently? I'm digging through the package and can't find SafeFunction...

[bmcfee]

"versions are organized differently" == ?

this just sounds like an API change.

[ejhumphrey]

yes, that is what someone who's been properly caffeinated would say. I've found it in joblib._parallel_backends, which seems like something that shouldn't be used...

https://github.com/joblib/joblib/blob/99c1c0450a6fbb8e4ec4818ddc4ab1d5187b1112/joblib/_parallel_backends.py#L328

should we 'try/except' on the import in zmq_stream? or encapsulate this funniness in a parallel submodule?

[bmcfee]

should we 'try/except' on the import in zmq_stream?

sounds like a bad idea to me.

Presumably there's a right way to do this in the new api.

[ejhumphrey]

on cracking open zmq_stream.py, I see we've already got

try:
    import ujson as json
except ImportError:
    import json

so thoughts on

try:
    # joblib <= 0.9.4
    from joblib.parallel import SafeFunction
except ImportError:
    # joblib >= 0.10.0
    from joblib._parallel_backends import SafeFunction

?

[bmcfee]

Hah.. both are not-so-great ideas, but i'm okay with it for now.

I think it's worth considering reimplementing the safefunction/transport exception stack within pescador anyway, but that can be a separate issue.

[ejhumphrey]

👍

on it