navis.nblast fails in notebook

[dokato]

navis.nblast fails in the notebook but works fine from console. I set all the settings as advised in : #17 (comment)

Following the steps from the tutorial:

 import navis
nl = navis.example_neurons()
nl_um = nl / (1000 / 8)
dps = navis.make_dotprops(nl_um, k=5, resample=False)
nbl = navis.nblast(dps[:3], dps[3:])

I get:

BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-5-0557c052a238> in <module>
----> 1 nbl = navis.nblast(dps[:3], dps[3:])

~/anaconda3/lib/python3.8/site-packages/navis-0.5.2-py3.8.egg/navis/nbl/nblast_funcs.py in nblast(query, target, scores, normalized, use_alpha, n_cores, progress, k, resample)
    687                                scores=scores) for this in nblasters]
    688 
--> 689         results = [f.result() for f in futures]
    690 
    691     scores = pd.DataFrame(np.zeros((len(query_dps), len(target_dps))),

~/anaconda3/lib/python3.8/site-packages/navis-0.5.2-py3.8.egg/navis/nbl/nblast_funcs.py in <listcomp>(.0)
    687                                scores=scores) for this in nblasters]
    688 
--> 689         results = [f.result() for f in futures]
    690 
    691     scores = pd.DataFrame(np.zeros((len(query_dps), len(target_dps))),

~/anaconda3/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

~/anaconda3/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

For debugging:

I use Mac, BigSur 11.2.3, M1, Python 3.8 from Anaconda.

 navis.__version__
Out[7]: '0.5.2'

[schlegelp]

Hmmm... I'm afraid that will be difficult to debug on my end.

To start with, I have no clue why there would be a difference in Jupyter vs Console. Are you using the default iPython kernel for Jupiter? The bit that seems to be breaking is the ProcessPoolExecutor in concurrent.futures.

First could you check what happens if you use nbl = navis.nblast(dps[:3], dps[3:], n_cores=1) as that will not use ProcessPoolExecutor.

Depending on how that pans out, I would consider running some simple examples for ProcessPoolExecutor (from the docs or some tutorial) and see if you get it to break too (i.e. see if it is something navis-specific).

[schlegelp]

Just a thought but it could also be the fancy ipywidget progress bars. What happens if you try this:

navis.set_pbars(jupyter=False)
nbl = navis.nblast(dps[:3], dps[3:], progress=False)

[dokato]

Indeed, ProcessPoolExecutor was the problem. n_cores=1 helped. Thanks.

[schlegelp]

Let me know if you figure out what the problem with ProcessPoolExecutor is - maybe we can fix/catch it but minimally to leave a trail for others.

[schlegelp]

Just a ping @dokato - any more insights re this?