Cannot load notebooks: ModuleNotFoundError: Error processing dotted path

[lambdaofgod]

I have a notebook folder in ploomber project and ploomber tries to run something so I can't even load the notebook (which BTW is not referenced in pipeline.yaml in any way)

What happened? How is it possible that ploomber breaks jupyter?

Error message

[I 11:59:22.671 NotebookApp] [Ploomber] Requested model: work/wwf/notebooks/Enrich_Test.ipynb. Looking for DAG with root dir: /home/kuba/Projects                            
[E 11:59:22.786 NotebookApp] Uncaught exception GET /api/contents/work/wwf/notebooks/Enrich_Test.ipynb?type=notebook&_=1656064762442 (127.0.0.1)                             
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/contents/work/wwf/notebooks/Enrich_Test.ipynb?type=notebook&_=1656064762442', version='
HTTP/1.1', remote_ip='127.0.0.1')                                                                                                                                            
    Traceback (most recent call last):                                                                                                                                       
      File "/home/kuba/.local/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute                                                                            
        result = await result                                                                                                                                                
      File "/home/kuba/.local/lib/python3.8/site-packages/tornado/gen.py", line 234, in wrapper                                                                              
        yielded = ctx_run(next, result)                                                                                                                                      
      File "/home/kuba/.local/lib/python3.8/site-packages/notebook/services/contents/handlers.py", line 118, in get                                                          
        model = yield maybe_future(self.contents_manager.get(                  
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/jupyter/manager.py", line 267, in get                                                                     
        self.load_dag(                                                                                                                                                       
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/jupyter/manager.py", line 193, in load_dag                                                                
        pairs = [(resolve_path(                                                       
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/jupyter/manager.py", line 196, in <listcomp>                                                                      if t.source.loc is not None]                                                  
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/sources/pythoncallablesource.py", line 130, in loc                                                        
        self._loc = self._callable_loader.get_loc()                                                                                                                          
      File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/sources/pythoncallablesource.py", line 58, in get_loc                                                     
        loc, _ = lazily_locate_dotted_path(self._primitive)                                                                                                                        File "/home/kuba/.local/lib/python3.8/site-packages/ploomber/util/dotted_path.py", line 343, in lazily_locate_dotted_path                                              
        raise ModuleNotFoundError('Error processing dotted '
    ModuleNotFoundError: Error processing dotted path 'deepsense_wwf.data_utils.copy_zipped_data': 'deepsense_wwf' appears to be a namespace package, which are not supported
[W 11:59:22.790 NotebookApp] Unhandled error          
[E 11:59:22.791 NotebookApp] {                                                                                                                                               
      "Host": "localhost:8888",                                                                                                                                              
      "Accept": "application/json, text/javascript, */*; q=0.01",                                                                                                            
      "Referer": "http://localhost:8888/notebooks/work/wwf/notebooks/Enrich_Test.ipynb",                                                                                     
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:101.0) Gecko/20100101 Firefox/101.0"                                                                         
    }                                                          

[edublancas]

As a temporary measure, you may disable the extension with:

jupyter serverextension disable ploomber

Here's what's happening:

When installing the extension, ploomber runs a function whenever you open any notebook; then, it checks if such notebook belongs to the pipeline.yaml. If it does, it injects the cell, if it doesn't, it doesn't do anything. Is this breaking Jupyter for you? If so, please tell us a bit more to understand the problem since that should not happen. Ploomber should simply ignore the notebook and let you use it as usual.

Since ploomber needs to determine if the current notebooks belong or not to a pipeline, it needs to load your pipeline. It seems like during this loading process; something is breaking and it fails to load deepsense_wwf.data_utils.copy_zipped_data. Can you try running:

ploomber status

First in the same folder that contains your pipeline.yaml and then in the folder that contains the notebook you're opening (Enrich_Test.ipynb); let me know if you get the same error.

[lambdaofgod]

Thanks, it worked for jupyter notebook but not for jupyterlab

I disabled it for jupyterlab, and when I run jupyter labextension list I don't see ploomber, but it still loads when I run jupyter lab.

The funny part is I don't even remember installing ploomber extension...

[edublancas]

Ok, thanks for the feedback. The ploomber extension activates when you run pip install ploomber. So if you uninstall ploomber, it should remove it as well.

It's weird that disabling it didn't turn it off on jupyter, but many users have experienced issues with this in the past (not being able to activate it, the extension doesn't show up, etc). So who knows what's happening.

this is a pretty bad bug so we'll get to it. I'll work on it this week.

[lambdaofgod]

Wouldn't disabling automatic installation of the extension basically solve the problem? I would prefer to have the option to install the extension myself, and temporarily the biggest problem will go away

I didn't even know that I had this extension until it failed 😄

[edublancas]

good point. the problem is that if the extension doesn't enable automatically, then most people will never find it. so I think it's best to have it turned on but we should ensure it doesn't break Jupyter.

[edublancas]

I'm trying to debug this but I'm unable to reproduce it. Does deepsense_wwf.data_utils.copy_zipped_data in your pipeline.yaml? Or in any of them if you have more than one. It looks like the extension it's trying to load it since it's part of a pipeline but is unable to find it.

[lambdaofgod]

Yes. Basically the pipeline is broken.
But I find it is an overkill that I will not be able to open notebooks from the project folder, even though I didn't specify ploomber to do anything with notebooks.

[edublancas]

Yeah, I agree. We'll fix this as soon as possible. It's pretty bad that ploomber is breaking jupyter just because the pipeline isn't loading. I'll try to reproduce and follow up with more questions if needed

Notes for when we fix this, the problem is when accessing .source.loc, it'll break if unable to find the source for the function in the pipeline.yaml

ploomber/src/ploomber/jupyter/manager.py

Line 199 in 6986b5f

pairs = [(resolve_path(

, we need a try catch there

[edublancas]

@lambdaofgod can you install from git and let me know if the error persists?

pip uninstall ploomber -y
pip install git+https://github.com/ploomber/ploomber@master

[94rain]

Yeah, I agree. We'll fix this as soon as possible. It's pretty bad that ploomber is breaking jupyter just because the pipeline isn't loading. I'll try to reproduce and follow up with more questions if needed

Notes for when we fix this, the problem is when accessing .source.loc, it'll break if unable to find the source for the function in the pipeline.yaml

ploomber/src/ploomber/jupyter/manager.py

Line 199 in 6986b5f

pairs = [(resolve_path(

, we need a try catch there

How should we handle the exception in the except clause? Assigning an empty list to pairs = []?

[edublancas]

@94rain before you start working on this, ensure you get the latest version from master since I recently pushed changes to this file.

I couldn't reproduce @lambdaofgod error, but the overall problem is that we should catch any exceptions raised by load_dag.

ploomber/src/ploomber/jupyter/manager.py

Line 268 in 48b5ac0

def load_dag(self, starting_dir=None, log=True, model=None):

so on a second thought, the solution should be more general (as opposed to only covering the pairs = ... statement.

something like this should work:

def _load_dag(self, ...):
   # actual implpementation

def load_dag(self, ...):
   try:
     self._load_dag(...)
   except Exception as e:
      # log exception along with the message 
     # a problem happened when loading your pipeline

I'm unsure if this would case any side-effects so let's get this change done and then see if any tests break

[edublancas]

what we want if the load_dag fails to execute is for Jupyter to work appropriately

[94rain]

I just opened a PR #923 (with all tests passing) that hopefully will fix this.

[lambdaofgod]

I can confirm this is fixed. Thanks!

[idomic]

@lambdaofgod feel free posting our delivery speed in socials 2 weeks from bug to production 😁🙌

[edublancas]

click here to tweet something nice