shell_plus notebook settings Import error in subdirectory

[ericmillsio]

I have an issue where when I run python manage.py shell_plus --notebook I get an error where it can't find my Django settings when I navigate and create a notebook in a subdirectory of the project:

WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
[IPKernelApp] WARNING | Error in loading extension: django_extensions.management.notebook_extension
Check your config files in /home/eric/.ipython/profile_default
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/shellapp.py", line 269, in init_extensions
    self.shell.extension_manager.load_extension(ext)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/extensions.py", line 86, in load_extension
    if self._call_load_ipython_extension(mod):
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/extensions.py", line 133, in _call_load_ipython_extension
    mod.load_ipython_extension(self.shell)
  File "/usr/local/lib/python2.7/dist-packages/django_extensions/management/notebook_extension.py", line 5, in load_ipython_extension
    style=no_style())
  File "/usr/local/lib/python2.7/dist-packages/django_extensions/management/shells.py", line 102, in import_objects
    setup()
  File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 17, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 44, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 92, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named MY_DJANGO_APP.settings.common

It works when I create a python notebook from the root directory of my project, but not from any subdirectory.

Any suggestions?

[chid]

If you haven't resolved the issue you can start the notebook shell from the subdirectory where your notebooks are.

[ericmillsio]

This issue is still unresolved, yes. But thanks for the tip in the interim.

[klimanek]

@crearc Did you really manage to open/edit notebooks while using Django Shell Plus kernel? And Jupyter sees Django project and all the components while working in a subfolder? How can I do that? I asked here but I was not able to resolve it: http://stackoverflow.com/questions/34929528/django-and-ipython-subdirectories

[ericmillsio]

Yes I was able to get them open/editable, but as I said the pynbs only worked from the root directory, in other directories the kernel did not work. But yeah I was able to import all my libraries just fine.

I did add some config stuff, not sure if it was necessary though:

settings.py

IPYTHON_ARGUMENTS = [
    '--ext', 'django_extensions.management.notebook_extension',
]
NOTEBOOK_ARGUMENTS = [
    '--config=jupyter_notebook_config.py',
]

jupyter_notebook_config.py

c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False # This was so it didn't try to open a browser in production
c.NotebookApp.port = 8888

Other than that there wasn't much else that I added.

[klimanek]

Hold on. :o) Let me ask you clearly: are you having the same issue like me? Or can you import models etc from Django when working in subdirectory? As I explained, I tried to copy all notebooks to subdirectory called Calculations, all notebooks contain imports from Django models etc., but this does not work (cannot import Django stuff). So for now, I have to keep them in Django root directory... so many files, it gets pretty messy there.

[ericmillsio]

Ahh yeah, I thought you couldn't get notebooks working at all. Yes I am
having the exact same issue and have not resolved it.

On Thu, Jul 7, 2016, 1:53 PM klimanek notifications@github.com wrote:

Hold on. :o) Let me ask you clearly: are you having the same issue like
me? Or can you import models etc from Django when working in subdirectory?
As I explained, I tried to copy all notebooks to subdirectory called
Calculations, all notebooks contain imports from Django models etc., but
this does not work (cannot import Django stuff). So for now, I have to keep
them in Django root directory... so many files, it gets pretty messy there.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#865 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAYHiBlLdwl4GS5q8cUDsmy879UEoQQxks5qTUtGgaJpZM4H_j_a
.

[naimetti]

Same problem here.

[jonathanstrong]

+1 would love a solution to this problem - i have like 150 notebooks in the root dir of my project, it's getting out of hand

[ross-urban]

I'm also having this problem. 😦

[seanjh]

I managed to work around this problem, in case it's useful for anyone else. Basically, I added an ipython_config.py file to my notebook sub-directory, then manually inserted my base project directory into sys.path.

Project layout:

/project
    /someapp
    /notebooks
        ipython_config.py
    settings.py

settings.py

NOTEBOOK_ARGUMENTS = [
    '--notebook-dir', 'notebooks',
]

IPYTHON_ARGUMENTS = [
    '--debug',
]

notebooks/ipython_config.py

import sys
import os

FILE_PATH = os.path.abspath(os.path.dirname(__file__))
PROJECT_BASE_PATH = os.path.abspath(os.path.join(FILE_PATH, '..'))

# Allows the kernel to "see" the project during initialization. This
# FILE_PATH corresponds to Jupyter's "notebook-dir", but we want notebooks to
# behave as though they resided in the base directory to allow for clean
# imports.
print("sys.path BEFORE = {}".format(sys.path))
sys.path.insert(1, PROJECT_BASE_PATH)
print("sys.path AFTER = {}".format(sys.path))

# Any additional initialization logic goes here

In the logs, you should observe IPython locating that ipython_config.py file in your subdirectory. For example:

[D 13:28:12.612 NotebookApp] Starting kernel: [u'/usr/bin/python', u'-m', u'ipykernel', u'-f', u'MY_USER_HOME/.local/share/jupyter/runtime/kernel-....json', u'--debug', u'--ext', u'django_extensions.management.notebook_extension']
[D 13:28:12.616 NotebookApp] Connecting to: tcp://127.0.0.1:45969
[I 13:28:12.616 NotebookApp] Kernel restarted: f2c2cfa1-680e-4b17-bd95-03e1affb9b77
[D 13:28:12.617 NotebookApp] Connecting to: tcp://127.0.0.1:52312
[IPKernelApp] IPYTHONDIR set to: MY_USER_HOME/.ipython
[IPKernelApp] Using existing profile dir: u'MY_USER_HOME/.ipython/profile_default'
[IPKernelApp] Searching path [u'MY_PROJECT_BASE_PATH/notebooks', u'MY_USER_HOME/.ipython/profile_default', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[IPKernelApp] Attempting to load config file: ipython_config.py
[IPKernelApp] Looking for ipython_config in /etc/ipython
[IPKernelApp] Looking for ipython_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_config in MY_USER_HOME/.ipython/profile_default
[IPKernelApp] Looking for ipython_config in MY_PROJECT_BASE_PATH/notebooks
sys.path BEFORE = ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/IPython/extensions', 'MY_USER_HOME/.ipython']
sys.path AFTER = ['', MY_PROJECT_BASE_PATH, '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/IPython/extensions', 'MY_USER_HOME/.ipython']
[IPKernelApp] Loaded config file: MY_PROJECT_BASE_PATH/notebooks/ipython_config.py
[IPKernelApp] Attempting to load config file: ipython_kernel_config.py
[IPKernelApp] Looking for ipython_kernel_config in /etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in MY_USER_HOME/.ipython/profile_default
[IPKernelApp] Looking for ipython_kernel_config in MY_PROJECT_BASE_PATH/notebooks
...
[IPKernelApp] Starting the kernel at pid: 209
...
[IPKernelApp] Loading IPython extensions...
[IPKernelApp] Loading IPython extension: storemagic
[IPKernelApp] Loading IPython extension: django_extensions.management.notebook_extension
[IPKernelApp] 
*** MESSAGE TYPE:kernel_info_request***
[IPKernelApp]    Content: {}
   --->
...

[trbs]

If anybody wants to create a PR with documentation update, I'm happy to merge it.
Otherwise closing this for now :-)