Introduce Notebook Magics

[tkf]

This change introduces two notebook magic commands:

1. %pnd -- Return the notebook directory.
   This is like %pwd but returns where the notebook is.
2. %cdnb -- Change directory to where the notebook locates.

These commands are useful when you want to access some supplemental python scripts in the notebook. Having hard-coded %cd IPYTHON/NOTEBOOK/DIRECTORY is not good solution.

These magic commands work only when the notebook is in the directory which is configured by the profile. If notebook server using different directory than the configured variable (for example, notebook_dir is set via command line option), these magic commands misbehaves.

To fix this issue, I think it's better to send some information about notebook to the kernel from the notebook server when the kernel is started.

[takluyver]

This might take a bit more thought, because I understand that before long, the notebook server will be able to navigate to different directories, so the idea of a fixed 'notebook directory' isn't something to rely on. I agree that there should be some means of accessing the location of the notebook, though - perhaps we should define the __file__ variable in the interactive namespace.

[tkf]

Yes, probably this magic should not go into the master until some server-kernel communication is available. I don't think having file which points to non-python file is a good idea. I guess it should be in the shell instance. Maybe providing short-hand notation such as __notebook__ is good idea.

[Carreau]

Also, maybe not register the Notebooks magics where it is registered now, as it would appear in the qtconsole menu...

[takluyver]

Yes, __file__ might not be the best name. We shouldn't use other __foo__ names, but we could call it something like nb_file (we already define get_ipython(), so there's some precedent).

[tkf]

@Carreau Could you tell me where it should go, as I am not yet familiar with the code base? I just grep through the source code and find that place. Although maybe having the magic in Qt is OK because you can share the kernel with notebook.

So, what is the opinion about server-kernel interaction? I guess current notebook server just pass the message around to/from kernel and do not care about what is inside. Adding server-kernel interaction changes this design so I'd like to know if this change is acceptable.

[Carreau]

hum, notebook i'm not sure.
you can inspire from IPython/fontend/qt/console/ipython_widget.py and _load_guiref_magic that is launched at kernel startup to register the new magic.

You should be able to find the corresponding code somewhere in IPython/frontend/html/notebook , but it is not my area of expertise.

Right now you could just build this as an extension, that can be loaded with %load_ext. It would then be available event if it does not make it's way into 0.13

[takluyver]

To echo @Carreau, I suspect doing this properly will be postponed until after 0.13. As Fernando puts it, a half-working feature is often worse than none at all.

If you want to put them in an extension, I'd suggest changing %pnd to something a bit more descriptive - perhaps %notebook_dir. But if you can wait to do this the right way, there's no obligation to make an extension.

[tkf]

@Carreau Thanks, I got an idea. It looks like enabling this magic must happen at the time sending the notebook information from the server, at start time. So, again, server to kernel communication is needed.

@takluyver I can wait until the right way to do it is available. Currently I am fine with hard-coded directory in my notebooks.

I think sending information and enabling magic from server to kernel is rather easy (although it may change the design principle; but I am not sure about what is the principle anyway).

The problem is where and how to put the notebook information. My idea is to put (possibly None) notebook_info attribute in the instance of ZMQInteractiveShell and this attribute holds an instance of hypothetical NotebookInfo class. Are there any class something like this? NotebookInfo should have, at least, nodebook_dir and notebook_path attribute and get_notebook method to get the notebook contents from the file system, so that other utility methods can use metadata information in the notebook file.

[takluyver]

I don't know much of the inner workings of the notebook system. But one complication with that model is that it requires duplicate state - the file of the notebook can be changed while the kernel is running, so the kernel and the server would have to be kept in sync.

Another approach - no doubt with its own problems - is for the kernel to have some way to request data from the notebook server.

[tkf]

Yes, you need to update information when the notebook name is changed or the notebook is deleted. In the future, maybe there will be more events that requires update, for example when the notebook is move to another directory (if the notebook server supports multiple directory).

I think the question is that whether we want to access the notebook contents from the kernel in the different machine than the notebook server. Currently you have the kernel and the server in the same machine, but if the system is changed and having them in different machines is allowed, file system based notebook contents retrieve doesn't work. If you want the kernel in the different machine to access notebook metadata/contents, I guess you need another ZMQ sockets. But as the primary aim is to access notebook file/directory from kernel, I don't think we need to consider that situation.

[takluyver]

That reminds me of another point: the plan is to support other storage backends, i.e. databases. So there's no guarantee that the notebook will even have a file or a directory.

Maybe this isn't worth doing. There's already %pushd and %popd magics to navigate around the filesystem and have a way back.

[tkf]

I think returning notebook location only when it exists on the file system is still useful.

This is the use case I have in mind. Let's say you have this files in the notebook_dir.

MyNotebook.ipynb
script.sh
sommequickmodule.py

and this is what you have in the notebook:

In [1]: %cdnb

In [2]: !sh script.sh

In [3]: import sommequickmodule

What is important is that the result is always the same if you have these file in the same place, regardless the place of notebook_dir.

This cannot be done with %pushd and %popd, right?

[takluyver]

I think we should make the kernel start up in the same directory that the notebook lives in, then your example would work without cell 1. In fact I'm surprised it doesn't already behave like that.

[takluyver]

Wait, my mistake, it does behave like that.

[tkf]

How about this?

MyNotebook.ipynb
MyNotebook/script.sh
MyNotebook/sommequickmodule.py

In [1]: %cd %notebook_path

In [2]: !sh script.sh

In [3]: import sommequickmodule

[tkf]

Ah, no. What I mean was something like this:

In [1]: %cdnb

In [2]: %cd %notebook_name

In [3]: !sh script.sh

In [4]: import sommequickmodule

[minrk]

@takluyver is right that the notebook file itself is not necessarily guaranteed to be there in the long run, but the starting dir of the Kernel is set, and presumably that's where you want to get back to.

So this can be covered by any one of the following commands in your startup:

%pushd # only works if you don't actually use pushd/popd for other things

%bookmark nbd # unsafe if you have multiple values for nbd, because bookmarks are shared across a profile

nbd = %pwd # should always work, and have no problems.

The last one seems to work best, as you can always do cd $nbd to get back, and it doesn't have any conflict with other uses of the tools.

But just because it can be done easily doesn't mean it shouldn't be exposed in a manner similar to this.

[tkf]

I think cwd of the newly created of the kernel is same as the cwd of the server. So it won't be same as the notebook_dir unless you start the kernel at the notebook_dir, no? At least, newly created kernels in my machine is always $HOME, because it is where I start the notebook server.

[takluyver]

Well, the notebook server only currently offers you notebooks in the directory where it's started. When it gains the ability to navigate the file system, I would consider it a bug if kernels didn't start where the notebook was saved.

[minrk]

@tkf is right - kernels start where the notebook server is running, and the notebook-dir is configurable. I always thought we set cwd to the notebook-dir, and I think we certainly should be doing that. We probably just never noticed because people ~always do: cd /path/to/notebooks; ipython notebook rather than ipython notebook --notebook-dir=/path/to/notebooks.

This particular approach (reading config files) will not work in the long run, because ultimately it will not be guaranteed that kernels are on the same filesystem as the notebook server, nor that the notebook server's config is even readable by the kernel's user/process. It also doesn't help if the notebook dir was specified at the command-line, as in my sample above.

[tkf]

I didn't know that cwd of the notebook server will be notebook_dir if it is not set...

[tkf]

#1951 fixes this issue. Thanks.