IPEP 16: Notebook multi directory dashboard and URL mapping

[Zsailer]

See the details at IPEP 16

[ellisonbg]

Let's not put the entire IPEP in the issue though. Can you remove it and just post a link to the IPEP wiki page?

[ellisonbg]

Also can you post a link on the IPEP page that points to this issue?

[ellisonbg]

Some initial work on this has been done in PRs #3168 and #2982. Those PRs have been closed until we finish the design, but can be used as reference for the design.

[minrk]

My main problem with the design proposed here is that it does not actually solve the issue of multiple notebook servers. Since @ellisonbg clarified that the notebook server will remain confined to some 'server' directory, the only way a user can reasonably run their whole universe from a single notebook server (the goal) is to always run it in $HOME.

I still think we want to preserve the notion of 'projects', where a project is a directory that can contain notebooks, data, etc. (hierarchically). What I think this proposal encapsulates is still a single project. What I would like to add to this is the notion of project, and the notebook server's awareness of multiple projects, analogous to the GitHub github.com/organization landing page. The remaining question there is how does one tell the notebook server about project directories? config files? a special /create-project handler that takes a path, and adds it to the project list?

[paddymul]

Can we set it up so that kernels are always started in the same directory as the ipynb file? Why does each notebook instance need to share a global notebook directory?

[minrk]

@paddymul there are two logical choices for the kernel cwd - one is the ipynb location, and the other is the project directory (so you could have a small amount of notebook organization, without having to keep specifying cd ..). If, as proposed here so far, we are abandoning the notion of the project, the cwd will have to be the ipynb location.

[Zsailer]

I added another section to the IPEP under "Things to consider." I mention the notion of "projects" and even chart out a possible URL scheme for the future. After Min and I talked, I like the idea of making projects a second stage to this proposal. We'll work on implementing a hierarchical dashboard in a single project, then move to a multi-project design on a single server for the notebook.

[takluyver]

I would leave the 'project' notion for now, and just have it able to open files that are in any location, like a text editor.

What we do with the dashboard, I'm not sure. Arguably the logical continuation of our current dashboard is a web-based file manager, but we've said we don't want to go down that route.

[Zsailer]

I agree. Projects are something to consider in the future. We should just keep this notion in our minds, however, and create a multi-directory set of web-services that behave in the backend like a 'single' project. That way, it will be easier to extend this to a multiple project notebook/dashboard when we want to introduce this notion.

[Zsailer]

Also, @ellisonbg and I have changed the web-services and URL mapping for this proposal quite a bit. I'll update this IPEP soon.

[minrk]

Projects are something we already have - a directory of notebooks (and data), the notebook server currently supports just one project at a time. The only piece missing from a full single-project server is exposure of sub-directories. The logical next step after that is a mechanism / UI to add new projects from the filesystem, and UI for listing projects. That would cover opening any notebook anywhere on the filesystem, and we never need to introduce the notion of notebooks independent of projects.

[minrk]

We have a fairly straightforward plan of expansion based on projects:

1. complete the single-project tree view (notebooks in sub-dirs, list other files as well - current dashboard is analog of GitHub's USER/REPO/tree/BRANCH/)
2. add multiple projects, top-level view becomes project list, rather than per-project notebook list, a la GitHub's /user page. Previous dashboard for each project moves down a level to /project-name URLs
3. add multiple users, adding login. dashboard remains the same, but listings move to /user/project-name URLs.

By the time we get there, the view currently called 'dashboard' will be accessible as localhost:8888/USER/PROJECT/tree/[SUB-PATH] - a listing of notebooks (and files and subdirs) at the top-level of USER's PROJECT.

In this way, nothing in the previously finished stages really change, we just add a multiplexing layer and a level to the URL. Adding support for notebooks independent of projects would complicate things unnecessarily, and should just wait for proper project support.

[Zsailer]

Projects should also include files that aren't notebooks, though. Are we not looking to include those in the dashboard as well?

[minrk]

Yes - one and only one tree view for everything, very much like GitHub's tree view (comment updated to reflect that).

[takluyver]

A few of us here have been discussing integrating the notebook in the
desktop, i.e. so you can double click an ipynb file and have it open. For
that, how would we derive how many levels up the directory tree counts as
the 'project' directory? Just take the immediate containing folder of the
notebook?

On 28 June 2013 23:33, Min RK notifications@github.com wrote:

Yes - one and only one tree view for everything, very much like GitHub's
tree view (comment updated to reflect that).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3166#issuecomment-20218179
.

[Carreau]

Bumped to 2.0

[andreabedini]

Hi there, my 3c

• I don't know if/when/where this has been discussed before but I think the right direction to go is to have session management separated and independent from the notebook storage. If I want to save my notebook offsite (say Dropbox or S3) having notebook_name and notebook_path stored with the session makes little sense and should be replaced by an uri. That do you guys think?
• To push this further, I'm not sure the concept of sessions and contents are relevant. Sessions: Can't the frontend remember the kernel_id and ask to connect to that directly? Contents: I don't see the use case. Local data could be passed directly to the kernel from the frontend (What I have in mind is drag&drop the csv file from the desktop to the notebook, and have it loaded into the kernel or into a secure temporary file-system) and other data can be fetched remotely. Same for projects.
• I don't understand why the backend should care about anything else other than managing the kernels. The frontend can deal with everything else.

I don't want to come across wrong, I have been using ipython for my research (monte-carlo simulations) for the past three years and I simply loved it. I only recently discovered notebooks and I'm amazed by its potentiality, which, I think, can only be achieved by keeping it lean and flexible.

Thanks for listening!

[minrk]

I don't know if/when/where this has been discussed before but I think the right direction to go is to have session management separated and independent from the notebook storage.

It already is. The NotebookManager doesn't know anything about kernels, and the KernelManager doesn't know anything about notebooks. The SessionManager maintains a mapping of notebooks to kernels.

If I want to save my notebook offsite (say Dropbox or S3) having notebook_name and notebook_path stored with the session makes little sense and should be replaced by an uri. That do you guys think?

Properly abstracting the identifying information for a notebook is an important piece of this. We already have non-filesystem stores of notebooks (Azure, Openstack Swift, Rackspace cloud), so we need to make sure the APIs make sense in these contexts.

Sessions: Can't the frontend remember the kernel_id and ask to connect to that directly?

No - multiple frontends must be able to share kernels associated with a given notebook. This mapping should not live in the frontends themselves.

Contents: I don't see the use case.

The use case is getting contents of files, e.g. a full-page single CodeMirror editor for editing non-notebook files (c.f. GitHub edit view). Your drag&drop example would be a POST request to the contents service (if I recall correctly), so it is another supporting example, not a counter one.

I don't understand why the backend should care about anything else other than managing the kernels. The frontend can deal with everything else.

The frontend can't interact with the filesystem, so it can't load notebooks, save notebooks, or do anything, really, without a notebook management service somewhere, even if it is S3 or Dropbox.

Note that the way we have designed these things, the kernel service should know nothing about notebooks and the notebook service should know nothing about kernels. And you can absolutely run each of these independently without the other. In the default notebook web application, we happen to run both of these services in a single server process, and a Session service that maintains a relationship of notebooks and kernels, while kernels know nothing about notebooks and notebooks know nothing about kernels. It should absolutely be possible to run all three services entirely separate from each other, so you could have a notebook editor webapp that doesn't support execution, or a pure kernel webservice that manages execution but no notebooks (examples of the latter already exist.

Thanks for listening!

Thanks for participating! We have lots of ideas, and lots of people have opinions on what we should do. Ultimately, we have to make decisions and figure out our own scope. One of the most challenging aspects of developing IPython is confining the scope of the project, and building modular pieces is one of the best ways that we can say "no, we aren't going to do that, but you can re-use these pieces of ours in your own app."

[andreabedini]

Thanks for participating! We have lots of ideas, and lots of people have opinions on what we should do. Ultimately, we have to make decisions and figure out our own scope. One of the most challenging aspects of developing IPython is confining the scope of the project, and building modular pieces is one of the best ways that we can say "no, we aren't going to do that, but you can re-use these pieces of ours in your own app."

Thanks for this. I realise that IPython is way more complex and customizable than I thought! The singlecell implementation is very interesting, I will redirect my experiments there.

[ellisonbg]

Work continuing on PR #4303.

[ellisonbg]

This work is done and merged into master. UI will be a separate effort in 3.0.