Refactoring notebook managers and adding Azure backed storage. #2045

ellisonbg · 2012-06-26T23:17:56Z

I have created a base class for all notebook managers. Our existing, file-based store, is now in filenbmanager.py. I have also created a new Azure Blob based backed notebook manager.

fperez · 2012-06-26T23:46:51Z

This looks good on first view, but it's obviously 0.14 material, tagging as such.

ellisonbg · 2012-06-26T23:54:39Z

Yep, strongly so. One thing that I am running into is that we have hardcoded logic about the notebook_dir everywhere. But that notion only makes sense for the file based notebook manager. I have going to commit some fixes that work around this fact, but we are going to have to rethink this.

fperez · 2012-06-27T00:04:46Z

Furthermore, the notebook_dir support is kind of borked now: it refers to the storage dir, but the cwd of the actual kernels is still where the server starts. Ideally it would do both, so one can (in file-based storage) configure the notebook dir in a config profile and simply start things from wherever, knowing that all kernels will come up in the right place.

I'd actually like to see that fixed for 0.13, but I don't see it happening.

fperez · 2012-06-27T00:06:55Z

Scratch that! It was #1958, and Min already fixed it in #1951 :) So all clear for 0.13 :)

Gee, we're merging so much code so fast, that I don't even remember issues I opened only a week ago. The problems we have :)

ellisonbg · 2012-07-17T20:53:59Z

@fperez do you think this is ready to go in?

fperez · 2012-07-18T03:11:22Z

IPython/frontend/html/notebook/azurenbmanager.py

+
+import datetime
+
+import azure


Is this guaranteed to be available by default on all azure images, or do users need to manually install the necessary sdk on their images? If so, do we need to update our setup.py to declare this as an optional dependency?

Users have to install this by hand and we should add it to our setup.py.

fperez · 2012-07-18T03:14:37Z

Overall the code looks clean, but there are a few things that worry me:

untested/untestable: since the only way to test this stuff for real is to run it on azure, it's not exactly test suite material. But I am becoming increasingly uncomfortable with the amount of new code we're putting in without proper test coverage. I'm wondering if at least using a mock object that mocks azure would be viable...
docs: I think it needs somewhere a bit of mention in the documentation. I know we have very little in the way of docs about the notebook architecture and possibilities, but we need to start somewhere. Might as well be here :)

The former I'm not really sure what to do about, the latter is eminently addressable... Otherwise the code does look great, and I'm thrilled to see it in. Good job!

ellisonbg · 2012-07-18T04:51:31Z

I will add some docs for this.

minrk · 2012-07-18T14:30:46Z

untested/untestable: since the only way to test this stuff for real is to run it on azure, it's not exactly test suite material. But I am becoming increasingly uncomfortable with the amount of new code we're putting in without proper test coverage. I'm wondering if at least using a mock object that mocks azure would be viable...

This is like the PBS/etc. Launchers, which simply cannot be automatically tested. We can do some amount of testing with dummy objects, but I'm not sure how useful that is. Perhaps what is more useful, if more cumbersome, is manual tests a la docs/examples/tests, to make sure things work. When changing this code, run the test script / step through the test process, etc.

For instance, we managed to cut 0.13 with PBS/SGE support in ipcluster that totally doesn't work, so we need to figure something out.

re: notebook_dir

We use notebook_dir for three technically unrelated values, that are all the same by coincidence right now:

the place where notebooks are stored (only relevant for file-based nbserver)
the place where kernels are started (no reason to force these to be the same)
the path used for serving local files from the notebook area (again, easily decoupled)

Is there more to decouple than that?

ellisonbg · 2012-07-18T15:53:01Z

On Wed, Jul 18, 2012 at 7:31 AM, Min RK
reply@reply.github.com
wrote:

untested/untestable: since the only way to test this stuff for real is to run it on azure, it's not exactly test suite material. But I am becoming increasingly uncomfortable with the amount of new code we're putting in without proper test coverage. I'm wondering if at least using a mock object that mocks azure would be viable...

This is like the PBS/etc. Launchers, which simply cannot be automatically tested. We can do some amount of testing with dummy objects, but I'm not sure how useful that is. Perhaps what is more useful, if more cumbersome, is manual tests a la docs/examples/tests, to make sure things work. When changing this code, run the test script / step through the test process, etc.

The main difficulty with testing this is that it requires the Azure
credentials, which we definitely can't version control. The test
script in this case would be a config file that has those credentials
and sets the right notebook manager.

For instance, we managed to cut 0.13 with PBS/SGE support in ipcluster that totally doesn't work, so we need to figure something out.

re: notebook_dir

We use notebook_dir for three technically unrelated values, that are all the same by coincidence right now:

the place where notebooks are stored (only relevant for file-based nbserver)

the place where kernels are started (no reason to force these to be the same)

the path used for serving local files from the notebook area (again, easily decoupled)

Is there more to decouple than that?

3 is the hardest to decouple. The reason is that eventually the
kernel will be run on a different host than the notebook server. The
notebook server simply won't have access to the local files of the
kernel. Don't know how to handle that. The other two should be easy
to decouple.

Reply to this email directly or view it on GitHub:
#2045 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

minrk · 2012-07-21T03:45:16Z

The main difficulty with testing this is that it requires the Azure
credentials, which we definitely can't version control. The test
script in this case would be a config file that has those credentials
and sets the right notebook manager.

Precisely - just like the launchers, it could only be run by users with an appropriate environment, and cannot be involved in the automatic test suite.

3 is the hardest to decouple. The reason is that eventually the
kernel will be run on a different host than the notebook server.

Is it possible that local files would simply be unavailable in cases where the kernel's filesystem is not visible to the nbserver? That honestly seems like a reasonable answer to me, since this is not a 100% critical feature.

minrk · 2012-07-21T05:35:35Z

This one is only waiting for docs, right? And a rebase, now.

ellisonbg · 2012-07-22T20:45:11Z

Yes, I just need to finish the docs.

On Fri, Jul 20, 2012 at 10:35 PM, Min RK
reply@reply.github.com
wrote:

This one is only waiting for docs, right? And a rebase, now.

Reply to this email directly or view it on GitHub:
#2045 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

minrk · 2012-07-23T19:21:03Z

Is there a reason for the base class to be called BaseNotebookManager rather than NotebookManager? This change would mean that config for 0.13 will be broken in 0.14, that would work perfectly fine otherwise.

Since we have coupled class names to config files, I think we need a pretty high bar for renaming configurable classes.

minrk · 2012-07-23T19:28:28Z

IPython/frontend/html/notebook/filenbmanager.py

-        self.save_notebook_object(notebook_id, nb)
-        return notebook_id
+    def log_info(self):
+        self.log.info("Serving notebooks from local directory: %s", self.notebook_manager.notebook_dir)


This should be self.notebook_dir, not self.notebook_manager.notebook_dir

benjiec · 2012-08-08T16:20:44Z

I closed pull request #2155 and adjusted my notebook manager to use Brian's work. Thanks Brian.

Noted the following bug in the azurenb branch, causing the default file system based nb manager to fail

diff --git a/IPython/frontend/html/notebook/filenbmanager.py b/IPython/frontend/html/notebook/filenbmanager.py
index 6389ec1..34d7d26 100644
--- a/IPython/frontend/html/notebook/filenbmanager.py
+++ b/IPython/frontend/html/notebook/filenbmanager.py
@@ -193,4 +193,4 @@ class FileNotebookManager(BaseNotebookManager):
         return name

     def log_info(self):
-        self.log.info("Serving notebooks from local directory: %s", self.notebook_manager.notebook_dir)
+        self.log.info("Serving notebooks from local directory: %s", self.notebook_dir)

benjiec · 2012-08-08T17:08:43Z

Here is another bug fix to the azurenb branch, it was still referring to NotebookManager


diff --git a/IPython/frontend/html/notebook/notebookapp.py b/IPython/frontend/html/notebook/notebookapp.py
index 167965d..f867155 100644
--- a/IPython/frontend/html/notebook/notebookapp.py
+++ b/IPython/frontend/html/notebook/notebookapp.py
@@ -50,7 +50,7 @@ from .handlers import (LoginHandler, LogoutHandler,
     RSTHandler, AuthenticatedFileHandler, PrintNotebookHandler,
     MainClusterHandler, ClusterProfileHandler, ClusterActionHandler
 )
-from .notebookmanager import NotebookManager
+from .basenbmanager import BaseNotebookManager
 from .clustermanager import ClusterManager

 from IPython.config.application import catch_config_error, boolean_flag
@@ -215,7 +215,7 @@ flags['read-only'] = (
 )

 # Add notebook manager flags
-flags.update(boolean_flag('script', 'NotebookManager.save_script',
+flags.update(boolean_flag('script', 'BaseNotebookManager.save_script',
                'Auto-save a .py script everytime the .ipynb notebook is saved',
                'Do not auto-save .py scripts for every notebook'))

@@ -232,7 +232,7 @@ aliases.update({
     'port-retries': 'NotebookApp.port_retries',
     'keyfile': 'NotebookApp.keyfile',
     'certfile': 'NotebookApp.certfile',
-    'notebook-dir': 'NotebookManager.notebook_dir',
+    'notebook-dir': 'BaseNotebookManager.notebook_dir',
     'browser': 'NotebookApp.browser',
 })

@@ -260,7 +260,7 @@ class NotebookApp(BaseIPythonApplication):
     """
     examples = _examples

-    classes = IPythonConsoleApp.classes + [MappingKernelManager, NotebookManager]
+    classes = IPythonConsoleApp.classes + [MappingKernelManager, BaseNotebookManager]
     flags = Dict(flags)
     aliases = Dict(aliases)

@@ -410,7 +410,7 @@ class NotebookApp(BaseIPythonApplication):
             else:
                 self.file_to_run = f
                 nbdir = os.path.dirname(f)
-            self.config.NotebookManager.notebook_dir = nbdir
+            self.config.BaseNotebookManager.notebook_dir = nbdir

     def init_configurables(self):
         # force Session default to be secure

benjiec · 2012-08-08T17:12:53Z

My patches are in https://github.com/benjiec/ipython/tree/azurenb

I will create a pull request to Brian's branch

I have created a base class for all notebook managers. Our existing, file-based store, is now in filenbmanager.py. I have also created a new Azure Blob based backed notebook manager.

ellisonbg · 2012-08-11T00:45:12Z

OK, I have addressed all of the comments and the test suite passes without the azure package installed. I have also added some minimal docs about using a custom notebook manager. Merging.

Refactoring notebook managers and adding Azure backed storage.

minrk · 2012-08-11T02:07:42Z

I wish you had pinged, since you did not actually address all comments above.

Unaddressed:

you, yourself mentioned adding something about azure to setup.py, but nothing came of it
The question about renaming NotebookManager to BaseNotebookManager, which breaks config unnecessarily went unanswered.

And the rebased commits re-introduced use of the deprecated assertEquals name, which was otherwise removed.

fperez · 2012-08-11T02:13:13Z

@ellisonbg, in general I think it's a good idea to give one last ping on PRs that have had good back-and-forth (such as this one). Typically what we've been doing is that then the reviewer, if all looks good, ends up merging right away. That way it doesn't really add any unnecessary lag to the process, it helps catch situations like this one, and it means that when we see the merge logs, the main reviewer tends to stand out right away without having to come back to read the PR process itself.

ellisonbg · 2012-08-11T03:03:07Z

I decided that it probably doesn't make sense to add the azure package to
setup.py.
I completely missed the NotebookManager naming thing. I will submit a PR
for that.
I was mostly sick of this one hanging over my head, but points taken to
heart...

On Fri, Aug 10, 2012 at 7:13 PM, Fernando Perez notifications@github.comwrote:

@ellisonbg https://github.com/ellisonbg, in general I think it's a good
idea to give one last ping on PRs that have had good back-and-forth (such
as this one). Typically what we've been doing is that then the reviewer, if
all looks good, ends up merging right away. That way it doesn't really add
any unnecessary lag to the process, it helps catch situations like this
one, and it means that when we see the merge logs, the main reviewer tends
to stand out right away without having to come back to read the PR process
itself.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2045#issuecomment-7663107.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

fperez · 2012-08-11T03:08:07Z

On Fri, Aug 10, 2012 at 8:03 PM, Brian E. Granger
notifications@github.comwrote:

I decided that it probably doesn't make sense to add the azure package
to
setup.py.

I completely missed the NotebookManager naming thing. I will submit a PR
for that.

I was mostly sick of this one hanging over my head, but points taken to
heart...

No worries :) Min quickly updated the deprecated asserts and I reviewed it
and merged it, so no biggie.

ellisonbg · 2012-08-11T03:20:45Z

OK new PR submitted.

On Fri, Aug 10, 2012 at 8:08 PM, Fernando Perez notifications@github.comwrote:

On Fri, Aug 10, 2012 at 8:03 PM, Brian E. Granger
notifications@github.comwrote:

I decided that it probably doesn't make sense to add the azure package
to
setup.py.

I completely missed the NotebookManager naming thing. I will submit a
PR
for that.

I was mostly sick of this one hanging over my head, but points taken to
heart...

No worries :) Min quickly updated the deprecated asserts and I reviewed it
and merged it, so no biggie.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2045#issuecomment-7663431.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

Refactoring notebook managers and adding Azure backed storage.

fperez reviewed Jul 18, 2012
View reviewed changes

minrk reviewed Jul 23, 2012
View reviewed changes

Carreau mentioned this pull request Aug 8, 2012

Hook to add alternative notebook managers #2155

Closed

ellisonbg added 6 commits August 10, 2012 17:43

Refactoring notebook managers and adding Azure backed storage.

e3e9d9c

I have created a base class for all notebook managers. Our existing, file-based store, is now in filenbmanager.py. I have also created a new Azure Blob based backed notebook manager.

Fixing minor things for the Azure backed nb storage.

20f2b03

Fixing minor bug.

53930d6

Fixing minor bugs in notebookapp related to base class name.

cad3600

Adding exclusion for the azure module in iptest.

daf45eb

Adding AzureNotebookManager docs.

f4a7dcb

ellisonbg added a commit that referenced this pull request Aug 11, 2012

Merge pull request #2045 from ellisonbg/azurenb

c8c254c

Refactoring notebook managers and adding Azure backed storage.

ellisonbg merged commit c8c254c into ipython:master Aug 11, 2012

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request ipython#2045 from ellisonbg/azurenb

9c05615

Refactoring notebook managers and adding Azure backed storage.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring notebook managers and adding Azure backed storage. #2045

Refactoring notebook managers and adding Azure backed storage. #2045

ellisonbg commented Jun 26, 2012

fperez commented Jun 26, 2012

ellisonbg commented Jun 26, 2012

fperez commented Jun 27, 2012

fperez commented Jun 27, 2012

ellisonbg commented Jul 17, 2012

fperez Jul 18, 2012

ellisonbg Jul 18, 2012

fperez commented Jul 18, 2012

ellisonbg commented Jul 18, 2012

minrk commented Jul 18, 2012

ellisonbg commented Jul 18, 2012

minrk commented Jul 21, 2012

minrk commented Jul 21, 2012

ellisonbg commented Jul 22, 2012

minrk commented Jul 23, 2012

minrk Jul 23, 2012

benjiec commented Aug 8, 2012

benjiec commented Aug 8, 2012

benjiec commented Aug 8, 2012

ellisonbg commented Aug 11, 2012

minrk commented Aug 11, 2012

fperez commented Aug 11, 2012

ellisonbg commented Aug 11, 2012

fperez commented Aug 11, 2012

ellisonbg commented Aug 11, 2012

Refactoring notebook managers and adding Azure backed storage. #2045

Refactoring notebook managers and adding Azure backed storage. #2045

Conversation

ellisonbg commented Jun 26, 2012

fperez commented Jun 26, 2012

ellisonbg commented Jun 26, 2012

fperez commented Jun 27, 2012

fperez commented Jun 27, 2012

ellisonbg commented Jul 17, 2012

fperez Jul 18, 2012

Choose a reason for hiding this comment

ellisonbg Jul 18, 2012

Choose a reason for hiding this comment

fperez commented Jul 18, 2012

ellisonbg commented Jul 18, 2012

minrk commented Jul 18, 2012

ellisonbg commented Jul 18, 2012

minrk commented Jul 21, 2012

minrk commented Jul 21, 2012

ellisonbg commented Jul 22, 2012

minrk commented Jul 23, 2012

minrk Jul 23, 2012

Choose a reason for hiding this comment

benjiec commented Aug 8, 2012

benjiec commented Aug 8, 2012

benjiec commented Aug 8, 2012

ellisonbg commented Aug 11, 2012

minrk commented Aug 11, 2012

fperez commented Aug 11, 2012

ellisonbg commented Aug 11, 2012

fperez commented Aug 11, 2012

ellisonbg commented Aug 11, 2012