Improve interactive session save/restore facilities #112

Open
ipython opened this Issue May 10, 2010 · 15 comments

Comments

Projects
None yet
5 participants
Collaborator

ghost commented May 10, 2010

Original Launchpad bug 488953: https://bugs.launchpad.net/ipython/+bug/488953
Reported by: fdo.perez (Fernando Perez).

Early in IPython's life I tried to have full session saving, but that doesn't blindly work because things like sockets, interactively defined functions and many other objects can fail to pickle, and I simply punted on the problem. But it's a common and valid request to have convenient session state management; currently we have %store but we should offer better facilities.

  • A session saving command that can save as much of the state as possible, simply printing a warning of what variables didn't make it. Note that Mike McKern's dill pickler could be very useful here:

http://dev.danse.us/trac/pathos
http://dev.danse.us/trac/pathos/browser/dill/dill.py

  • Convenient functions for pulling any dict into the user's namespace (this is very easy, just updating user_ns with the other dict)
  • Expose ipvars() functionality in the builtin namespace, very useful for debugging.

For reference, see this thread on the nipy list:

http://mail.scipy.org/pipermail/nipy-devel/2009-November/002273.html

In particular the comments made by Gael here:
http://mail.scipy.org/pipermail/nipy-devel/2009-November/002349.html

minrk added a commit to minrk/ipython that referenced this issue Jul 1, 2013

Merge pull request #112 from juliantaylor/static-path
use notebookapp.DEFAULT_STATIC_FILES_PATH to get static path

allows use of nbconvert with system packaged versions of IPython
closes #97
@niallrobinson

This comment has been minimized.

Show comment Hide comment
@niallrobinson

niallrobinson Jul 30, 2013

Has there been any more progress on this issue? For what its worth I think it would be a killer feature that would attract everyone where I work to use IPython, especially if it was integrated into the notebook

Has there been any more progress on this issue? For what its worth I think it would be a killer feature that would attract everyone where I work to use IPython, especially if it was integrated into the notebook

@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Aug 23, 2013

Owner

Mike McKerns, the author of Dill, says he has fixed some bugs so that save_session() works from inside IPython:

http://trac.mystic.cacr.caltech.edu/project/pathos/changeset/511

http://mail.scipy.org/pipermail/ipython-dev/2013-August/012204.html

Owner

takluyver commented Aug 23, 2013

Mike McKerns, the author of Dill, says he has fixed some bugs so that save_session() works from inside IPython:

http://trac.mystic.cacr.caltech.edu/project/pathos/changeset/511

http://mail.scipy.org/pipermail/ipython-dev/2013-August/012204.html

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

It looks like there's going to be at least one more issue...

This code (inside of interactiveshell.InteractiveShell's prepare_user_module):

        if user_module is None and user_ns is not None:
            user_ns.setdefault("__name__", "__main__")
            class DummyMod(object):
                "A dummy module used for IPython's interactive namespace."
                pass
            user_module = DummyMod()
            user_module.__dict__ = user_ns

in the case where user_module == None, but user_ns != None, you are returned
a user_module where the instance is <IPython.core.interactiveshell.DummyMod at 0x165d310>...
which is bogus because DummyMod is defined inside of the above method in the InteractiveShell class
and is not located at the namespace where the module code places it.

This will make pickling fail for these objects unless: (1) you define DummyMod at the namespace the code assigns the DummyMod object to, or (2) don't monkey with the namespaces.

Anyone want to tell me if it's important to have DummyMod defined inside the method of the class and not where it shows up in the namespace? (e.g. does faking the namespace trigger something important for you in other parts of the code?). Defining DummyMod at IPython.core.interactiveshell.DummyMod would likely clear up pickling issues for whatever ipython is doing for interactively defined user modules here.

I don't have a test case here, just noticed it in the code, created the DummyMod object, and noted that it fails to pickle.

s = InteractiveShell()
m = s.prepare_user_module(None, {})
dill.dumps(m0)

You get a:

PicklingError: Can't pickle <class 'IPython.core.interactiveshell.DummyMod'>: it's not found as IPython.core.interactiveshell.DummyMod
Contributor

mmckerns commented Sep 9, 2013

It looks like there's going to be at least one more issue...

This code (inside of interactiveshell.InteractiveShell's prepare_user_module):

        if user_module is None and user_ns is not None:
            user_ns.setdefault("__name__", "__main__")
            class DummyMod(object):
                "A dummy module used for IPython's interactive namespace."
                pass
            user_module = DummyMod()
            user_module.__dict__ = user_ns

in the case where user_module == None, but user_ns != None, you are returned
a user_module where the instance is <IPython.core.interactiveshell.DummyMod at 0x165d310>...
which is bogus because DummyMod is defined inside of the above method in the InteractiveShell class
and is not located at the namespace where the module code places it.

This will make pickling fail for these objects unless: (1) you define DummyMod at the namespace the code assigns the DummyMod object to, or (2) don't monkey with the namespaces.

Anyone want to tell me if it's important to have DummyMod defined inside the method of the class and not where it shows up in the namespace? (e.g. does faking the namespace trigger something important for you in other parts of the code?). Defining DummyMod at IPython.core.interactiveshell.DummyMod would likely clear up pickling issues for whatever ipython is doing for interactively defined user modules here.

I don't have a test case here, just noticed it in the code, created the DummyMod object, and noted that it fails to pickle.

s = InteractiveShell()
m = s.prepare_user_module(None, {})
dill.dumps(m0)

You get a:

PicklingError: Can't pickle <class 'IPython.core.interactiveshell.DummyMod'>: it's not found as IPython.core.interactiveshell.DummyMod
@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Sep 9, 2013

Owner

@mmckerns : I wrote that, and I don't think there was any special reason to define DummyMod within the function, it was just convenience. It's not a code path we test very much, because when you start IPython normally, both user_module and user_ns are passed in as None, in which case, IIRC, we create an instance of the regular ModuleType, and use its __dict__ as user_ns. But it is an important case, because a lot of projects that offer an IPython shell to control other code (e.g. Django shell) will pass in user_ns.

Do you want to do a pull request to move it up to the module level? And double check that there are no obvious ill effects if you pass user_ns in - you can call IPython.start_ipython(user_ns={'a':123}). Thanks for taking the time to look into this.

Owner

takluyver commented Sep 9, 2013

@mmckerns : I wrote that, and I don't think there was any special reason to define DummyMod within the function, it was just convenience. It's not a code path we test very much, because when you start IPython normally, both user_module and user_ns are passed in as None, in which case, IIRC, we create an instance of the regular ModuleType, and use its __dict__ as user_ns. But it is an important case, because a lot of projects that offer an IPython shell to control other code (e.g. Django shell) will pass in user_ns.

Do you want to do a pull request to move it up to the module level? And double check that there are no obvious ill effects if you pass user_ns in - you can call IPython.start_ipython(user_ns={'a':123}). Thanks for taking the time to look into this.

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

@takluyver: Yeah, I'll see if I can alleviates the issue, and if I can and doesn't cause any new one I can see, I'll do a pull request.

Contributor

mmckerns commented Sep 9, 2013

@takluyver: Yeah, I'll see if I can alleviates the issue, and if I can and doesn't cause any new one I can see, I'll do a pull request.

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

@takluyver: can you tell me a little bit more about the reason for this DummyMod? It seems that it's a substitute for __main__ as a namespace for objects built in an interactive session. Why? Do you save this DummyMod instance anywhere when you shut down an ipython session? It doesn't look like it on cursory exam. In 0.13, ipython dumped all interactively built code into __main__... and dill can pickle everything in the session built in __main__ pretty easily... but if it's being masked in DummyMod... I'll have to know more about what the intent of this guy is.

What I see happening is that if I use 0.13, I can pickle anything built interactively, and save the entire session in a .pkl file. Then if I start 1.0, and load the .pkl file, I get everything from my saved session. However, if I try to save the session in 1.0, I just get the DummyMod stub as the only thing I'm saving in the .pkl file. A fix would be that if you are saving a session, then dump DummyMod's __dict__ into __main__'s. However, that either means that I have to add it to dill and it's getting too much of injecting a ipython workaround into dill for my comfort... or we'd have to overload dill's dump_session with a ipython specific ipython.dump_session.

Thoughts here before I dig into this deeper? I'd like to learn as little of ipython's internals as possible in making this work. ;)

Contributor

mmckerns commented Sep 9, 2013

@takluyver: can you tell me a little bit more about the reason for this DummyMod? It seems that it's a substitute for __main__ as a namespace for objects built in an interactive session. Why? Do you save this DummyMod instance anywhere when you shut down an ipython session? It doesn't look like it on cursory exam. In 0.13, ipython dumped all interactively built code into __main__... and dill can pickle everything in the session built in __main__ pretty easily... but if it's being masked in DummyMod... I'll have to know more about what the intent of this guy is.

What I see happening is that if I use 0.13, I can pickle anything built interactively, and save the entire session in a .pkl file. Then if I start 1.0, and load the .pkl file, I get everything from my saved session. However, if I try to save the session in 1.0, I just get the DummyMod stub as the only thing I'm saving in the .pkl file. A fix would be that if you are saving a session, then dump DummyMod's __dict__ into __main__'s. However, that either means that I have to add it to dill and it's getting too much of injecting a ipython workaround into dill for my comfort... or we'd have to overload dill's dump_session with a ipython specific ipython.dump_session.

Thoughts here before I dig into this deeper? I'd like to learn as little of ipython's internals as possible in making this work. ;)

@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Sep 9, 2013

Owner

For others reading this, I've tried to explain the rationale for DummyMod in @mmckerns PR #4186.

I'm puzzled about the difference between 0.13 and 1.0 - I don't think we changed any of this code since 0.12.

Owner

takluyver commented Sep 9, 2013

For others reading this, I've tried to explain the rationale for DummyMod in @mmckerns PR #4186.

I'm puzzled about the difference between 0.13 and 1.0 - I don't think we changed any of this code since 0.12.

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

I get the three bullets you present in PR #4186 ... that's just what the encapsulating method does.
What I mean is why use DummyMod for the main ipython namespace? That seems to be what's going on.

I'll have to look into it, but on quick exam it looks like in 0.13 ipython's __main__ is simply the python interpreter's __main__; while in 1.0, ipython's __main__ is actually created with DummyMod.

I didn't dig to far, but it seems like the init_create_namespaces in core.interactiveshell.InteractiveShell might be getting called with user_ns != None by default... and that would put you in the case that the default interactive shell would be a DummyMod... which you seem to be saying is supposed to be a corner case.

Maybe what I suggest above is not actually the case... but is seem like it is. It wouldn't be that code around DummyMod was changed, it would be that there was a change in how the __main__ namespace was initialized, I believe.

It just looks like InteractiveShell is doing some funny stuff with namespacing that I'll have to actually sit and think about... and I don't really like thinking too much (about how code works). I'm assuming at this point it's best I just suck it up and go digging.

Contributor

mmckerns commented Sep 9, 2013

I get the three bullets you present in PR #4186 ... that's just what the encapsulating method does.
What I mean is why use DummyMod for the main ipython namespace? That seems to be what's going on.

I'll have to look into it, but on quick exam it looks like in 0.13 ipython's __main__ is simply the python interpreter's __main__; while in 1.0, ipython's __main__ is actually created with DummyMod.

I didn't dig to far, but it seems like the init_create_namespaces in core.interactiveshell.InteractiveShell might be getting called with user_ns != None by default... and that would put you in the case that the default interactive shell would be a DummyMod... which you seem to be saying is supposed to be a corner case.

Maybe what I suggest above is not actually the case... but is seem like it is. It wouldn't be that code around DummyMod was changed, it would be that there was a change in how the __main__ namespace was initialized, I believe.

It just looks like InteractiveShell is doing some funny stuff with namespacing that I'll have to actually sit and think about... and I don't really like thinking too much (about how code works). I'm assuming at this point it's best I just suck it up and go digging.

@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Sep 9, 2013

Owner

Hmm, you seem to be right. I'll dig into that, I'm not sure what has changed.

Owner

takluyver commented Sep 9, 2013

Hmm, you seem to be right. I'll dig into that, I'm not sure what has changed.

@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Sep 9, 2013

Owner

It's an exciting result of our application/config system. I hate debugging this stuff...

Owner

takluyver commented Sep 9, 2013

It's an exciting result of our application/config system. I hate debugging this stuff...

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

Yes, I'm into that code as well... it's hard to follow.

Contributor

mmckerns commented Sep 9, 2013

Yes, I'm into that code as well... it's hard to follow.

@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Sep 9, 2013

Owner

See #4188. If the fix is that simple, we might be able to get it backported for a 1.x release.

Owner

takluyver commented Sep 9, 2013

See #4188. If the fix is that simple, we might be able to get it backported for a 1.x release.

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

That looks like that would cause it. I'll manually make the change, and see if that fixes the issue (#112) here too.

Contributor

mmckerns commented Sep 9, 2013

That looks like that would cause it. I'll manually make the change, and see if that fixes the issue (#112) here too.

@takluyver

This comment has been minimized.

Show comment Hide comment
@takluyver

takluyver Sep 9, 2013

Owner

I've checked that that does avoid it using DummyMod in the default case.

Owner

takluyver commented Sep 9, 2013

I've checked that that does avoid it using DummyMod in the default case.

@mmckerns

This comment has been minimized.

Show comment Hide comment
@mmckerns

mmckerns Sep 9, 2013

Contributor

Awesome. Just tried it. Your change in #4188 also restores the ability to use dill.dump_session to save the entire ipython session.

Contributor

mmckerns commented Sep 9, 2013

Awesome. Just tried it. Your change in #4188 also restores the ability to use dill.dump_session to save the entire ipython session.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment