Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a %with magic #3191

Open
asmeurer opened this issue Apr 16, 2013 · 25 comments

Comments

@asmeurer
Copy link
Contributor

commented Apr 16, 2013

I'm willing to take a shot at implementing this, but first I want to clear up how it should work. Context managers are becoming increasingly popular, and I think it would be useful to be able to work with them interactively. That is, I want to be able to do something like

In [1]: %with open('file') as f
Context manager f started

In [2]: lines = f.read()

In [3]: %endwith
Context manager f ended

In [4]: f
Out[4]: <closed file 'file', mode 'r' at 0x10c821540>

This would be equivalent to

with open('file') as f:
    lines = f.read()
f

Of course, reading a file interactively is no big deal, but I also deal with context managers where you may want to deal with their temporary change in state for an arbitrary number of commands (like the assuming context manager in SymPy).

So my questions:

  • This isn't implemented already is it (I didn't find it)?
  • Is this indeed something that the IPython project wants?
  • (Assuming no and yes), how should the syntax look? In particular, what should the names of the commands that enter and end the magic be? Is it possible to name a magic after a reserved name (with) (see also #2984)? If not, I was thinking %With.
  • Where would the code for this go? The best fit after a quick perusing of the code to me seemed to be IPython.core.magics.basic.
  • Any hints on a similar magic that I could base the implementation off of? :)
@jasongrout

This comment has been minimized.

Copy link
Member

commented Apr 17, 2013

Interesting. So this would be a magic that affects not just the line or cell, but many lines. I presume you would handle a whole stack of %with statements, so you could nest them?

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

Yes, naturally. It should also support the multiple with statement syntax.

@bfroehle

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2013

How do you plan on handling the ability for a context manager to suppress exceptions? Or will that feature be ignored?

It is possible to create a %with magic, but there might be side effects I don't foresee.

In [11]: %alias_magic with pwd

In [12]: with
Out[12]: u'/home/bfroehle'
@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

I was planning on emulating it exactly, since in general an exception means something went wrong and thus the context should end. I guess I could make it configurable.

@bfroehle

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2013

I've got a simple implementation which you could improve on at https://gist.github.com/bfroehle/5401361.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

Well that makes things easier :) I'll look at this later. Some thoughts:

  • I wonder if we can somehow borrow the stdlib parsers instead of using a regular expression
  • Requiring the final : seems unnecessary, no?
  • Shouldn't the + for the as be included with the variable name. You can either have with code or with code as name. Actually, the full spec uses target, not name. For example, with ctx as (a, b) is valid syntax (see http://docs.python.org/2/reference/compound_stmts.html#with). That's why I think we should try to use the stdlib parsers if possible. @takluyver is there an elegant way to do this with the parsing stuff you've worked on?
  • So apparently it is possible to use %with. That's good. I was thinking that just %with with no arguments could end the context. Or would it be better to use %endwith.
  • I really think we should emulate exception catching. This is fundamental to how some context managers work, and I think it will be expected by users. So it should be wrapped in a try, finally block.

This is already pretty cool, though.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

Maybe we should special-case SyntaxErrors so that people with butter fingers don't get pissed off at it, though.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

Or maybe on an exception it should ask. I have so little experience with context managers that I really don't know what the common use cases are, except for open. I'd really appreciate feedback on this particular point.

@takluyver

This comment has been minimized.

Copy link
Member

commented Apr 17, 2013

What is the advantage of this? The example you gave could just as easily be done with f=open(...) and f.close(). I'm willing to be convinced, but I'd like to see an example of something that's actually easier or better with this magic.

The advantage of context managers is that they do something automatically (close a file, release a lock, or whatever) when execution leaves that block, whether by execution continuing normally or an exception being raised. If you have to invoke the cleanup manually, why not just do that with the existing syntax?

I'm also wary of adding IPython syntax so similar to Python syntax (especially since magic commands can be used without the % prefix). This could be really confusing for someone learning about context managers - you leave off the colon, and instead of throwing a SyntaxError, IPython assumes you meant something similar but subtly different. I don't like With either - new programmers often have trouble with case-sensitivity, W looks a lot like w, and our convention for magic commands is lowercase. One of the main reasons we disabled autocall by default was because the syntax laxity was confusing new programmers.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

open is perhaps a bad example because it doesn't really matter if you close the file or not when working interactively. My personal example would be SymPy's assuming context manager, which works like

with assuming(Q.positive(x)):
    print ask(Q.real(x)) # prints True
    print refine(sqrt(x**2)) # prints x

You can also add the assumption globally manually, but this syntax is cleaner, and easier to remember. I'll try to research some other context managers.

Regarding the name, we could choose something more verbose like %with_context. It would be more typing, though.

@takluyver

This comment has been minimized.

Copy link
Member

commented Apr 17, 2013

I'm inclined to think that it should be up to libraries to provide an API that works with or without context managers, rather than relying on IPython to add some odd context-manager-but-not-really wrapper. More broadly, I think sticking to Python syntax as much as possible makes code samples more readable.

Names: I'd avoid with altogether, because it's doing something quite different from the with statement. I'd say it should be something like %enter_context or %start_context, with a corresponding exit_context/end_context magic to close it.

If you want to do this, I'd suggest you do it in an extension for now - which anyone is always free to do, of course. If we see a lot of people using it, we might want to incorporate it into IPython later, but I'm not sold on that at present.

@minrk

This comment has been minimized.

Copy link
Member

commented Apr 17, 2013

I don't know about this one in particular, but we have actually been trying to reduce the number of execution-state-affecting magics (e.g. %autopx).

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

OK, I can make it an extension. @takluyver what I really wanted to know from you is if it's possible easily to parse the arguments of %with as a with statement.

@bfroehle

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2013

Aaron: You can certainly use the ast module in Python:

>>> import ast
>>> line = 'f(x) as y, g(z):'
>>> M = ast.parse('with ' + line + ' pass')
>>> W = M.body[0]
>>> while True:
...     expression = ast.Expression(W.context_expr)
...     names = W.optional_vars
...     code = compile(expression, '<with ...>', 'eval')
...     # Run code to produce context manger, enter context manager, assign to names
...
...     W = W.body[0]
...     if isinstance(W, ast.Pass):
...         break
...     elif isinstance(w, ast.With):
...         continue
...     else:
...         raise TypeError('Should not get here')
@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

Yes, I know. I was wondering if there is a nice way to do it given the (relatively) new ast transformer stuff that's built-into IPython thanks to @takluyver's work.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2013

What if it indented the code somehow, so that it looks just like a "with block", but unlike a normal block of code, it ran line-by-line (I realize that this doesn't generalize nicely outside the terminal and possibly qtconsole)? The issue for me is that unlike every other block construct in Python, if, for, while, def, class, and try, there is no real reason that you need to "finish" the block to know completely what to do with each line with with. Therefore, interactivity makes sense on a line-by-line basis for with, rather than on a block basis.

@takluyver

This comment has been minimized.

Copy link
Member

commented Apr 18, 2013

I don't see a way to do it just with ast transformers, because that requires that it's already parsed as valid Python. As far as I know you can only feed Python's parser complete statements or expressions, not snippets like a with-expression.

I see what you're after with the line-by-line execution, but at best it would require an awful lot of messing with the internals of IPython's execution model, and taking apart Python code to run it a different way. The added complexity hugely outweighs any gain I can see, given that most uses of a context manager have an easy non-context-manager API. Plus, as you point out, it doesn't fit well with a cell-based frontend like the notebook.

@bfroehle

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2013

You might want to consider doing this outside the land of IPython by just creating a ContextManagerStack class.

>>> cm = ContextManagerStack()
>>> f = cm.push(open('file'))
>>> pass # use f
>>> cm.pop()
>>> f.closed
True
@ellisonbg

This comment has been minimized.

Copy link
Member

commented May 1, 2013

I have to agree with Min that we are trying to reduce magics that affect execution state. Also, this one strikes me as sort of a needless argument with the syntax of the language. It is not at all clear why

%with
this
that 
the other thing

Is better than

with:
    this
    that
    the other thing

The added code complexity just isn't worth it - at least not to exist inside IPython's code base.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2013

Because you want to see the output of each line as you execute it.

@ellisonbg

This comment has been minimized.

Copy link
Member

commented May 1, 2013

Hmm, I am missing something, how does a magic help?

@bfroehle

This comment has been minimized.

Copy link
Contributor

commented May 1, 2013

@ellisonbg I think @asmeurer meant you could have multiple cells between the opening and closing of the context manager, as in this example.

In [1]: %load_ext withmagic

In [2]: %with open('withmagic.py') as f:
Context manager f started

In [3]: print(f.readline())
# -*- coding: utf-8 -*-


In [4]: len(f.readlines())
Out[4]: 54

In [5]: %endwith
Context manager f ended
@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2013

This honestly makes less sense in a cell-based environment, and even less in the notebook where you ideally want the output to look nice. In those cases, definitely use a normal with block. My motivation was the line-based terminal IPython, which I use as a throw away interactive calculator (usually for SymPy, hence my motivating with assuming example).

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2013

The other advantage here is that you cannot easily get IPython output of code executed in a with statement. The best you can do is

with stuff:
    a = result
a

which makes things even more annoying if you're trying to do interactive exploration.

@asmeurer

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2013

Fabric has a lot of good examples of context managers that aren't just open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.