Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add magic function %drun to run code in debugger #3071

Merged
merged 2 commits into from

8 participants

@tkf

%drun is pretty much like %prun but instead it starts debugger rather than profiler.

This PR is made on top of #3066. I will rebase it after #3066 is merged if needed.

@takluyver
Owner

Is there an argument for this besides being quicker to type? The name drun isn't particularly self-explanatory.

@tkf

If there was not %pdb or %debug, I would use it but I couldn't. So I thought I'd make it consistent with %prun. Another option I had in mind was %with_debugger (or the one without underscore).

@takluyver
Owner

To my mind, a slightly longer, more descriptive name is preferable. Especially with the notebook, we're trending towards better readability, rather than saving every keystroke. But I'm not a big user of %run -d, so I'll leave it up to others.

@minrk
Owner

We've generally been moving away from cryptic abbreviated names (e.g. replacing %hist with %history). Like @takluyver, I almost never %run -d, so I don't think it earns a first-class shortcut (I would actually say the same for %prun, were it not for historical reasons).

@tkf

In the notebook I think you can't use debugger as it does not support STDIN. I'd use it mostly in terminal. Though I don't have strong opinion and I am OK with long name.

@Carreau
Owner

In the notebook I think you can't use debugger as it does not support STDIN.

We are waiting for @minrk PR:

img

@tkf

@Carreau coooool!

So we want to rename it to %with_debugger or another name? An alternative way to add this functionality is to have %exec magic command which is like %run but works for statements. Then we can integrate commands such as %prun, %timeit, %time, %capture etc. into this. This is probably better because you can do thing like "capturing output while profiling".

@minrk
Owner

That PR is going to be dependent on #3011 (it's my stdin branch, if you want to check it out).

@takluyver
Owner

Ah, I'd missed that it worked on statements, I thought you were just proposing a shortcut for %run -d.

Perhaps %debug_run or %rundebug. We could even overload the %debug magic, as it doesn't take any options at present, but maybe that's confusing.

@minrk
Owner

I was thinking the same thing about extending %debug, but I, too, worry about too much overload. The fact that %drun works on statements makes it very clear to me that it should not have run anywhere in the name, because run really means running files in IPython.

So if we are extending %debug, we would get things like:

# old behavior
%debug 
# debug a statement:
%debug my_statement()
# debug a cell:
%%debug
def foo():
    1/0
foo()

That doesn't seem too confusing to me, as it strictly adds functionality and doesn't change anything, right? It also seems fairly clear, as the sentence structure of debug statement actually describes what is going to happen. If anything, I think %debug is a better name for this new functionality than what it does currently.

@tkf

I think overloading %debug is good idea.

The fact that %drun works on statements makes it very clear to me that it should not have run anywhere in the name

Then I think %prun should be renamed, because it does not act on python file. I took the idea of the name %drun from it.

@takluyver
Owner
@minrk
Owner

It could be my own personal hangup about prun - I wouldn't touch it, and maybe others don't have this association of run == file.

@tkf

I remember that I thought %prun is inconsistent with %run when I first saw it. How about %profile_exec or %exec -p? Or can't we just rename %profile to %ipy_profile or something like that? Current %profile seems not a function that is used in user's program.

@takluyver
Owner
@tkf

Ah, I forgot that you can omit %, so that's why you want to avoid exec.

IPython is about executing code so I think it does not make sense to have better and shorter name for kind of meta management magic function (%profile) than the one about executing code (%profile_code). Also, probably 1.0 is a good timing for changing this.

@minrk
Owner

I don't have any attachment to the existing %profile magic - I doubt it has much use, and can be renamed to %show_profile or just removed.

@minrk
Owner

I would like to get @fperez's opinion about the run nomenclature - it could be that what I said above is plain wrong, and this being identical to prun, but with d for debug, nothing actually needs to change.

@fperez
Owner

I've been thinking about this quite a bit, and I think it's a good time to rationalize things out in the code execution magics department. Since running code is more or less the heart of ipython, we should do our best to have a clean and consistent api here.

I mostly concur with @minrk in the idea that run is for scripts, the others for code blocks; this wasn't really by explicit design but rather as an accident of history, but we might as well run with it as it lets us present a consistent message.

The rationale for what comes below: %run takes special flags for three things: timing, profiling and debugging. And we also several dedicated magics for those things. We should harmonize all this with cleaner names and an easy to understand mapping between %run and the dedicated functions, where the only difference is that %run applies to scripts/modules and the others are for local blocks (line/cell).

So here's what I think we should do:

  • Rename %profile to %active_profile or similar (current?). The word 'profile' is just too important to leave it to this slightly obscure use case.

  • Unify the proposed %drun with %debug as indicated above. Plain %debug would continue to work as today, but with arguments it would be used to debug simple statements or whole blocks (in cell magic mode).

  • Rename %prun to %profile, leaving %prun to just raise a UsageError about the renaming.

  • Unify %time and %timeit into a single function, named %time, that does effectively what %timeit does today. The simplistic behavior that %time has would be obtained with a -s flag or similar. There really shouldn't be two timing functions.

  • While we do this, the timing ones should grown an option to return the timing value(s) in seconds. I can't count the number of times I've needed this to automate timing comparisons.

In practice this plan really should be odne in separate PRs, one per topic (timing, profiling, debugging). I just wanted to lay out the plan here for feedback. Once we agree, this PR can focus on debugging and hopefully two more will come for profiling and timing.

How does this sound?

@jstenar
Collaborator
@fperez
Owner

@jstenar, you mean instead of %debug, %time, %profile? It seems to me that keeping them as separate commands for blocks is not a bad option, since they will be most likely used in the notebook/qtconsole, and that makes them short and readable. We're trying to make %run itself simpler, not more complex, and adding sub-command machinery into it seems to me like moving in the wrong direction...

Do you see any gains with your idea?

@jstenar
Collaborator

I believe the discoverability would be better with subcommands. I mean %debug, %time, and %profile are just different ways of %run-ning code. To me it makes more sense since these commands just represent different ways of %running code.

I believe using subcommands can simplify the code. By switching to argparse and subcommands you can have a separate function for each subcommand instead of a giant function as it is now. However our argparse decorators would have to be updated to handle subcommands.

@bfroehle

Discoverability could also be handled by improving the corresponding docstrings. For example %run could include something just like man pages do:

See Also
--------
    %debug, %profile, %time

I'm personally -1 to putting everything in %run. It's already way too complicated. Instead, what we should focus on is a nice class based design for these types of commands with several entry points (%run, %debug, %time, %profile) which can share common machinery.

@fperez
Owner

I tend to agree with @bfroehle here, for two reasons:

  1. run is already huge and complex, adding sub-command machinery into a magic is only going to make it worse.

  2. It lets us keep the mental model of " run for scripts, explicit commands for code blocks". Which I think happens to also work visually quite well, because %run tends to be more used in the terminal/console/qtconsole, whereas the individual commands for multiline blocks are more of a notebook thing, and by being explicit words, they make for more readable notebooks in the long run.

@jstenar, does that seem reasonable?

@jstenar
Collaborator
@takluyver
Owner

To frame this a different way, there are at least three special kinds of execution (timed, profiled, debugged) that people might want for either a block of code in IPython, or code from an external file. What's the most intuitive way to handle that.

At present:

  • %run means run code from a file, and it takes flags -t, -d and -p for timing, debugging and profiling. Concise once you know it, but not especially obvious to newcomers.
  • There are separate %time[it], %prun (to become %profile?) and proposed %drun magics for using code inline. %prun and %drun are rather non-obvious.

Another possibility might be to introduce -f (file) flags to the execution magic commands, so you could do things like:

# Code inline
%profile do_something(x, y)

# Code from file
%profile -f script.py
@takluyver takluyver closed this
@takluyver takluyver reopened this
@takluyver
Owner

(Sorry, didn't mean to close)

@tkf
tkf commented

How about %exe [options] CODE (or even %ex?) to execute (eval) code? %run supports specifying multiple execution "modes". For example, you can profile and measure timing simultaneously by %run -t -p FILE.py. This feature cannot be implemented if you choose "execution mode as a command" approach (i.e., %time, %profile and %debug). One negative point of %exe is that %exc CODE is equivalent to CODE so that you are always going to give at least one option. Probably command such as %debug can be an alias of %exe -d.

@fperez
Owner

I'm -1 on %exe: I think %run is fairly close to common language usage and also has a long tradition in IPython, so it should stay as-is, but for timing/profiling/debugging of chunks of code, especially in the notebook, I much prefer clear and descriptive terms like %time or %debug to having notebook cells tagged with %exe -t, which is fairly cryptic.

Keep in mind that %run is mostly used at the terminal clients (including qt console), so it's more transient. But tagging cells for timing or profiling is more likely to be persistent, so I prefer to have self-explanatory terms used there.

ps - note that this PR needs a rebase. @tkf, if you can do that and the minor cleanups indicated above, I think we're close to merge point.

@tkf

cells tagged with %exe -t, which is fairly cryptic

It is easy to add long options, like %exe --time. Also as I said, it is pretty straight forward to provide %time as an alias of %exe --time.

Keep in mind that %run is mostly used at the terminal clients

Well, I use %run a lot in notebook and I don't understand why it should not be used or discouraged in notebook. Yes, I know that explicit is good but in notebook you need to balance explicitness and conciseness. And I think IPython magic as the system to balance the two.

I am not focusing on naming issue. I agree that %time looks better than %exe -t. I am just pointing out that you loose ability to combine command if you go %debug/%time/... way.

@fperez
Owner

You don't lose the ability to combine them, since it's perfectly reasonable to 'stack' magics (we do it all the time with %px):

%time
%profile

# ... code

I think that's fairly clear...

@tkf

Current line magic does not work in that way, right?

%prun
os.path.join('a', 'b')

and

%prun os.path.join('a', 'b')

are not equivalent, I suppose.

%%time
%%prun
os.path.join('a', 'b')

does not work either.

If what you mean is to make one of the above work, that makes sense. I like the approach of stacking cell magics.

@fperez
Owner

Yes, sorry, that's what I meant. They can all be updated to work as nested cell magics.

And btw, if you really want to have a single entry point for running blocks, rather than creating a new magic that needs to be taught, we can simply extend %run to also have cell magic behavior. It would then be:

%run [options] filename

as a line magic, and

%%run [options]
#...
# block of code
#...

as a cell one. That would reuse 100% of the %run implementation, extending it from files to blocks, without having to add a new command.

I actually think that's the cleanest solution: stackable individual magics for %time, %profile and %debug, and extending %run to work as a cell magic on blocks.

And all of these things should be reusing a few generic non-magic objects/functions, with the magics themselves being very thin syntax layers on top of these components.

@fperez
Owner

Oh, and btw, this one needs a rebase...

@tkf

I thought I'd wait until the discussion is settled, to avoid possibility that we decide to go other direction.

I also noticed that individual line magic has advantage over single line magic in other direction: You can avoid parsing options. %exe a -c is interpreted as code a and option -c rather than a - c. (This may be more important for %time and %timeit. If you want merge these line magics, you need to face this parsing problem.)

So, I can go ahead and implement %debug here, right? Then later we can merge it into %run when other magics like %profile and %time are ready.

@tkf

I am not sure if the discussion is settled but I guessed this patch will be pulled by 90% so I just rebased it.

@ellisonbg
Owner

@fperez can you look at this PR to see if all of the comment have been addressed?

@fperez
Owner

@tkf, I'm sorry for the slow progress, my fault!!

I'd like to see a deeper cleanup/refactoring of our execution magics along the lines of the discussion above, but I think this particular PR is already progress in the right direction, and there's no point in holding it up further.

Thanks @tkf for the good work! If you're up for digging further into execution.py to continue post-1.0, by all means do so :)

Merging now.

@fperez fperez merged commit e1ef88f into ipython:master

1 check passed

Details default The Travis CI build passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 26, 2013
  1. @tkf
  2. @tkf

    Document the new usage of %debug

    tkf authored
This page is out of date. Refresh to see the latest.
Showing with 80 additions and 35 deletions.
  1. +80 −35 IPython/core/magics/execution.py
View
115 IPython/core/magics/execution.py
@@ -293,10 +293,31 @@ def pdb(self, parameter_s=''):
self.shell.call_pdb = new_pdb
print 'Automatic pdb calling has been turned',on_off(new_pdb)
- @line_magic
- def debug(self, parameter_s=''):
- """Activate the interactive debugger in post-mortem mode.
+ @skip_doctest
+ @magic_arguments.magic_arguments()
+ @magic_arguments.argument('--breakpoint', '-b', metavar='FILE:LINE',
+ help="""
+ Set break point at LINE in FILE.
+ """
+ )
+ @magic_arguments.argument('statement', nargs='*',
+ help="""
+ Code to run in debugger.
+ You can omit this in cell magic mode.
+ """
+ )
+ @line_cell_magic
+ def debug(self, line='', cell=None):
+ """Activate the interactive debugger.
+
+ This magic command support two ways of activating debugger.
+ One is to activate debugger before executing code. This way, you
+ can set a break point, to step through the code from the point.
+ You can use this mode by giving statements to execute and optionally
+ a breakpoint.
+ The other one is to activate debugger in post-mortem mode. You can
+ activate this mode simply running %debug without any argument.
If an exception has just occurred, this lets you inspect its stack
frames interactively. Note that this will always work only on the last
traceback that occurred, so you must call this quickly after an
@@ -306,8 +327,27 @@ def debug(self, parameter_s=''):
If you want IPython to automatically do this on every exception, see
the %pdb magic for more details.
"""
+ args = magic_arguments.parse_argstring(self.debug, line)
+
+ if not (args.breakpoint or args.statement or cell):
+ self._debug_post_mortem()
+ else:
+ code = "\n".join(args.statement)
+ if cell:
+ code += "\n" + cell
+ self._debug_exec(code, args.breakpoint)
+
+ def _debug_post_mortem(self):
self.shell.debugger(force=True)
+ def _debug_exec(self, code, breakpoint):
+ if breakpoint:
+ (filename, bp_line) = breakpoint.split(':', 1)
+ bp_line = int(bp_line)
+ else:
+ (filename, bp_line) = (None, None)
+ self._run_with_debugger(code, self.shell.user_ns, filename, bp_line)
+
@line_magic
def tb(self, s):
"""Print the last traceback with the currently active exception mode.
@@ -560,8 +600,10 @@ def run(self, parameter_s='', runner=None,
stats = self._run_with_profiler(code, opts, code_ns)
else:
if 'd' in opts:
+ bp_file, bp_line = parse_breakpoint(
+ opts.get('b', ['1'])[0], filename)
self._run_with_debugger(
- code, code_ns, opts.get('b', ['1'])[0], filename)
+ code, code_ns, filename, bp_line, bp_file)
else:
if 'm' in opts:
def run():
@@ -629,7 +671,8 @@ def run():
return stats
- def _run_with_debugger(self, code, code_ns, break_point, filename):
+ def _run_with_debugger(self, code, code_ns, filename=None,
+ bp_line=None, bp_file=None):
"""
Run `code` in debugger with a break point.
@@ -639,19 +682,18 @@ def _run_with_debugger(self, code, code_ns, break_point, filename):
Code to execute.
code_ns : dict
A namespace in which `code` is executed.
- break_point : str
- Line number in the file specified by `filename` argument
- or a string in the format ``file:line``. In the latter
- case, `filename` is ignored.
- See also :func:`.parse_breakpoint`.
filename : str
+ `code` is ran as if it is in `filename`.
+ bp_line : int, optional
+ Line number of the break point.
+ bp_file : str, optional
Path to the file in which break point is specified.
+ `filename` is used if not given.
Raises
------
UsageError
- If no meaningful break point is given by `break_point` and
- `filename`.
+ If the break point given by `bp_line` is not valid.
"""
deb = debugger.Pdb(self.shell.colors)
@@ -660,34 +702,37 @@ def _run_with_debugger(self, code, code_ns, break_point, filename):
bdb.Breakpoint.next = 1
bdb.Breakpoint.bplist = {}
bdb.Breakpoint.bpbynumber = [None]
- # Set an initial breakpoint to stop execution
- maxtries = 10
- bp_file, bp_line = parse_breakpoint(break_point, filename)
- checkline = deb.checkline(bp_file, bp_line)
- if not checkline:
- for bp in range(bp_line + 1, bp_line + maxtries + 1):
- if deb.checkline(bp_file, bp):
- break
- else:
- msg = ("\nI failed to find a valid line to set "
- "a breakpoint\n"
- "after trying up to line: %s.\n"
- "Please set a valid breakpoint manually "
- "with the -b option." % bp)
- raise UsageError(msg)
- # if we find a good linenumber, set the breakpoint
- deb.do_break('%s:%s' % (bp_file, bp_line))
-
- # Mimic Pdb._runscript(...)
- deb._wait_for_mainpyfile = True
- deb.mainpyfile = deb.canonic(filename)
+ if bp_line is not None:
+ # Set an initial breakpoint to stop execution
+ maxtries = 10
+ bp_file = bp_file or filename
+ checkline = deb.checkline(bp_file, bp_line)
+ if not checkline:
+ for bp in range(bp_line + 1, bp_line + maxtries + 1):
+ if deb.checkline(bp_file, bp):
+ break
+ else:
+ msg = ("\nI failed to find a valid line to set "
+ "a breakpoint\n"
+ "after trying up to line: %s.\n"
+ "Please set a valid breakpoint manually "
+ "with the -b option." % bp)
+ raise UsageError(msg)
+ # if we find a good linenumber, set the breakpoint
+ deb.do_break('%s:%s' % (bp_file, bp_line))
+
+ if filename:
+ # Mimic Pdb._runscript(...)
+ deb._wait_for_mainpyfile = True
+ deb.mainpyfile = deb.canonic(filename)
# Start file run
print "NOTE: Enter 'c' at the",
print "%s prompt to start your script." % deb.prompt
try:
- #save filename so it can be used by methods on the deb object
- deb._exec_filename = filename
+ if filename:
+ # save filename so it can be used by methods on the deb object
+ deb._exec_filename = filename
deb.run(code, code_ns)
except:
Something went wrong with that request. Please try again.