Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Fix paste/cpaste bug and refactor/cleanup that code a lot. #1007

Merged
merged 3 commits into from

3 participants

Fernando Perez Min RK Stefan van der Walt
Fernando Perez
Owner

In fixing a pasting bug (mishandling of whitespace when input had
prompts) it became clear the pasting code hadn't been updated
when the new prefiltering machinery was added. Furthermore, the
pasting magics are only for the terminal, but the code was in the
base classes.

This refactors and simplifies the pasting code, moving it to the
terminal shell only, and removing unnecessary methods from the
main class (using small utility functions instead).

The tests were simplified because the previous regexp supported some
odd edge cases that are not valid in the normal prefiltering code. We
want to have a single location and set of rules for input
prefiltering, so I changed some of the test cases to be consistent
with what prefilter allows.

Fernando Perez
Owner

@stefanv, could you have a quick look and see if this fixes correctly the cases you'd seen?

Min RK
Owner

This is looking pretty good to me. One general question - store_or_execute() and rerun_pasted() are functions whose first argument is always the object that calls them, making them perfectly identical to instance methods in terms of implementation. What benefit does this approach have over them just being methods?

In this way, every single method of an object could be made a function. For future reference, which ones should be functions, and which should be methods?

Fernando Perez
Owner

They were actually methods, and you are totally correct that by this token we could completely strip the object down :) I don't have a hard rule yet and I'm just feeling my way through this, but it seems to me that our main interactive shell objects have become gigantic and unwieldy, so I was trying to reduce their method footprint and API a little bit.

Since these were little pieces of functionality only used by the terminal application, it seemed like a cleaner solution to just refactor them out into separate standalone functions. That would also make it much easier to test them in isolation with mock objects that only provide one or two methods, something harder to do when they are methods of the main thing.

I'm not trying to go and rip out everything from the main shell object, but these didn't really feel like 'core' functionality (in fact they were _private in the previous implementation). So it's really mostly an instinct design call, and one that I'd love feedback on. Do you agree in general with trying to (when reasonably easy and non-disruptive as was the case here) simplify a little bit the official public API of our big Shell object, and with this approach? I mostly want us to find a good design balance for this that will make the overall use and maintenance of the code easier...

Stefan van der Walt

E-mail pasting no longer works, e.g.

>> def foo(x):
>>     return x

Also, some valid Python causes confusion:

>>> def foo(x): return 1 \
...     > 1
Min RK
Owner

@fperez - I agree that we should move in a direction of more functions and fewer methods, but these seem like they really are methods, since copying/pasting them into the Class scope would require no internal changes at all (aside from changing invocation back from f(self to self.f(). They are still highly stateful, depending on and affecting the active state of the shell instance and various members. I'm not sure that anything that calls shell.run_cell() is better off being split off into a function.

I don't feel strongly and I don't know where the line should be drawn, but my first inclination is that if you have to provide the shell instance itself, rather than some of its members, it's probably a method.

Fernando Perez
Owner

Just rebased to sort out a conflict and force-pushed. Too tired now to work any more on this, will get back to it in a couple of days.

fperez added some commits
Fernando Perez fperez Fix paste/cpaste bug and refactor/cleanup that code a lot.
In fixing a pasting bug (mishandling of whitespace when input had
prompts) it became clear the pasting code hadn't been updated
when the new prefiltering machinery was added.  Furthermore, the
pasting magics are only for the terminal, but the code was in the
base classes.

This refactors and simplifies the pasting code, moving it to the
terminal shell only, and removing unnecessary methods from the
main class (using small utility functions instead).

The tests were simplified because the previous regexp supported some
odd edge cases that are not valid in the normal prefiltering code.  We
want to have a single location and set of rules for input
prefiltering, so I changed some of the test cases to be consistent
with what prefilter allows.
8bb887c
Fernando Perez fperez Move tests for magics that are terminal-specific to their own file. 8259953
Fernando Perez fperez Add tests for email quote stripping.
Also, clarify why certain utilities are kept as standalone functions
instead of making them methods.
84830ec
Fernando Perez
Owner

@stefanv, I fixed the email case and added some tests for that kind of format, let me know if this looks OK now.

@minrk, here's how I view the ones in this particular case: while I agree with you that those are basically methods in all but name right now, I hope that soon we'll be able to refactor our magic system into standalone objects. And in this case, those methods are really utilities of the paste/cpaste magics, not of the shell itself. If those magics were in a separate object (say one added by a user in an extension), they would never modify the shell as methods, they'd simply pass it as an argument. So I think that leaving them as standalone functions now is the right thing to do, in anticipation for an eventual refactoring into a new object, a Magic, that would put them together with paste/cpaste.

If you guys agree with this, I'll go ahead and merge. Thanks for the review!

Min RK
Owner

@fperez - I think this is fine. It does seem a bit odd to pull methods out into functions, with the ultimate goal that they become methods of a different object in the future. I do appreciate the desire to pull things off of the Shell, so it makes sense.

If @stefanv's says his cases work, go for it.

Fernando Perez
Owner

Thanks, @minrk. I did add a comment around them to try and clarify the intent of the somewhat odd refactoring. I hope to find the time soon to draft my ideas on the magics and post them on the dev list for feedback.

@stefanv, note that I fixed email pasting, but not that very odd corner case you found. The logic for handling continuation lines gets pretty complicated if we want to mix in automatic removal of special characters, so I think it's OK if the pasting utilities don't cover them. I don't want them to become overly complex. As long as we handle that code in the normal input prompt, I think that's an acceptable compromise. What do you think?

Stefan van der Walt

@fperez Thanks, the e-mail case is the most common one I deal with. Dealing with all the corner cases cleanly may be very hard, so I'm be happy with the current behaviour.

Fernando Perez
Owner

Great, merging and closing. Thanks for the feedback!

Fernando Perez fperez merged commit 780b7c5 into from
Fernando Perez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 26, 2011
  1. Fernando Perez

    Fix paste/cpaste bug and refactor/cleanup that code a lot.

    fperez authored
    In fixing a pasting bug (mishandling of whitespace when input had
    prompts) it became clear the pasting code hadn't been updated
    when the new prefiltering machinery was added.  Furthermore, the
    pasting magics are only for the terminal, but the code was in the
    base classes.
    
    This refactors and simplifies the pasting code, moving it to the
    terminal shell only, and removing unnecessary methods from the
    main class (using small utility functions instead).
    
    The tests were simplified because the previous regexp supported some
    odd edge cases that are not valid in the normal prefiltering code.  We
    want to have a single location and set of rules for input
    prefiltering, so I changed some of the test cases to be consistent
    with what prefilter allows.
  2. Fernando Perez
  3. Fernando Perez

    Add tests for email quote stripping.

    fperez authored
    Also, clarify why certain utilities are kept as standalone functions
    instead of making them methods.
This page is out of date. Refresh to see the latest.
60 IPython/core/magic.py
View
@@ -25,7 +25,6 @@
import shutil
import re
import time
-import textwrap
from StringIO import StringIO
from getopt import getopt,GetoptError
from pprint import pformat
@@ -3203,65 +3202,6 @@ def magic_pycat(self, parameter_s=''):
page.page(self.shell.pycolorize(cont))
- def _rerun_pasted(self):
- """ Rerun a previously pasted command.
- """
- b = self.user_ns.get('pasted_block', None)
- if b is None:
- raise UsageError('No previous pasted block available')
- print "Re-executing '%s...' (%d chars)"% (b.split('\n',1)[0], len(b))
- exec b in self.user_ns
-
- def _get_pasted_lines(self, sentinel):
- """ Yield pasted lines until the user enters the given sentinel value.
- """
- from IPython.core import interactiveshell
- print "Pasting code; enter '%s' alone on the line to stop." % sentinel
- while True:
- try:
- l = self.shell.raw_input_original(':')
- if l == sentinel:
- return
- else:
- yield l
- except EOFError:
- print '<EOF>'
- return
-
- def _strip_pasted_lines_for_code(self, raw_lines):
- """ Strip non-code parts of a sequence of lines to return a block of
- code.
- """
- # Regular expressions that declare text we strip from the input:
- strip_re = [r'^\s*In \[\d+\]:', # IPython input prompt
- r'^\s*(\s?>)+', # Python input prompt
- r'^\s*\.{3,}', # Continuation prompts
- r'^\++',
- ]
-
- strip_from_start = map(re.compile,strip_re)
-
- lines = []
- for l in raw_lines:
- for pat in strip_from_start:
- l = pat.sub('',l)
- lines.append(l)
-
- block = "\n".join(lines) + '\n'
- #print "block:\n",block
- return block
-
- def _execute_block(self, block, par):
- """ Execute a block, or store it in a variable, per the user's request.
- """
- if not par:
- b = textwrap.dedent(block)
- self.user_ns['pasted_block'] = b
- self.run_cell(b)
- else:
- self.user_ns[par] = SList(block.splitlines())
- print "Block assigned to '%s'" % par
-
def magic_quickref(self,arg):
""" Show a quick reference sheet """
import IPython.core.usage
122 IPython/core/tests/test_magic.py
View
@@ -9,14 +9,9 @@
#-----------------------------------------------------------------------------
import os
-import sys
-import tempfile
-import types
-from StringIO import StringIO
import nose.tools as nt
-from IPython.utils.path import get_long_path_name
from IPython.testing import decorators as dec
from IPython.testing import tools as tt
from IPython.utils import py3compat
@@ -24,6 +19,7 @@
#-----------------------------------------------------------------------------
# Test functions begin
#-----------------------------------------------------------------------------
+
def test_rehashx():
# clear up everything
_ip = get_ipython()
@@ -202,67 +198,6 @@ def test_numpy_clear_array_undec():
yield (nt.assert_false, 'a' in _ip.user_ns)
-# Multiple tests for clipboard pasting
-@dec.parametric
-def test_paste():
- _ip = get_ipython()
- def paste(txt, flags='-q'):
- """Paste input text, by default in quiet mode"""
- hooks.clipboard_get = lambda : txt
- _ip.magic('paste '+flags)
-
- # Inject fake clipboard hook but save original so we can restore it later
- hooks = _ip.hooks
- user_ns = _ip.user_ns
- original_clip = hooks.clipboard_get
-
- try:
- # Run tests with fake clipboard function
- user_ns.pop('x', None)
- paste('x=1')
- yield nt.assert_equal(user_ns['x'], 1)
-
- user_ns.pop('x', None)
- paste('>>> x=2')
- yield nt.assert_equal(user_ns['x'], 2)
-
- paste("""
- >>> x = [1,2,3]
- >>> y = []
- >>> for i in x:
- ... y.append(i**2)
- ...
- """)
- yield nt.assert_equal(user_ns['x'], [1,2,3])
- yield nt.assert_equal(user_ns['y'], [1,4,9])
-
- # Now, test that paste -r works
- user_ns.pop('x', None)
- yield nt.assert_false('x' in user_ns)
- _ip.magic('paste -r')
- yield nt.assert_equal(user_ns['x'], [1,2,3])
-
- # Also test paste echoing, by temporarily faking the writer
- w = StringIO()
- writer = _ip.write
- _ip.write = w.write
- code = """
- a = 100
- b = 200"""
- try:
- paste(code,'')
- out = w.getvalue()
- finally:
- _ip.write = writer
- yield nt.assert_equal(user_ns['a'], 100)
- yield nt.assert_equal(user_ns['b'], 200)
- yield nt.assert_equal(out, code+"\n## -- End pasted text --\n")
-
- finally:
- # Restore original hook
- hooks.clipboard_get = original_clip
-
-
def test_time():
_ip.magic('time None')
@@ -317,61 +252,6 @@ def test_dirops():
os.chdir(startdir)
-def check_cpaste(code, should_fail=False):
- """Execute code via 'cpaste' and ensure it was executed, unless
- should_fail is set.
- """
- _ip.user_ns['code_ran'] = False
-
- src = StringIO()
- if not hasattr(src, 'encoding'):
- # IPython expects stdin to have an encoding attribute
- src.encoding = None
- src.write('\n')
- src.write(code)
- src.write('\n--\n')
- src.seek(0)
-
- stdin_save = sys.stdin
- sys.stdin = src
-
- try:
- context = tt.AssertPrints if should_fail else tt.AssertNotPrints
- with context("Traceback (most recent call last)"):
- _ip.magic('cpaste')
-
- if not should_fail:
- assert _ip.user_ns['code_ran']
- finally:
- sys.stdin = stdin_save
-
-
-def test_cpaste():
- """Test cpaste magic"""
-
- def run():
- """Marker function: sets a flag when executed.
- """
- _ip.user_ns['code_ran'] = True
- return 'run' # return string so '+ run()' doesn't result in success
-
- tests = {'pass': ["> > > run()",
- ">>> > run()",
- "+++ run()",
- "++ run()",
- " >>> run()"],
-
- 'fail': ["+ + run()",
- " ++ run()"]}
-
- _ip.user_ns['run'] = run
-
- for code in tests['pass']:
- check_cpaste(code)
-
- for code in tests['fail']:
- check_cpaste(code, should_fail=True)
-
def test_xmode():
# Calling xmode three times should be a no-op
xmode = _ip.InteractiveTB.mode
164 IPython/core/tests/test_magic_terminal.py
View
@@ -0,0 +1,164 @@
+"""Tests for various magic functions specific to the terminal frontend.
+
+Needs to be run by nose (to make ipython session available).
+"""
+from __future__ import absolute_import
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+import sys
+from StringIO import StringIO
+
+import nose.tools as nt
+
+from IPython.testing import decorators as dec
+from IPython.testing import tools as tt
+
+#-----------------------------------------------------------------------------
+# Test functions begin
+#-----------------------------------------------------------------------------
+
+def check_cpaste(code, should_fail=False):
+ """Execute code via 'cpaste' and ensure it was executed, unless
+ should_fail is set.
+ """
+ _ip.user_ns['code_ran'] = False
+
+ src = StringIO()
+ if not hasattr(src, 'encoding'):
+ # IPython expects stdin to have an encoding attribute
+ src.encoding = None
+ src.write('\n')
+ src.write(code)
+ src.write('\n--\n')
+ src.seek(0)
+
+ stdin_save = sys.stdin
+ sys.stdin = src
+
+ try:
+ context = tt.AssertPrints if should_fail else tt.AssertNotPrints
+ with context("Traceback (most recent call last)"):
+ _ip.magic('cpaste')
+
+ if not should_fail:
+ assert _ip.user_ns['code_ran']
+ finally:
+ sys.stdin = stdin_save
+
+
+def test_cpaste():
+ """Test cpaste magic"""
+
+ def run():
+ """Marker function: sets a flag when executed.
+ """
+ _ip.user_ns['code_ran'] = True
+ return 'run' # return string so '+ run()' doesn't result in success
+
+ tests = {'pass': ["run()",
+ "In [1]: run()",
+ "In [1]: if 1:\n ...: run()",
+ "> > > run()",
+ ">>> run()",
+ " >>> run()",
+ ],
+
+ 'fail': ["1 + run()",
+ "++ run()"]}
+
+ _ip.user_ns['run'] = run
+
+ for code in tests['pass']:
+ check_cpaste(code)
+
+ for code in tests['fail']:
+ check_cpaste(code, should_fail=True)
+
+
+# Multiple tests for clipboard pasting
+def test_paste():
+ _ip = get_ipython()
+
+ def paste(txt, flags='-q'):
+ """Paste input text, by default in quiet mode"""
+ hooks.clipboard_get = lambda : txt
+ _ip.magic('paste '+flags)
+
+ # Inject fake clipboard hook but save original so we can restore it later
+ hooks = _ip.hooks
+ user_ns = _ip.user_ns
+ original_clip = hooks.clipboard_get
+
+ try:
+ # Run tests with fake clipboard function
+ user_ns.pop('x', None)
+ paste('x=1')
+ nt.assert_equal(user_ns['x'], 1)
+
+ user_ns.pop('x', None)
+ paste('>>> x=2')
+ nt.assert_equal(user_ns['x'], 2)
+
+ paste("""
+ >>> x = [1,2,3]
+ >>> y = []
+ >>> for i in x:
+ ... y.append(i**2)
+ ...
+ """)
+ nt.assert_equal(user_ns['x'], [1,2,3])
+ nt.assert_equal(user_ns['y'], [1,4,9])
+
+ # Now, test that paste -r works
+ user_ns.pop('x', None)
+ nt.assert_false('x' in user_ns)
+ _ip.magic('paste -r')
+ nt.assert_equal(user_ns['x'], [1,2,3])
+
+ # Test pasting of email-quoted contents
+ paste("""
+ >> def foo(x):
+ >> return x + 1
+ >> x = foo(1.1)
+ """)
+ nt.assert_equal(user_ns['x'], 2.1)
+
+ # Email again; some programs add a space also at each quoting level
+ paste("""
+ > > def foo(x):
+ > > return x + 1
+ > > x = foo(2.1)
+ """)
+ nt.assert_equal(user_ns['x'], 3.1)
+
+ # Email quoting of interactive input
+ paste("""
+ >> >>> def f(x):
+ >> ... return x+1
+ >> ...
+ >> >>> x = f(2.5)
+ """)
+ nt.assert_equal(user_ns['x'], 3.5)
+
+ # Also test paste echoing, by temporarily faking the writer
+ w = StringIO()
+ writer = _ip.write
+ _ip.write = w.write
+ code = """
+ a = 100
+ b = 200"""
+ try:
+ paste(code,'')
+ out = w.getvalue()
+ finally:
+ _ip.write = writer
+ nt.assert_equal(user_ns['a'], 100)
+ nt.assert_equal(user_ns['b'], 200)
+ nt.assert_equal(out, code+"\n## -- End pasted text --\n")
+
+ finally:
+ # Restore original hook
+ hooks.clipboard_get = original_clip
109 IPython/frontend/terminal/interactiveshell.py
View
@@ -19,6 +19,7 @@
import os
import re
import sys
+import textwrap
try:
from contextlib import nested
@@ -35,7 +36,7 @@
from IPython.utils.terminal import toggle_set_term_title, set_term_title
from IPython.utils.process import abbrev_cwd
from IPython.utils.warn import warn, error
-from IPython.utils.text import num_ini_spaces
+from IPython.utils.text import num_ini_spaces, SList
from IPython.utils.traitlets import Integer, CBool, Unicode
#-----------------------------------------------------------------------------
@@ -52,6 +53,73 @@ def get_default_editor():
ed = 'notepad' # same in Windows!
return ed
+
+def get_pasted_lines(sentinel, input=raw_input):
+ """ Yield pasted lines until the user enters the given sentinel value.
+ """
+ print "Pasting code; enter '%s' alone on the line to stop or use Ctrl-D." \
+ % sentinel
+ while True:
+ try:
+ l = input(':')
+ if l == sentinel:
+ return
+ else:
+ yield l
+ except EOFError:
+ print '<EOF>'
+ return
+
+
+def strip_email_quotes(raw_lines):
+ """ Strip email quotation marks at the beginning of each line.
+
+ We don't do any more input transofrmations here because the main shell's
+ prefiltering handles other cases.
+ """
+ lines = [re.sub(r'^\s*(\s?>)+', '', l) for l in raw_lines]
+ return '\n'.join(lines) + '\n'
+
+
+# These two functions are needed by the %paste/%cpaste magics. In practice
+# they are basically methods (they take the shell as their first argument), but
+# we leave them as standalone functions because eventually the magics
+# themselves will become separate objects altogether. At that point, the
+# magics will have access to the shell object, and these functions can be made
+# methods of the magic object, but not of the shell.
+
+def store_or_execute(shell, block, name):
+ """ Execute a block, or store it in a variable, per the user's request.
+ """
+ # Dedent and prefilter so what we store matches what is executed by
+ # run_cell.
+ b = shell.prefilter(textwrap.dedent(block))
+
+ if name:
+ # If storing it for further editing, run the prefilter on it
+ shell.user_ns[name] = SList(b.splitlines())
+ print "Block assigned to '%s'" % name
+ else:
+ shell.user_ns['pasted_block'] = b
+ shell.run_cell(b)
+
+
+def rerun_pasted(shell, name='pasted_block'):
+ """ Rerun a previously pasted command.
+ """
+ b = shell.user_ns.get(name)
+
+ # Sanity checks
+ if b is None:
+ raise UsageError('No previous pasted block available')
+ if not isinstance(b, basestring):
+ raise UsageError(
+ "Variable 'pasted_block' is not a string, can't execute")
+
+ print "Re-executing '%s...' (%d chars)"% (b.split('\n',1)[0], len(b))
+ shell.run_cell(b)
+
+
#-----------------------------------------------------------------------------
# Main class
#-----------------------------------------------------------------------------
@@ -523,9 +591,9 @@ def magic_autoindent(self, parameter_s = ''):
def magic_cpaste(self, parameter_s=''):
"""Paste & execute a pre-formatted code block from clipboard.
- You must terminate the block with '--' (two minus-signs) or Ctrl-D alone on the
- line. You can also provide your own sentinel with '%paste -s %%' ('%%'
- is the new sentinel for this operation)
+ You must terminate the block with '--' (two minus-signs) or Ctrl-D
+ alone on the line. You can also provide your own sentinel with '%paste
+ -s %%' ('%%' is the new sentinel for this operation)
The block is dedented prior to execution to enable execution of method
definitions. '>' and '+' characters at the beginning of a line are
@@ -562,18 +630,14 @@ def magic_cpaste(self, parameter_s=''):
Hello world!
"""
- opts,args = self.parse_options(parameter_s,'rs:',mode='string')
- par = args.strip()
- if opts.has_key('r'):
- self._rerun_pasted()
+ opts, name = self.parse_options(parameter_s, 'rs:', mode='string')
+ if 'r' in opts:
+ rerun_pasted(self.shell)
return
- sentinel = opts.get('s','--')
-
- block = self._strip_pasted_lines_for_code(
- self._get_pasted_lines(sentinel))
-
- self._execute_block(block, par)
+ sentinel = opts.get('s', '--')
+ block = strip_email_quotes(get_pasted_lines(sentinel))
+ store_or_execute(self.shell, block, name)
def magic_paste(self, parameter_s=''):
"""Paste & execute a pre-formatted code block from clipboard.
@@ -606,14 +670,13 @@ def magic_paste(self, parameter_s=''):
--------
cpaste: manually paste code into terminal until you mark its end.
"""
- opts,args = self.parse_options(parameter_s,'rq',mode='string')
- par = args.strip()
- if opts.has_key('r'):
- self._rerun_pasted()
+ opts, name = self.parse_options(parameter_s, 'rq', mode='string')
+ if 'r' in opts:
+ rerun_pasted(self.shell)
return
try:
text = self.shell.hooks.clipboard_get()
- block = self._strip_pasted_lines_for_code(text.splitlines())
+ block = strip_email_quotes(text.splitlines())
except TryNext as clipboard_exc:
message = getattr(clipboard_exc, 'args')
if message:
@@ -623,15 +686,16 @@ def magic_paste(self, parameter_s=''):
return
# By default, echo back to terminal unless quiet mode is requested
- if not opts.has_key('q'):
+ if 'q' not in opts:
write = self.shell.write
write(self.shell.pycolorize(block))
if not block.endswith('\n'):
write('\n')
write("## -- End pasted text --\n")
- self._execute_block(block, par)
+ store_or_execute(self.shell, block, name)
+ # Class-level: add a '%cls' magic only on Windows
if sys.platform == 'win32':
def magic_cls(self, s):
"""Clear screen.
@@ -640,7 +704,8 @@ def magic_cls(self, s):
def showindentationerror(self):
super(TerminalInteractiveShell, self).showindentationerror()
- print("If you want to paste code into IPython, try the %paste magic function.")
+ print("If you want to paste code into IPython, try the "
+ "%paste and %cpaste magic functions.")
InteractiveShellABC.register(TerminalInteractiveShell)
Something went wrong with that request. Please try again.