Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Shlex unicode #1116

Closed
wants to merge 3 commits into from

3 participants

@jstenar
Collaborator

Fix for issue #1115

@takluyver do you think this is the correct way to convert back to unicode?

J�rgen Stena... added some commits
J�rgen Stenarson Fixing #1112 removing failing asserts for test_carriage_return and te…
…st_beep.

The failing asserts were checking the count property of the Action
tuple objects there is no count field in these Action tuples so
the count method of the regular tuple class is resolved.
8c6d953
J�rgen Stenarson Fixing shlex_split to return unicode on py2.x 3fcb420
@takluyver
Owner

If we know that the output will always be in the native str type, we can use py3compat.str_to_unicode to do the conversion without requiring an if statement. Or have a look at the arg_split function @rkern mentioned on #1115.

J�rgen Stenarson Replaced shlex_split with arg_split from _process_common.
shlex_split was removed since it was a unicode unsafe version of
arg_split. Tests were added to test magic_run_completer.
52664ef
@takluyver
Owner

@jstenar Unfortunately it looks like there's a merge conflict now. It's probably trivial, but if you've got time, could you rebase it.

@jstenar
Collaborator
@minrk
Owner

The conflict is just that the first commit has already been merged as a different PR, but GitHub can't tell. So, @takluyver if this looks fine to you, I will go ahead and merge with the necessary adjustments.

@takluyver
Owner

The shlex_split function we're removing here handles exceptions with this trick involving removing the last character of the string. Have we worked out what that was achieving, and whether it needs to be preserved or replaced?

@minrk
Owner

Ah, that's exactly what my PR #1130 does - change the behavior from raising, to just keeping the last blob.

So if that fix was ever necessary (I presume it was), then this doesn't work unless we also merge 1130.

@takluyver
Owner

OK, that makes sense. And when that's merged, we should test this and make sure it gets the completions right in whatever situation it's handling.

@minrk minrk referenced this pull request from a commit in minrk/ipython
@minrk minrk add strict flag to arg_split, to optionally ignore shlex parse errors
Sometimes we pass things that aren't really command-line args to arg_split, e.g:

    %timeit python_code(" ")

This commit adds a `strict` flag, which defaults to the same raising behavior
as before.

Currently magic_timeit is the *only* place, we use strict=False, but it should
also be done in completions (PR #1116).

closes #1109
4f1e79b
@minrk
Owner

I've now changed my code in #1130 to be more conservative (it takes a strict flag, which defaults to True, which is identical to the old behavior). After merging #1130, this code can use arg_split(s, strict=False), to preserve the error-ignoring behavior in shlex_split.

@minrk
Owner

@takluyver - I actually don't see that we are getting any completions in the cases that this handles, even on master. It only serves to suppress the shlex ValueError. We don't complete "foo.py" from "fo, which I would consider a separate issue.

@takluyver
Owner
@minrk
Owner

Weird - it completes fine on Linux, but not OSX, so who knows what's up with that. In any case, it shouldn't be relevant to this PR.

I can merge #1130 and this (adding strict=False to match old behavior) if they look good to you, @takluyver.

@takluyver
Owner

Assuming the "fo tab completion on Linux still works, go for it.

@minrk
Owner

It does, tested both before and after the change.

Thanks! I'll go ahead and merge both.

@minrk minrk referenced this pull request from a commit in minrk/ipython
@minrk minrk Merge shlex PRs (#1130, #1116)
* arg_split now takes optional strict flag, to ignore ValueErrors in
  shlex parsing
* %timeit uses strict=False, to avoid errors parsing python code
* %run completer uses arg_split(strict=False) for its unicode behavior, instead
  of custom shlex derivative, which is now redundant.

closes #1109
closes #1115
closes #1116
closes #1130
790cb14
@minrk minrk closed this pull request from a commit
@minrk minrk Merge shlex PRs (#1130, #1116)
* arg_split now takes optional strict flag, to ignore ValueErrors in
  shlex parsing
* %timeit uses strict=False, to avoid errors parsing python code
* %run completer uses arg_split(strict=False) for its unicode behavior, instead
  of custom shlex derivative, which is now redundant.

closes #1109
closes #1115
closes #1116
closes #1130
790cb14
@minrk minrk closed this in 790cb14
@ellisonbg ellisonbg referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
@ellisonbg ellisonbg referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
@mattvonrocketstein mattvonrocketstein referenced this pull request from a commit in mattvonrocketstein/ipython
@minrk minrk add strict flag to arg_split, to optionally ignore shlex parse errors
Sometimes we pass things that aren't really command-line args to arg_split, e.g:

    %timeit python_code(" ")

This commit adds a `strict` flag, which defaults to the same raising behavior
as before.

Currently magic_timeit is the *only* place, we use strict=False, but it should
also be done in completions (PR #1116).

closes #1109
52b3af3
@mattvonrocketstein mattvonrocketstein referenced this pull request from a commit in mattvonrocketstein/ipython
@minrk minrk Merge shlex PRs (#1130, #1116)
* arg_split now takes optional strict flag, to ignore ValueErrors in
  shlex parsing
* %timeit uses strict=False, to avoid errors parsing python code
* %run completer uses arg_split(strict=False) for its unicode behavior, instead
  of custom shlex derivative, which is now redundant.

closes #1109
closes #1115
closes #1116
closes #1130
5799471
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 7, 2011
  1. Fixing #1112 removing failing asserts for test_carriage_return and te…

    J�rgen Stenarson authored
    …st_beep.
    
    The failing asserts were checking the count property of the Action
    tuple objects there is no count field in these Action tuples so
    the count method of the regular tuple class is resolved.
  2. Fixing shlex_split to return unicode on py2.x

    J�rgen Stenarson authored
Commits on Dec 8, 2011
  1. Replaced shlex_split with arg_split from _process_common.

    J�rgen Stenarson authored
    shlex_split was removed since it was a unicode unsafe version of
    arg_split. Tests were added to test magic_run_completer.
This page is out of date. Refresh to see the latest.
View
33 IPython/core/completerlib.py
@@ -20,7 +20,6 @@
import inspect
import os
import re
-import shlex
import sys
# Third-party imports
@@ -31,6 +30,7 @@
from IPython.core.completer import expand_user, compress_user
from IPython.core.error import TryNext
from IPython.utils import py3compat
+from IPython.utils._process_common import arg_split
# FIXME: this should be pulled in with the right call via the component system
from IPython.core.ipapi import get as get_ipython
@@ -56,35 +56,6 @@
# Local utilities
#-----------------------------------------------------------------------------
-def shlex_split(x):
- """Helper function to split lines into segments.
- """
- # shlex.split raises an exception if there is a syntax error in sh syntax
- # for example if no closing " is found. This function keeps dropping the
- # last character of the line until shlex.split does not raise
- # an exception. It adds end of the line to the result of shlex.split
- #
- # Example:
- # %run "c:/python -> ['%run','"c:/python']
-
- # shlex.split has unicode bugs in Python 2, so encode first to str
- if not py3compat.PY3:
- x = py3compat.cast_bytes(x)
-
- endofline = []
- while x != '':
- try:
- comps = shlex.split(x)
- if len(endofline) >= 1:
- comps.append(''.join(endofline))
- return comps
-
- except ValueError:
- endofline = [x[-1:]]+endofline
- x = x[:-1]
-
- return [''.join(endofline)]
-
def module_list(path):
"""
Return the list containing the names of the modules available in the given
@@ -265,7 +236,7 @@ def module_completer(self,event):
def magic_run_completer(self, event):
"""Complete files that end in .py or .ipy for the %run command.
"""
- comps = shlex_split(event.line)
+ comps = arg_split(event.line)
relpath = (len(comps) > 1 and comps[-1] or '').strip("'\"")
#print("\nev=", event) # dbg
View
61 IPython/core/tests/test_completerlib.py
@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+"""Tests for completerlib.
+
+"""
+from __future__ import absolute_import
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+import os
+import shutil
+import sys
+import tempfile
+import unittest
+from os.path import join
+
+import nose.tools as nt
+from nose import SkipTest
+
+from IPython.core.completerlib import magic_run_completer
+from IPython.testing import decorators as dec
+from IPython.testing import tools as tt
+from IPython.utils import py3compat
+
+
+class MockEvent(object):
+ def __init__(self, line):
+ self.line = line
+
+#-----------------------------------------------------------------------------
+# Test functions begin
+#-----------------------------------------------------------------------------
+class Test_magic_run_completer(unittest.TestCase):
+ def setUp(self):
+ self.BASETESTDIR = tempfile.mkdtemp()
+ for fil in [u"aaå.py", u"a.py", u"b.py"]:
+ with open(join(self.BASETESTDIR, fil), "w") as sfile:
+ sfile.write("pass\n")
+ self.oldpath = os.getcwdu()
+ os.chdir(self.BASETESTDIR)
+
+ def tearDown(self):
+ os.chdir(self.oldpath)
+ shutil.rmtree(self.BASETESTDIR)
+
+ def test_1(self):
+ """Test magic_run_completer, should match two alterntives
+ """
+ event = MockEvent(u"%run a")
+ mockself = None
+ match = magic_run_completer(mockself, event)
+ self.assertEqual(match, [u"a.py", u"aaå.py",])
+
+ def test_2(self):
+ """Test magic_run_completer, should match one alterntive
+ """
+ event = MockEvent(u"%run aa")
+ mockself = None
+ match = magic_run_completer(mockself, event)
+ self.assertEqual(match, [u"aaå.py",])
View
2  IPython/frontend/qt/console/tests/test_ansi_code_processor.py
@@ -110,7 +110,6 @@ def test_carriage_return(self):
self.assertEquals(len(self.processor.actions), 1)
action = self.processor.actions[0]
self.assertEquals(action.action, 'carriage-return')
- self.assertEquals(action.count, 1)
def test_beep(self):
""" Are beep characters processed correctly?
@@ -120,7 +119,6 @@ def test_beep(self):
self.assertEquals(len(self.processor.actions), 1)
action = self.processor.actions[0]
self.assertEquals(action.action, 'beep')
- self.assertEquals(action.count, 1)
if __name__ == '__main__':
Something went wrong with that request. Please try again.