Skip to content
This repository

Shlex unicode #1116

Closed
wants to merge 3 commits into from

3 participants

Jörgen Stenarson Thomas Kluyver Min RK
Jörgen Stenarson
Collaborator

Fix for issue #1115

@takluyver do you think this is the correct way to convert back to unicode?

Jörgen Stena... added some commits December 07, 2011
Fixing #1112 removing failing asserts for test_carriage_return and te…
…st_beep.

The failing asserts were checking the count property of the Action
tuple objects there is no count field in these Action tuples so
the count method of the regular tuple class is resolved.
8c6d953
Fixing shlex_split to return unicode on py2.x 3fcb420
Thomas Kluyver
Collaborator

If we know that the output will always be in the native str type, we can use py3compat.str_to_unicode to do the conversion without requiring an if statement. Or have a look at the arg_split function @rkern mentioned on #1115.

Replaced shlex_split with arg_split from _process_common.
shlex_split was removed since it was a unicode unsafe version of
arg_split. Tests were added to test magic_run_completer.
52664ef
Thomas Kluyver
Collaborator

@jstenar Unfortunately it looks like there's a merge conflict now. It's probably trivial, but if you've got time, could you rebase it.

Jörgen Stenarson
Collaborator
Min RK
Owner

The conflict is just that the first commit has already been merged as a different PR, but GitHub can't tell. So, @takluyver if this looks fine to you, I will go ahead and merge with the necessary adjustments.

Thomas Kluyver
Collaborator

The shlex_split function we're removing here handles exceptions with this trick involving removing the last character of the string. Have we worked out what that was achieving, and whether it needs to be preserved or replaced?

Min RK
Owner

Ah, that's exactly what my PR #1130 does - change the behavior from raising, to just keeping the last blob.

So if that fix was ever necessary (I presume it was), then this doesn't work unless we also merge 1130.

Thomas Kluyver
Collaborator

OK, that makes sense. And when that's merged, we should test this and make sure it gets the completions right in whatever situation it's handling.

Min RK minrk referenced this pull request from a commit in minrk/ipython December 08, 2011
Min RK add strict flag to arg_split, to optionally ignore shlex parse errors
Sometimes we pass things that aren't really command-line args to arg_split, e.g:

    %timeit python_code(" ")

This commit adds a `strict` flag, which defaults to the same raising behavior
as before.

Currently magic_timeit is the *only* place, we use strict=False, but it should
also be done in completions (PR #1116).

closes #1109
4f1e79b
Min RK
Owner

I've now changed my code in #1130 to be more conservative (it takes a strict flag, which defaults to True, which is identical to the old behavior). After merging #1130, this code can use arg_split(s, strict=False), to preserve the error-ignoring behavior in shlex_split.

Min RK
Owner

@takluyver - I actually don't see that we are getting any completions in the cases that this handles, even on master. It only serves to suppress the shlex ValueError. We don't complete "foo.py" from "fo, which I would consider a separate issue.

Thomas Kluyver
Collaborator
Min RK
Owner

Weird - it completes fine on Linux, but not OSX, so who knows what's up with that. In any case, it shouldn't be relevant to this PR.

I can merge #1130 and this (adding strict=False to match old behavior) if they look good to you, @takluyver.

Thomas Kluyver
Collaborator

Assuming the "fo tab completion on Linux still works, go for it.

Min RK
Owner

It does, tested both before and after the change.

Thanks! I'll go ahead and merge both.

Min RK minrk referenced this pull request from a commit in minrk/ipython December 12, 2011
Min RK Merge shlex PRs (#1130, #1116)
* arg_split now takes optional strict flag, to ignore ValueErrors in
  shlex parsing
* %timeit uses strict=False, to avoid errors parsing python code
* %run completer uses arg_split(strict=False) for its unicode behavior, instead
  of custom shlex derivative, which is now redundant.

closes #1109
closes #1115
closes #1116
closes #1130
790cb14
Min RK minrk closed this pull request from a commit December 12, 2011
Min RK Merge shlex PRs (#1130, #1116)
* arg_split now takes optional strict flag, to ignore ValueErrors in
  shlex parsing
* %timeit uses strict=False, to avoid errors parsing python code
* %run completer uses arg_split(strict=False) for its unicode behavior, instead
  of custom shlex derivative, which is now redundant.

closes #1109
closes #1115
closes #1116
closes #1130
790cb14
Min RK minrk closed this in 790cb14 December 12, 2011
Brian E. Granger ellisonbg referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Brian E. Granger ellisonbg referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 3 unique commits by 1 author.

Dec 07, 2011
Fixing #1112 removing failing asserts for test_carriage_return and te…
…st_beep.

The failing asserts were checking the count property of the Action
tuple objects there is no count field in these Action tuples so
the count method of the regular tuple class is resolved.
8c6d953
Fixing shlex_split to return unicode on py2.x 3fcb420
Dec 08, 2011
Replaced shlex_split with arg_split from _process_common.
shlex_split was removed since it was a unicode unsafe version of
arg_split. Tests were added to test magic_run_completer.
52664ef
This page is out of date. Refresh to see the latest.
33  IPython/core/completerlib.py
@@ -20,7 +20,6 @@
20 20
 import inspect
21 21
 import os
22 22
 import re
23  
-import shlex
24 23
 import sys
25 24
 
26 25
 # Third-party imports
@@ -31,6 +30,7 @@
31 30
 from IPython.core.completer import expand_user, compress_user
32 31
 from IPython.core.error import TryNext
33 32
 from IPython.utils import py3compat
  33
+from IPython.utils._process_common import arg_split
34 34
 
35 35
 # FIXME: this should be pulled in with the right call via the component system
36 36
 from IPython.core.ipapi import get as get_ipython
@@ -56,35 +56,6 @@
56 56
 # Local utilities
57 57
 #-----------------------------------------------------------------------------
58 58
 
59  
-def shlex_split(x):
60  
-    """Helper function to split lines into segments.
61  
-    """
62  
-    # shlex.split raises an exception if there is a syntax error in sh syntax
63  
-    # for example if no closing " is found. This function keeps dropping the
64  
-    # last character of the line until shlex.split does not raise
65  
-    # an exception. It adds end of the line to the result of shlex.split
66  
-    #
67  
-    # Example:
68  
-    # %run "c:/python -> ['%run','"c:/python']
69  
-
70  
-    # shlex.split has unicode bugs in Python 2, so encode first to str
71  
-    if not py3compat.PY3:
72  
-        x = py3compat.cast_bytes(x)
73  
-
74  
-    endofline = []
75  
-    while x != '':
76  
-        try:
77  
-            comps = shlex.split(x)
78  
-            if len(endofline) >= 1:
79  
-                comps.append(''.join(endofline))
80  
-            return comps
81  
-
82  
-        except ValueError:
83  
-            endofline = [x[-1:]]+endofline
84  
-            x = x[:-1]
85  
-
86  
-    return [''.join(endofline)]
87  
-
88 59
 def module_list(path):
89 60
     """
90 61
     Return the list containing the names of the modules available in the given
@@ -265,7 +236,7 @@ def module_completer(self,event):
265 236
 def magic_run_completer(self, event):
266 237
     """Complete files that end in .py or .ipy for the %run command.
267 238
     """
268  
-    comps = shlex_split(event.line)
  239
+    comps = arg_split(event.line)
269 240
     relpath = (len(comps) > 1 and comps[-1] or '').strip("'\"")
270 241
 
271 242
     #print("\nev=", event)  # dbg
61  IPython/core/tests/test_completerlib.py
... ...
@@ -0,0 +1,61 @@
  1
+# -*- coding: utf-8 -*-
  2
+"""Tests for completerlib.
  3
+
  4
+"""
  5
+from __future__ import absolute_import
  6
+
  7
+#-----------------------------------------------------------------------------
  8
+# Imports
  9
+#-----------------------------------------------------------------------------
  10
+
  11
+import os
  12
+import shutil
  13
+import sys
  14
+import tempfile
  15
+import unittest
  16
+from os.path import join
  17
+
  18
+import nose.tools as nt
  19
+from nose import SkipTest
  20
+
  21
+from IPython.core.completerlib import magic_run_completer
  22
+from IPython.testing import decorators as dec
  23
+from IPython.testing import tools as tt
  24
+from IPython.utils import py3compat
  25
+
  26
+
  27
+class MockEvent(object):
  28
+    def __init__(self, line):
  29
+        self.line = line
  30
+
  31
+#-----------------------------------------------------------------------------
  32
+# Test functions begin
  33
+#-----------------------------------------------------------------------------
  34
+class Test_magic_run_completer(unittest.TestCase):
  35
+    def setUp(self):
  36
+        self.BASETESTDIR = tempfile.mkdtemp()
  37
+        for fil in [u"aaå.py", u"a.py", u"b.py"]:
  38
+            with open(join(self.BASETESTDIR, fil), "w") as sfile:
  39
+                sfile.write("pass\n")
  40
+        self.oldpath = os.getcwdu()
  41
+        os.chdir(self.BASETESTDIR)
  42
+
  43
+    def tearDown(self):
  44
+        os.chdir(self.oldpath)
  45
+        shutil.rmtree(self.BASETESTDIR)
  46
+
  47
+    def test_1(self):
  48
+        """Test magic_run_completer, should match two alterntives
  49
+        """
  50
+        event = MockEvent(u"%run a")
  51
+        mockself = None
  52
+        match = magic_run_completer(mockself, event) 
  53
+        self.assertEqual(match, [u"a.py", u"aaå.py",])
  54
+
  55
+    def test_2(self):
  56
+        """Test magic_run_completer, should match one alterntive
  57
+        """
  58
+        event = MockEvent(u"%run aa")
  59
+        mockself = None
  60
+        match = magic_run_completer(mockself, event) 
  61
+        self.assertEqual(match, [u"aaå.py",])
2  IPython/frontend/qt/console/tests/test_ansi_code_processor.py
@@ -110,7 +110,6 @@ def test_carriage_return(self):
110 110
         self.assertEquals(len(self.processor.actions), 1)
111 111
         action = self.processor.actions[0]
112 112
         self.assertEquals(action.action, 'carriage-return')
113  
-        self.assertEquals(action.count, 1)
114 113
 
115 114
     def test_beep(self):
116 115
         """ Are beep characters processed correctly?
@@ -120,7 +119,6 @@ def test_beep(self):
120 119
         self.assertEquals(len(self.processor.actions), 1)
121 120
         action = self.processor.actions[0]
122 121
         self.assertEquals(action.action, 'beep')
123  
-        self.assertEquals(action.count, 1)
124 122
 
125 123
 
126 124
 if __name__ == '__main__':
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.