Bad indentation in urllib import fixer with multiple imports

[davidmalcolm]

BPO	9386
Nosy	@benjaminp, @merwok, @davidmalcolm
Files	urllib-indentation-issue.patch lib2to3.diff: patch against the trunk.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/benjaminp'
closed_at = <Date 2010-08-07.23:55:39.000>
created_at = <Date 2010-07-26.17:28:23.867>
labels = ['type-bug', 'expert-2to3']
title = 'Bad indentation in urllib import fixer with multiple imports'
updated_at = <Date 2010-08-07.23:55:38.999>
user = 'https://github.com/davidmalcolm'

bugs.python.org fields:

activity = <Date 2010-08-07.23:55:38.999>
actor = 'benjamin.peterson'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2010-08-07.23:55:39.000>
closer = 'benjamin.peterson'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2010-07-26.17:28:23.867>
creator = 'dmalcolm'
dependencies = []
files = ['18212', '18227']
hgrepos = []
issue_num = 9386
keywords = ['patch']
message_count = 4.0
messages = ['111649', '111788', '111794', '113219']
nosy_count = 4.0
nosy_names = ['benjamin.peterson', 'eric.araujo', 'dmalcolm', 'ysj.ray']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9386'
versions = ['Python 2.7']

[davidmalcolm]

Running 2to3 on lxml-2.2.6 failed with broken indentation.

This fragment from lxml/html/init.py:

def open_http_urllib(method, url, values):
## FIXME: should test that it's not a relative URL or something
try:
from urllib import urlencode, urlopen
except ImportError: # Python 3

became:

def open_http_urllib(method, url, values):
## FIXME: should test that it's not a relative URL or something
try:
from urllib.parse import urlencode
from urllib.request import urlopen
except ImportError: # Python 3

which is syntactically invalid: note the invalid indentation for the second "import" line.

Seems to work when there's a single name imported per line; fails when more than one name is imported from urlib.

Am attaching a patch to test_fixers (for the 2.7 branch) which adds a reproducer. I don't have a fix.

test test_lib2to3 failed -- Traceback (most recent call last):
  File "/home/david/coding/python-svn/2.7-2to3/Lib/lib2to3/tests/test_fixers.py", line 1858, in test_import_indentation
    self.check(b, a)
  File "/home/david/coding/python-svn/2.7-2to3/Lib/lib2to3/tests/test_fixers.py", line 37, in check
    tree = self._check(before, after)
  File "/home/david/coding/python-svn/2.7-2to3/Lib/lib2to3/tests/test_fixers.py", line 33, in _check
    self.assertEqual(after, unicode(tree))
AssertionError: u"\ndef foo():\n    from urllib.parse import urlencode\n    from urllib.parse im [truncated]... != u"\ndef foo():\n    from urllib.parse import urlencode\nfrom urllib.request impo [truncated]...
  
  def foo():
      from urllib.parse import urlencode
-     from urllib.parse import urlopen
? ----            --  ^
+ from urllib.request import urlopen
?              ++++ ^
      print('got here')
  
(Note to self: tracking this downstream in Fedora for lxml as https://bugzilla.redhat.com/show_bug.cgi?id=600036 )

[ysjray]

I guess it's the problem with lib2to3/fixes/fix_urllib.py. Indentation is not taken into consideration when fix "import". Fix it with indentation taken into consideration maybe a little complex, but here is a simple and ugly fix: when one import statement was transformed to two or more statement, use semicolon(';') instead of '\n' as separator, between many import statements. In this case, the
"""
def fun():
from urllib import urlopen, urlencode
"""

is transformed in to:

"""
def fun():
from urllib.request import urlopen;from urllib.parse import urlencode
"""

It will takes time to work out a better fix.

[ysjray]

When one import statement is split to two or more, we encounter this problem: the indentation of the import statements except the first one is unknown, and is difficult to fix this problem, since a import maybe in a multi-statement line, like: 'a=1;import sys'. I wonder if there is way to fix this problem perfectly. Maybe we could just put all the resulting import statements into one single multi-statement line, joined by ';', as my patch specified.

[benjaminp]

Fixed in r83798.