2to3 -d adds extra whitespace

[VPeric]

BPO	12771
Nosy	@benjaminp, @asmeurer, @tirkarthi
Superseder	bpo-45544: Close 2to3 issues and list them here

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-20.23:07:56.696>
created_at = <Date 2011-08-17.14:59:47.295>
labels = ['3.7', '3.8', 'type-bug', 'expert-2to3']
title = '2to3 -d adds extra whitespace'
updated_at = <Date 2021-10-20.23:07:56.695>
user = 'https://bugs.python.org/VPeric'

bugs.python.org fields:

activity = <Date 2021-10-20.23:07:56.695>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.23:07:56.696>
closer = 'iritkatriel'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2011-08-17.14:59:47.295>
creator = 'VPeric'
dependencies = []
files = []
hgrepos = []
issue_num = 12771
keywords = []
message_count = 3.0
messages = ['142281', '221914', '338345']
nosy_count = 4.0
nosy_names = ['benjamin.peterson', 'VPeric', 'Aaron.Meurer', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '45544'
type = 'behavior'
url = 'https://bugs.python.org/issue12771'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[VPeric]

When running 2to3 -d on this doctest (from this file[0] in SymPy):

        >>> class SzUpKet(Ket):
        ...     def _represent_SzOp(self, basis, **options):
        ...         return Matrix([1,0])
        ...

2to3 adds an extra space in the last line. This then raises an error for our automated whitespace tests (and is generally annoying). I haven't seen this happen anywhere else (and SymPy is a big codebase).

It's really a minor issue, though (I can't set the priority myself, though).

[0] https://github.com/sympy/sympy/blob/master/sympy/physics/quantum/represent.py

[BreamoreBoy]

Can we have a response to this please.

[tirkarthi]

PS2 = "... " is defined with a trailing space which is not stripped for empty lines with only PS2 in the doctest. A patch would be to strip the trailing space in PS2 for empty lines and a unittest would be as below. There are no test cases for this scenario asserting against exact format and hence the patch doesn't break any tests. Currently I don't see any doctest in the codebase having this leading whitespace. I can try converting this patch as PR if Benjamin is okay with the change.

diff --git a/Lib/lib2to3/refactor.py b/Lib/lib2to3/refactor.py
index 7841b99a5c..135678b46e 100644
--- a/Lib/lib2to3/refactor.py
+++ b/Lib/lib2to3/refactor.py
@@ -599,7 +599,11 @@ class RefactoringTool(object):
                 new[-1] += "\n"
             block = [indent + self.PS1 + new.pop(0)]
             if new:
-                block += [indent + self.PS2 + line for line in new]
+                for line in new:
+                    if not line.strip():
+                        block += [indent + self.PS2.strip() + line]
+                    else:
+                        block += [indent + self.PS2 + line]
         return block
 
     def summarize(self):
diff --git a/Lib/lib2to3/tests/test_refactor.py b/Lib/lib2to3/tests/test_refactor.py
index 9e3b8fbb90..f6a5e2d589 100644
--- a/Lib/lib2to3/tests/test_refactor.py
+++ b/Lib/lib2to3/tests/test_refactor.py
@@ -331,3 +331,22 @@ from __future__ import print_function"""
                 break
         else:
             self.fail("explicit fixer not loaded")
+
+    def test_refactor_doctest(self):
+        rt = self.rt()
+
+        expected = """
+>>> class Foo():
+...    def cheese(self):
+...        pass
+...
+"""
+
+        doc = """
+>>> class Foo():
+...    def parrot(self):
+...        pass
+...
+"""
+        out = rt.refactor_docstring(doc, "<test>")
+        self.assertEqual(out, expected)

Without patch this fails as below :

$ ./python.exe -m unittest -v lib2to3.tests.test_refactor.TestRefactoringTool.test_refactor_doctest
test_refactor_doctest (lib2to3.tests.test_refactor.TestRefactoringTool) ... FAIL

======================================================================
FAIL: test_refactor_doctest (lib2to3.tests.test_refactor.TestRefactoringTool)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/lib2to3/tests/test_refactor.py", line 352, in test_refactor_doctest
    self.assertEqual(out, expected)
AssertionError: '\n>>> class Foo():\n...    def cheese(self):\n...        pass\n... \n' != '\n>>> class Foo():\n...    def cheese(self):\n...        pass\n...\n'

  >>> class Foo():
  ...    def cheese(self):
  ...        pass
- ...
?    -
+ ...

---

Ran 1 test in 0.032s

FAILED (failures=1)