2to3 does not preserve line endings

[bialix]

BPO	11594
Nosy	@jaraco, @benjaminp, @merwok, @ambv, @miss-islington, @aaronang
PRs	bpo-11594: Ensure line-endings are respected when using 2to3 #6483 [3.7] bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483) #6515
Files	2to3-crlf-test.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/benjaminp'
closed_at = <Date 2018-07-27.19:59:12.142>
created_at = <Date 2011-03-18.16:27:27.463>
labels = ['3.7', '3.8', 'type-bug', 'expert-2to3', 'easy']
title = '2to3 does not preserve line endings'
updated_at = <Date 2018-07-27.19:59:12.141>
user = 'https://bugs.python.org/bialix'

bugs.python.org fields:

activity = <Date 2018-07-27.19:59:12.141>
actor = 'jaraco'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2018-07-27.19:59:12.142>
closer = 'jaraco'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2011-03-18.16:27:27.463>
creator = 'bialix'
dependencies = []
files = ['22292']
hgrepos = []
issue_num = 11594
keywords = ['patch', 'easy']
message_count = 15.0
messages = ['131335', '131349', '131626', '131639', '131645', '131652', '137983', '307865', '315313', '315319', '315349', '315423', '315426', '322449', '322511']
nosy_count = 8.0
nosy_names = ['jaraco', 'bialix', 'benjamin.peterson', 'eric.araujo', 'lukasz.langa', 'miss-islington', 'Aaron Ang', 'Slawomir Nowaczyk']
pr_nums = ['6483', '6515']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11594'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[bialix]

I'm using LF-only line-endings for development of my IntelHex library. I'm working on Windows most of the time.

After 2to3 tool has been ran on my library it has not only changed the Python syntax, but it also saved all files with CRLF line-endings. As result I have all changed files completelly modified and diff shows change in every line.

2to3 tool should respect my line-endings and must not use simple open(xxx, "wt") mode for writing modified files.

[merwok]

Thanks for the report. Can you run “python -m test.test_lib2to3”, if possible with a Python 3.x version? I’ve seen that the tests use binary mode to compare file contents, so maybe you will get an error message that can get us started.

[bialix]

@Éric Araujo: I've ran tests with python 3.2. All tests have passed:

----------------------------------------------------------------------
Ran 540 tests in 37.688s

OK

[merwok]

Thanks. Would you like to work on a unit test or full patch?

[bialix]

Éric, thank you for the proposal, but I'm not familiar enough with the codebase to work on it.

The short scan over the tests reveals that there is at least one test which tries to test CRLF behavior, in the file test_refactor.py, but I don't understand what it doing?

    def test_crlf_newlines(self):
        old_sep = os.linesep
        os.linesep = "\r\n"
        try:
            fn = os.path.join(TEST_DATA_DIR, "crlf.py")
            fixes = refactor.get_fixers_from_package("lib2to3.fixes")
            self.check_file_refactoring(fn, fixes)
        finally:
            os.linesep = old_sep

So, in theory I can modify that test to check what if the file has LF-only line-endings originally, but os.linesep is CRLF, but then I don't know what the content I should create and how to run fixer over that.

[benjaminp]

I can fix it. I just need to find time. :)

[merwok]

I was surprised to see that the crlf.py file was not using CRLF in the new Mercurial repo. It is also not in the .hgeol file. I changed it locally, but it doesn’t change anything, the tests pass before and after the change.

[jaraco]

This issue affected me today. I'm editing a codebase that has mixed line endings in different files. I'd like to patch for Python 3 support without changing line endings. Even invoking a single fixer (print), the line endings mutate. Since I'm running on macOS, the files with CRLF get LF line endings.

Answers in this question suggest the mutation can be suppressed by altering the _to_system_newlines function, but the proposed fix has no effect on the Python 3.6 version of this routine.

I thought I'd encountered this issue before, but I realized after searching that I was thinking of bpo-10639, which may present a model for retaining the newlines when refactoring the code.

I found I was able to work around the issue by putting lib2to3-clean.py in my current directory:

import lib2to3.refactor
import runpy
import functools


if __name__ == '__main__':
	lib2to3.refactor._open_with_encoding = functools.partial(open, newline='')
	runpy.run_module('lib2to3')

And invoking python -m lib2to3-clean instead of -m lib2to3. The addition of newline='' follows the guidance in the docs on how to avoid mutating newlines.

I've released this functionality in jaraco.develop 4.0 so others can readily invoke it with rwt jaraco.develop -- -m jaraco.develop.lib2to3.

[aaronang]

I couldn't reproduce this issue. I tried reproducing this problem by extending the TestRefactoringTool class and creating two files: one file with LF line-endings and one file with CRLF line-endings.

The changes that I made can be found here: aaronang@55e8bd3.

The output of the tests:

Before: b'print("hi")\n\nprint("Like bad Windows newlines?")\n'
After: b'print("hi")\n\nprint("Like bad Windows newlines?")\n'

Before: b'print("hi")\r\n\r\nprint("Like bad Windows newlines?")\r\n'
After: b'print("hi")\r\n\r\nprint("Like bad Windows newlines?")\r\n'

Maybe this problem has been resolved?

[jaraco]

I do still see the issue on Python 3.7b3:

$ python ~/Dropbox/bin/scripts/which-line-ending onefile.py
Line ending is '\n'
$ python ~/Dropbox/bin/scripts/which-line-ending otherfile.py
Line ending is '\r\n'
$ python -V
Python 3.7.0b3
$ python -m lib2to3 . -w
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Refactored ./onefile.py
--- ./onefile.py        (original)
+++ ./onefile.py        (refactored)
@@ -1 +1 @@
-print 'hello world'
+print('hello world')
RefactoringTool: Refactored ./otherfile.py
--- ./otherfile.py      (original)
+++ ./otherfile.py      (refactored)
@@ -1 +1 @@
-print 'hello world'
+print('hello world')
RefactoringTool: Files that were modified:
RefactoringTool: ./onefile.py
RefactoringTool: ./otherfile.py
$ $ python ~/Dropbox/bin/scripts/which-line-ending onefile.py
Line ending is '\n'
$ python ~/Dropbox/bin/scripts/which-line-ending otherfile.py
Line ending is '\n'

[aaronang]

@jason R. Coombs
You are right. I managed to reproduce the problem with a test. It only occurs when a fix is applied.

Also, I figured out that the refactoring reads in the file using open(file, 'r'), which basically transforms all line-endings to LF regardless the used line-endings. I think I fixed the problem, looking forward to receiving feedback 😬

[ambv]

New changeset c127a86 by Łukasz Langa (Aaron Ang) in branch 'master':
bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483)
c127a86

[miss-islington]

New changeset 3b3be1f by Miss Islington (bot) in branch '3.7':
bpo-11594: Ensure line-endings are respected when using 2to3 (GH-6483)
3b3be1f

[SlawomirNowaczyk]

I've just tried 2to3 in 3.7.0, and it still breaks line endings for me...

Seems like one also needs to fix line 517 in refactor.py
fp = io.open(filename, "w", encoding=encoding, newline='')

[jaraco]

Slawomir, you're probably looking for bpo-34108.