Python 3's 2to3 does not handle non-ascii source files

[zzzeek]

BPO	7922
Nosy	@benjaminp

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-02-13.02:19:53.452>
created_at = <Date 2010-02-13.01:56:54.607>
labels = ['expert-2to3', 'type-crash']
title = "Python 3's 2to3 does not handle non-ascii source files"
updated_at = <Date 2010-02-13.02:20:37.406>
user = 'https://bugs.python.org/zzzeek'

bugs.python.org fields:

activity = <Date 2010-02-13.02:20:37.406>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2010-02-13.02:19:53.452>
closer = 'benjamin.peterson'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2010-02-13.01:56:54.607>
creator = 'zzzeek'
dependencies = []
files = []
hgrepos = []
issue_num = 7922
keywords = []
message_count = 6.0
messages = ['99298', '99299', '99300', '99301', '99302', '99303']
nosy_count = 2.0
nosy_names = ['benjamin.peterson', 'zzzeek']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue7922'
versions = ['Python 3.1']

[zzzeek]

given the following Python 2 source file:

# -*- encoding: utf-8

print 'bien mangé'

It can be converted to Python 3 using 2's 2to3 tool:

classic$ 2to3 test.py
 ... omitted ...
--- test.py (original)
+++ test.py (refactored)
@@ -1,3 +1,3 @@
 \# -*- encoding: utf-8

-print 'bien mangé'
+print('bien mangé')

However that of Python 3.1.1 fails:

    classic$ 2to3-3.1 test.py
       ... omitted ...
    --- test.py (original)
    +++ test.py (refactored)
    @@ -1,3 +1,3 @@
     # -*- encoding: utf-8
 
    Traceback (most recent call last):
      File "/usr/local/bin/2to3-3.1", line 6, in <module>
        sys.exit(main("lib2to3.fixes"))
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/main.py", line 159, in main
        options.processes)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 616, in refactor
        items, write, doctests_only)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 276, in refactor
        self.refactor_file(dir_or_file, write, doctests_only)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 656, in refactor_file
        *args, **kwargs)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 332, in refactor_file
        write=write, encoding=encoding)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/refactor.py", line 432, in processed_file
        self.print_output(old_text, new_text, filename, equal)
      File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/lib2to3/main.py", line 64, in print_output
        print(line)
    UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 17: ordinal not in range(128)

[benjaminp]

Please try 2to3 from 2.7a3 or the 2to3 trunk.

[benjaminp]

Sorry, I meant from the py3k branch.

[zzzeek]

yes, its handled:

WARNING: couldn't encode test.py's diff for your terminal

is that fix specific to 2to3 or is that just how "print" works in 3.2 ?

[benjaminp]

2010/2/12 mike bayer <report@bugs.python.org>:

mike bayer <mike_mp@zzzcomputing.com> added the comment:

yes, its handled:

WARNING: couldn't encode test.py's diff for your terminal

is that fix specific to 2to3 or is that just how "print" works in 3.2 ?

It's just that whatever python guesses your terminal encoding is (try
print(sys.stdin.encoding)) might not be able to handle whatever
characters are in the source file, so we just don't print it. You can
change the default encoding with the PYTHONIOENCODING env var.

[benjaminp]

You can just use the --no-diffs option, btw, to avoid this problem on earlier versions.