UnicodeDecodeError when 2to3 is run on a dir with numpy .npy files

[zbysz]

BPO	10416
Nosy	@benjaminp
Files	diff.diff: patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/benjaminp'
closed_at = <Date 2010-11-14.15:29:06.901>
created_at = <Date 2010-11-14.11:34:03.937>
labels = ['expert-2to3', 'type-crash']
title = 'UnicodeDecodeError when 2to3 is run on a dir with numpy .npy files'
updated_at = <Date 2010-11-14.15:29:06.900>
user = 'https://bugs.python.org/zbysz'

bugs.python.org fields:

activity = <Date 2010-11-14.15:29:06.900>
actor = 'benjamin.peterson'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2010-11-14.15:29:06.901>
closer = 'benjamin.peterson'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2010-11-14.11:34:03.937>
creator = 'zbysz'
dependencies = []
files = ['19605']
hgrepos = []
issue_num = 10416
keywords = ['patch']
message_count = 2.0
messages = ['121188', '121196']
nosy_count = 2.0
nosy_names = ['benjamin.peterson', 'zbysz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue10416'
versions = ['Python 3.2']

[zbysz]

1. 2to3 should work only only files ending with '.py', but it takes
   anything which has a dot and ends with 'py'. I'm having trouble
   with numpy .npy files.
2. 2to3 tries to decode the file and fails with traceback that is not useful:
   the name of the failing file is not given.

A patch is attached.

% ls *.npy|head -n1
S_18_7000_899811b572b309161cbb34f185b82fb618ed81da.npy

% 2to3-3.2 /usr/local/bin/2to3-3.2 .
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
Traceback (most recent call last):
  File "/usr/local/bin/2to3-3.2", line 6, in <module>
    sys.exit(main("lib2to3.fixes"))
  File "/usr/local/lib/python3.2/lib2to3/main.py", line 172, in main
    options.processes)
  File "/usr/local/lib/python3.2/lib2to3/refactor.py", line 699, in refactor
    items, write, doctests_only)
  File "/usr/local/lib/python3.2/lib2to3/refactor.py", line 294, in refactor
    self.refactor_dir(dir_or_file, write, doctests_only)
  File "/usr/local/lib/python3.2/lib2to3/refactor.py", line 313, in refactor_dir
    self.refactor_file(fullname, write, doctests_only)
  File "/usr/local/lib/python3.2/lib2to3/refactor.py", line 740, in refactor_file
    *args, **kwargs)
  File "/usr/local/lib/python3.2/lib2to3/refactor.py", line 335, in refactor_file
    input, encoding = self._read_python_source(filename)
  File "/usr/local/lib/python3.2/lib2to3/refactor.py", line 331, in _read_python_source
    return _from_system_newlines(f.read()), encoding
  File "/usr/local/lib/python3.2/codecs.py", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x93 in position 0: invalid start byte

[benjaminp]

r86464