os.walk fails on undecodable filenames

[fhoech]

BPO	22862
Nosy	@vstinner, @ezio-melotti

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-11-13.15:15:54.745>
created_at = <Date 2014-11-13.13:16:20.214>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'os.walk fails on undecodable filenames'
updated_at = <Date 2014-11-13.15:16:27.669>
user = 'https://bugs.python.org/fhoech'

bugs.python.org fields:

activity = <Date 2014-11-13.15:16:27.669>
actor = 'fhoech'
assignee = 'none'
closed = True
closed_date = <Date 2014-11-13.15:15:54.745>
closer = 'r.david.murray'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2014-11-13.13:16:20.214>
creator = 'fhoech'
dependencies = []
files = []
hgrepos = []
issue_num = 22862
keywords = []
message_count = 7.0
messages = ['231110', '231111', '231112', '231115', '231117', '231118', '231120']
nosy_count = 3.0
nosy_names = ['vstinner', 'ezio.melotti', 'fhoech']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22862'
versions = ['Python 2.7']

[fhoech]

If 'top' is an unicode directory name, os.listdir can still return non-unicode filenames if they can't be decoded. This case is not handled in the Python 2.x standard library version of os.walk and will cause join(top, name) to fail on such filenames with an UnicodeDecodeError.

[vstinner]

What is your OS?

[fhoech]

This problem only affects Linux as far as I know (in my case I'm using Fedora 21 Beta).

[vstinner]

Your problem has two solutions.

1. Upgrade to Python 3 which handles correctly your use case (thanks to the PEP-383, surrogateescape error handler)

2. Only process filenames as bytes, and encode/decode manually (so you can decide how to handle undecodable filenames)

[fhoech]

1. Is not yet possible for me unfortunately, some libraries I require are not yet available for Python 3 (but in the long run, this would be my preferred solution)

2. Would necessitate too many changes in a carefully crafted, unicode-only application. I think I'll just override os.listdir and filter out filenames that are not decodable, or override os.walk and do something equivalent.

[vstinner]

1. Is not yet possible for me unfortunately, some libraries I require are not yet available for Python 3 (but in the long run, this would be my preferred solution)

I'm curious, which libraries?

Oh, I forgot to say that it's not possible to fix this issue in Python 2. Backporting the PEP-383 in Python 2 requires deep changes in the Unicode machinery, starting by the UTF-8 codec. Currently, the UTF-8 encoder encodes surrogates which violates Unicode standard and makes impossible to use this codec with the surrogateescape error handler.

[fhoech]

I'm curious, which libraries?

wxPython and wexpect (wexpect I could probably port myself, so the problem is mainly with wx)

Oh, I forgot to say that it's not possible to fix this issue in Python 2. Backporting the PEP-383 in Python 2 requires deep changes in the Unicode machinery, starting by the UTF-8 codec.

Ok, that's understandable of course.