os functions return '??' for unicode characters in paths on windows

[CreationElemental]

BPO	35670
Nosy	@pfmoore, @tjguk, @zware, @serhiy-storchaka, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-06.12:27:04.885>
created_at = <Date 2019-01-06.10:26:47.304>
labels = ['type-bug', 'OS-windows']
title = "os functions return '??' for unicode characters in paths on windows"
updated_at = <Date 2019-01-06.20:46:19.597>
user = 'https://bugs.python.org/CreationElemental'

bugs.python.org fields:

activity = <Date 2019-01-06.20:46:19.597>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-06.12:27:04.885>
closer = 'serhiy.storchaka'
components = ['Windows']
creation = <Date 2019-01-06.10:26:47.304>
creator = 'Creation Elemental'
dependencies = []
files = []
hgrepos = []
issue_num = 35670
keywords = []
message_count = 3.0
messages = ['333100', '333106', '333118']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'Creation Elemental']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35670'
versions = ['Python 2.7']

[CreationElemental]

I have a few files that contain emojis in their names, and also a folder that has such. Commands like os.getcwd, os.listdir, os.path.realpath, etc. will cause this to happen. However, this is only, as far as I can tell, happening on pure windows distributions. This does not happen in the cygwin64 version I have, nor does it happen in python3.

For example, say you have a folder simply called '🔭'. If you run python inside of it and run os.getcwd() you will simply get '??' as the result. This breaks MANY of my programs that depend on knowing exactly where they are, and knowing the contents of a directory to pass to other functions.

[serhiy-storchaka]

This is a known issue. See for example bpo-13247 and bpo-16656. It cannot be fixed in Python 2. The only thing that can be done is to document it (see bpo-16700).

[zooba]

To be more clear, the fix is literally Python 3 and the str/bytes change. You've discovered one of the reasons that was necessary.