Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_csv(filename_with_asian_locale) failed in python 3.6 for windows #16602

mfmain opened this issue Jun 5, 2017 · 3 comments


None yet
3 participants
Copy link

commented Jun 5, 2017


Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
>>> pd.__version__
>>> import platform
>>> platform.platform()
>>> import pandas as pd
>>> df = pd.read_csv(r'c:\tmp\中.csv')
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-0cd6317422e5>", line 1, in <module>
    df = pd.read_csv(r'c:\tmp\中.csv')
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\", line 655, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\", line 405, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\", line 762, in __init__
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\", line 966, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\", line 1582, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas\_libs\parsers.pyx", line 394, in pandas._libs.parsers.TextReader.__cinit__ (pandas\_libs\parsers.c:4209)
  File "pandas\_libs\parsers.pyx", line 712, in pandas._libs.parsers.TextReader._setup_parser_source (pandas\_libs\parsers.c:8895)
OSError: Initializing from file failed

Problem description

python 3.6 changed sys.getfilesystemencoding() to return "utf-8" instead of "mbcs"
see PEP 529.

How to fix

Here is the problem: parsers.pyx

if isinstance(source, basestring):
     if not isinstance(source, bytes):
         source = source.encode(sys.getfilesystemencoding() or 'utf-8')

the source parameter is our filename, and will be encoded to 'utf-8', not legacy 'mbcs' in python 3.6
and finally passed to open() in io.c:new_file_source
thus interpreted as a mbcs string, so, the "File not found" exception is not suprised
maybe this should be the responsiblity of cython for python 3.6 to handle these things by using unicode version of windows API,
but for now, we just replace sys.getfilesystemencoding() to "mbcs"

@mfmain mfmain changed the title read_csv(filename_with_asian_locale) failed in python 3.6 read_csv(filename_with_asian_locale) failed in python 3.6 for windows Jun 5, 2017


This comment has been minimized.

Copy link

commented Jun 5, 2017

there is a workaround with speed compromised:
python df = pd.read_csv(r'c:\tmp\中文.csv', engine='python')

but it is a dirty work to modify every single call to read_csv in all your projects


This comment has been minimized.

Copy link

commented Jun 5, 2017

this is a duplicate of this: #15086

there is a PR attached but unfortunately it was blown away.

certainly would take a fix for this.

@jreback jreback closed this Jun 5, 2017

@jreback jreback added this to the No action milestone Jun 5, 2017


This comment has been minimized.

Copy link

commented Mar 21, 2018

Do NOT use the Chinese in the file name,change it to English.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.