UnicodeDecodeError in ini parser when file is UTF-8 but locale encoding is used

[yuji38kwmt]

UnicodeDecodeError is raised if unicode character exists in pytest.ini.

# content of test_sample.py
def inc(x):
    return x + 1

def test_answer():
    assert inc(3) == 5

[pytest]
# あ
addopts = --verbose --capture=no

$ pytest test_sample.py
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\pytest.exe\__main__.py", line 9, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 196, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "c:\programdata\anaconda3\lib\site-packages\pluggy\hooks.py", line 258, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\programdata\anaconda3\lib\site-packages\pluggy\manager.py", line 67, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\programdata\anaconda3\lib\site-packages\pluggy\manager.py", line 61, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "c:\programdata\anaconda3\lib\site-packages\pluggy\callers.py", line 196, in _multicall
    gen.send(outcome)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\helpconfig.py", line 93, in pytest_cmdline_parse
    config = outcome.get_result()
  File "c:\programdata\anaconda3\lib\site-packages\pluggy\callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "c:\programdata\anaconda3\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 675, in pytest_cmdline_parse
    self.parse(args)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 845, in parse
    self._preparse(args, addopts=addopts)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 790, in _preparse
    self._initini(args)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\__init__.py", line 729, in _initini
    config=self,
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\findpaths.py", line 127, in determine_setup
    rootdir, inifile, inicfg = getcfg([ancestor], config=config)
  File "c:\programdata\anaconda3\lib\site-packages\_pytest\config\findpaths.py", line 34, in getcfg
    iniconfig = py.iniconfig.IniConfig(p)
  File "c:\programdata\anaconda3\lib\site-packages\py\_vendored_packages\iniconfig.py", line 54, in __init__
    tokens = self._parse(iter(f))
  File "c:\programdata\anaconda3\lib\site-packages\py\_vendored_packages\iniconfig.py", line 82, in _parse
    for lineno, line in enumerate(line_iter):
UnicodeDecodeError: 'cp932' codec can't decode byte 0x82 in position 13: illegal multibyte sequence

[asottile]

I assume your configuration file is UTF-8 encoded and you're on windows (where the default encoding is unlikely to be UTF-8) -- can you also try the same thing using python3.7?

[yuji38kwmt]

My environtmental is as follows.
I executed using python 3.7 but UnicodeDecodeError was raised.

• Windows10
• Python 3.7.0(Anaconda3)
• pytest 4.0.0
• pytest.ini is encodes with UTF-8 and line separator is LF.

> pytest --version
This is pytest version 4.0.0, imported from c:\programdata\anaconda3\lib\site-packages\pytest.py
setuptools registered plugins:
  pytest-remotedata-0.3.0 at c:\programdata\anaconda3\lib\site-packages\pytest_remotedata\plugin.py
  pytest-openfiles-0.3.0 at c:\programdata\anaconda3\lib\site-packages\pytest_openfiles\plugin.py
  pytest-doctestplus-0.1.3 at c:\programdata\anaconda3\lib\site-packages\pytest_doctestplus\plugin.py
  pytest-arraydiff-0.2 at c:\programdata\anaconda3\lib\site-packages\pytest_arraydiff\plugin.py

[yuji38kwmt]

If pytest.ini is Shift_JIS encoded, UnicodeDecodeError is not raised.

[blueyed]

Looks like a bug in py/pylib.

[Zac-HD]

I think this has been fixed by the various refactorings of encodings and config handling over the last four years, closing.

[RonnyPfannschmidt]

It's still an issue with the ini parser