UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 20355: ordinal not in range(128)

[yarikoptic]

While building a fresh package (out of snapshot of master at bc7d48f, since the last release had way too many failures) for Debian I am running into

======================================================================
ERROR: test_file_like (pandas.io.tests.test_html.TestReadHtml)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/io/tests/test_html.py", line 284, in test_file_like
    df1 = self.read_html(f, '.*Water.*')
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/io/tests/test_html.py", line 92, in read_html
    return read_html(*args, **kwargs)
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/io/html.py", line 865, in read_html
    parse_dates, tupleize_cols, thousands, attrs, encoding)
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/io/html.py", line 728, in _parse
    raise_with_traceback(retained)
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/compat/__init__.py", line 708, in raise_with_traceback
    raise exc.with_traceback(traceback)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 20355: ordinal not in range(128)

======================================================================
ERROR: test_string (pandas.io.tests.test_html.TestReadHtml)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/io/tests/test_html.py", line 275, in test_string
    data = f.read()
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 20355: ordinal not in range(128)

======================================================================
ERROR: test_string_io (pandas.io.tests.test_html.TestReadHtml)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/pandas-0.16.1+git97-gbc7d48f/debian/tmp/usr/lib/python3/dist-packages/pandas/io/tests/test_html.py", line 264, in test_string_io
    data1 = StringIO(f.read())
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 20355: ordinal not in range(128)

when building with Python 3.4.3+ . Locale is set to C while running tests

[jreback]

show pd.get_option('display.encoding')

[jreback]

Try this out: jreback@2b93412

[jreback]

@cpcloud ping

[yarikoptic]

(Pdb) import pandas as pd
(Pdb) print(pd.get_option('display.encoding'))
ANSI_X3.4-1968

[yarikoptic]

I think I have patched it alright with the suggested patch but the same result

[teh]

Running into the same issue with pandas 0.16.1 but at a different byte offset:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 21552: ordinal not in range(128)

export | grep LANG
LANG="en_GB.UTF-8"

>>> pd.__version__
'0.16.1'
>>> pd.get_option('display.encoding')
'UTF-8'

[adnelson]

I also encountered this in the unit tests for pandas 0.17.1. For me, what fixed it was to replace the calls to open(self.spam_data) to open(self.spam_data, encoding='utf-8'). Hopefully this works for others. I should also note that the errors described here only occurred for me when the locale was set to C, as opposed to a UTF-8 locale (they passed with that).

[jreback]

@adnelson hmm, that might work for the last 3 erros above (as the test comparison was not decoded properly). I think the first one might need an encoding arg in the .read_html(...). Want to take a crack at this to make it fail on Travis? we already have some alt locale builds (look for LOCALE_OVERRIDE env var in the .travis.yml, but its possible they are skipping these tests.