Unicode tests fail with Python 2

[qauge]

I have trouble getting tests right from the repository freshly cloned from master.

Some "unicode tests" won't pass with Python 2 (UnicodeDecodeError). Those are test_ndjson for csvjson, test_unicode for csvlook, test_sort_date for csvsort, and test_sort_string_reverse for csvsort.

Sample output:

ERROR: test_ndjson (tests.test_utilities.test_csvjson.TestCSVJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/quentin/work/csvkit/tests/test_utilities/test_csvjson.py", line 108, in test_ndjson
    '{"text": "Unicode! Σ", "float": null, "datetime": null, "boolean": null, "time": null, "date": null, "integer": null}',
  File "/home/quentin/work/csvkit/tests/utils.py", line 56, in assertLines
    lines = self.get_output_as_list(args)
  File "/home/quentin/work/csvkit/tests/utils.py", line 44, in get_output_as_list
    return self.get_output(args).split('\n')
  File "/home/quentin/work/csvkit/tests/utils.py", line 37, in get_output
    utility.main()
  File "/home/quentin/work/csvkit/csvkit/utilities/csvjson.py", line 168, in main
    table = agate.Table.from_csv(self.input_file, sniff_limit=self.args.sniff_limit, column_types=self.get_column_types(), **self.reader_kwargs)
  File "/home/quentin/work/csvkit/.tox/pypy/site-packages/agate/table/__init__.py", line 362, in from_csv
    contents = contents.encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 295: ordinal not in range(128)

Is it something on my side only?

[jpmckinney]

This started erroring in Python 2 between agate 1.3.0 and 1.3.1 (i.e. tests pass using agate 1.3.0). @onyxfish @nbedi Any idea what the problem could be?

[nbedi]

Seems to do with this commit wireservice/agate@6fea5a0.

It added this bit to agate Table initialization:

if six.PY2:
          contents = contents.encode('utf-8')

I don't know much about that commit and issue but @onyxfish will know more.

[nbedi]

So I think the issue is that agate's from_csv now expects that a file has been read with proper encoding if passing a file-like object. But the encoding argument isn't used to open the file in _open_input_file if in Python 2. I may be wrong, does that seem like the issue @jpmckinney?

[nbedi]

These tests seem to pass if I give from_csv the path argument instead of the LazyFile object (and let agate figure out encoding). Would that solution affect performance? Not sure if a LazyFile helps performance when using from_csv.

[jpmckinney]

@nbedi Can you create a patch? I'm not sure what it looks like based only on your description.

[jpmckinney]

LazyFile was implemented to resolve #178.