csvw.dsv fails on Python 2 with encodings that are not 8bit-clean

[xflr6]

>>> from csvw.dsv import UnicodeWriter, UnicodeReader
>>> with UnicodeWriter(encoding='utf-16') as writer:
	writer.writerow([u'spam'])
>>> writer.f.seek(0)
>>> with UnicodeReader(writer.f, encoding='utf-16') as reader:
	print next(reader)
Traceback (most recent call last):
...
UnicodeDecodeError: 'utf16' codec can't decode byte 0xb3 in position 2: truncated data
>>> writer.f.getvalue()
'\xff\xfes\r\n'

To allow arbitary encodings with Python 2 csv, cells must be encoded into utf-8 before writing so that the output of the csv.writer can then be recoded into the wanted target encoding. On reading, first recode into utf-8 and then decode the cells from csv.reader (currently, only csvw.dsv.UnicodeReader but not UnicodeWriter does this re-encoding suggested in the csv docs).

As an optimization, the recoding can be skipped for 8bit-clean encodings, cf. csvkit/agate:
https://github.com/wireservice/agate/blob/233afefbc7c0b25084666a2dd2b315b6359a128a/agate/csv_py2.py#L14-L17

[xrotwang]

Limiting the whole library to utf-8 would probably not be a good idea, I guess. Although recoding of files could easily be done elsewhere ...

[xflr6]

+1, AFAIR some versions of excel will produce/want utf-16 for unicode.

[xflr6]

Test case is here:

csvw/tests/test_dsv.py

Lines 93 to 107 in 84b4628

	@pytest.mark.parametrize('encoding', [
	pytest.param('utf-16', marks=pytest.mark.xfail(sys.version_info.major == 2, reason='FIXME: #5',
	raises=UnicodeDecodeError)),
	pytest.param('utf-8-sig', marks=pytest.mark.xfail(sys.version_info.major == 2, reason='FIXME: #5')),
	'utf-8',
	])
	def test_roundtrip_multibyte(tmpdir, encoding, row=['spam', 'eggs'], expected='spam,eggs\r\n', n=2):
	filepath = tmpdir / 'spam.csv'
	kwargs = {'encoding': encoding}
	with UnicodeWriter(str(filepath), **kwargs) as writer:
	writer.writerows([row] * n)
	with UnicodeReader(str(filepath), **kwargs) as reader:
	result = next(reader)
	assert result == row
	assert filepath.read_binary() == (expected * n).encode(encoding)