test_csv struni fixes + unicode support in _csv

[hupp]

BPO	1767398
Nosy	@gvanrossum, @smontanaro
Files	py3k-struni-csv.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2008-01-06.22:29:45.759>
created_at = <Date 2007-08-04.00:11:43.000>
labels = []
title = 'test_csv struni fixes + unicode support in _csv'
updated_at = <Date 2008-01-06.22:29:45.759>
user = 'https://bugs.python.org/hupp'

bugs.python.org fields:

activity = <Date 2008-01-06.22:29:45.759>
actor = 'admin'
assignee = 'gvanrossum'
closed = True
closed_date = None
closer = None
components = ['None']
creation = <Date 2007-08-04.00:11:43.000>
creator = 'hupp'
dependencies = []
files = ['8154']
hgrepos = []
issue_num = 1767398
keywords = ['patch']
message_count = 4.0
messages = ['52984', '52985', '52986', '52987']
nosy_count = 3.0
nosy_names = ['gvanrossum', 'skip.montanaro', 'hupp']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1767398'
versions = ['Python 3.0']

[hupp]

This patch fixes test_csv.py for the struni branch and modifies _csv.c to support unicode strings.

Changes:

1. The test_csv.py failures caused by bytes/str conflicts have been resolved.

2. Uses of mkstemp have been replaced with TemporaryFile in a 'with' block.

3. The _csv.c module now uses unicode for string handling. I've uncommented the unicode read tests in test_csv.py, and added tests for writing unicode content and a unicode delimiter.

All tests are now passing on my system (linux).

[smontanaro]

Adam,

I've spent some time looking at this patch. Bear in mind this is my first foray into Py3k. Still, I'm confused about what's going on here. I'm hoping you can help me understand the changes. In parse_save_field, you replaced PyString_FromStringAndSize with PyUnicode_FromUnicode, however in get_nullchar_as_None you replaced it with PyUnicode_DecodeASCII.

When I execute the csv tests there are a number of assertion errors related to the default delimiter. The traceback goes something like this:

FAIL: test_writer_kw_attrs (main.Test_Csv)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "Lib/test/test_csv.py", line 88, in test_writer_kw_attrs
    self._test_kw_attrs(csv.writer, StringIO())
  File "Lib/test/test_csv.py", line 75, in _test_kw_attrs
    self.assertEqual(obj.dialect.delimiter, ':')
AssertionError: s'\x00' != ':'

Any idea how to solve that? It looks to me like some Unicode buffer might be getting interpreted as a char *, but I'm not sure.

Skip

[hupp]

Skip,
I think the error you're seeing is being caused by a conversion from Py_UNICODE -> char -> unicode through get_nullchar_as_None. That function should look like this:

static PyObject *
get_nullchar_as_None(Py_UNICODE c)
{
        if (c == '\0') {
                Py_INCREF(Py_None);
                return Py_None;
        }
        else
            return PyUnicode_FromUnicode((Py_UNICODE*)&c, 1);
}

Unfortunately I'm on the road right now so I can't test it.

Is there something I need to do with my build to trigger those assertions? I didn't see them.

[gvanrossum]

This looked good enough to submit.
I had to clean up the whitespace use in the C code. Please next time set your tabs to 8 spaces when editing C code. Also try to conform to the surrounding code's use of spaces or tab (unfortunately this file is inconsistent and sometimes uses spaces, other times tabs -- that's worth a separate cleanup).

Committed revision 56777.