CSV Writer can not write strings with nulls when no escapechar is specified

[eshmu]

Bug report

This code fails in Python 3.10 with the need to escape, but no escapechar set error:

import csv

value = 'my\x00string'

with io.StringIO() as buf:
    writer = csv.writer(buf)
    writer.writerow([value])

However, this worked on Python 3.9 and earlier, and it should work since I shouldn't have to specify an escapechar when quoting is available.

I believe this is caused by the fix for this issue: #56387

See this comment on how the C snippet added for the fix introduced this bug: #56387 (comment)

[mumbleskates]

ran into this today and i can confirm. it's pretty hard to see why it happens, though if escapechar is set to 0 rather than the more sensible NOT_SET (uint32_t(-1)) that would certainly explain it. unset chars should definitely be using NOT_SET instead in all situations

[WillAyd]

This appears to be working on the tip of main

[eshmu]

Yep, works with Python 3.11, it looks like. Closing.

[mumbleskates]

this is not fixed in 3.10.7, and i feel that it should probably be fixed in that branch as it is a clear regression. is 3.10 going into security-only maintenance with this bug still extant?

[merwok]

If this is a clear bug (behaviour contradicts documentation) and you can post a reproducer, it could be fixed unless that would require too many changes or adding new parameters.

[mumbleskates]

@merwok

Python 3.10.7 (main, Nov  2 2022, 18:49:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from csv import writer
>>> from io import StringIO
>>> writer(StringIO()).writerow(["\0"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_csv.Error: need to escape, but no escapechar set
>>> 

tip of main:

cpython/Modules/_csv.c

Line 490 in a309ad9

DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, NOT_SET);

tip of 3.10 (through 3.10.8):

cpython/Modules/_csv.c

Line 484 in e7781fd

DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, 0);

the fallback value should be NOT_SET, not 0. this was fixed in 3.11 here: b454e8e#diff-38fcce6bb475616052f5c9a0973eefd49489a4dff719f30e407534258e2a3ec3R487

[merwok]

From the way the github PR was titled («Add support for») and the discussion on the bugs.python.org ticket (mention of backward compat issue with docutils), my understanding is that this change was deemed a new feature, not a bug fix, and as such will not be backported.

[mumbleskates]

Commit messages notwithstanding, this is a clear regression, as was observed in the original post of this issue:

Python 3.9.9 (main, Nov  7 2022, 21:30:33) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> import io
>>> csv.writer(io.StringIO()).writerow(["\0"])
5
>>> 

Python 3.8.12 (default, Nov  7 2022, 21:31:36) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> import io
>>> csv.writer(io.StringIO()).writerow(["\0"])
5
>>> 

Python 3.7.12 (default, Nov  7 2022, 21:34:47) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> import io
>>> csv.writer(io.StringIO()).writerow(["\0"])
5
>>> 

Python 3.7.0 (default, Nov  7 2022, 21:59:14) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> import io
>>> csv.writer(io.StringIO()).writerow(["\0"])
5
>>>

Python 3.6.15 (default, Nov  7 2022, 22:06:34) 
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv
>>> import io
>>> csv.writer(io.StringIO()).writerow(["\0"])
5
>>>

All the prior versions of python3 I can get my hands on at the moment write b'"\0"\r\n', and python3.11 writes b"\0\r\n". Both of these options are permissible and coherent, but python3.10 fails by reporting that an escape character is required when escaping is not needed: the only characters that get escaped in python's conception of CSV are the escape character and the quote character, and python's dialects use double-quote mode to use the quote character to escape itself.

[serhiy-storchaka]

What is the result of writing strings like '"\0' or '\n\0' in different Python versions?

[mumbleskates]

with a script like

import csv
import io
b = io.StringIO()
csv.writer(b).writerow(["\0", "\"\0", "\n\0", "\0\r\n"])
print(repr(b.getvalue()))

the results are:

3.6.15
'"\x00","""\x00","\n\x00","\x00\r\n"\r\n'

3.7.0
'"\x00","""\x00","\n\x00","\x00\r\n"\r\n'

3.7.12
'"\x00","""\x00","\n\x00","\x00\r\n"\r\n'

3.8.12
'"\x00","""\x00","\n\x00","\x00\r\n"\r\n'

3.9.9
'"\x00","""\x00","\n\x00","\x00\r\n"\r\n'

3.10.1
Traceback (most recent call last):
  File "/home/widders/csvtest.py", line 7, in <module>
    csv.writer(b).writerow(["\0", "\"\0", "\n\0", "\0\r\n"])
_csv.Error: need to escape, but no escapechar set

3.11.0
'\x00,"""\x00","\n\x00","\x00\r\n"\r\n'