csv writer doesn't escape escapechar

[ebreck]

BPO	12178
Nosy	@taleinat, @merwok, @berkerpeksag, @xflr6, @avatarofhope2
PRs	bpo-12178: Fix escaping of escapechar in csv.writer() #13710
Files	pybug.zip 0eb420ce6567.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-09-20.06:40:59.124>
created_at = <Date 2011-05-25.18:27:10.453>
labels = ['extension-modules', 'type-bug', '3.10']
title = "csv writer doesn't escape escapechar"
updated_at = <Date 2021-07-13.19:26:06.249>
user = 'https://bugs.python.org/ebreck'

bugs.python.org fields:

activity = <Date 2021-07-13.19:26:06.249>
actor = 'xflr6'
assignee = 'none'
closed = True
closed_date = <Date 2020-09-20.06:40:59.124>
closer = 'taleinat'
components = ['Extension Modules']
creation = <Date 2011-05-25.18:27:10.453>
creator = 'ebreck'
dependencies = []
files = ['22112', '22596']
hgrepos = ['38']
issue_num = 12178
keywords = ['patch', 'needs review']
message_count = 13.0
messages = ['136881', '136973', '139953', '140002', '273746', '273784', '309815', '349720', '352447', '355118', '377206', '377207', '397440']
nosy_count = 11.0
nosy_names = ['taleinat', 'eric.araujo', 'catalin.iacob', 'ebreck', 'berker.peksag', 'xflr6', 'Apathymannequin', 'Dmitriy Shashkin', 'Alex Shpilkin', 'avatarofhope', 'Aldrian Obaja']
pr_nums = ['13710']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12178'
versions = ['Python 3.10']

[ebreck]

Consider the attached two files. A reader and writer with the same dialect parameters (escapechar \ quotechar " doublequote False) read, then write a CSV cell that looks like "C\\". It's written "C\". The problem is, when doublequote=False, the escapechar isn't used to escape itself, and the writer writes something that in the same dialect would be understood differently (\" isn't \ then end of string, it's an escaped quotechar within the string).

Execute python err.py first.csv to see.

[merwok]

Thanks for the report. It would be best if you could attach files as plain text instead of archives.

[cataliniacob]

I looked at this and tried to provide a patch + tests. Please review.

The bug is that a writer can use writerow on some input data but if a reader with the same dialect reads them back they are different from the input ones. This happens when the input data contains escapechar. Contrary to msg136881, this happens regardless whether doublequote is True or False.

The docs say "On reading, the escapechar removes any special meaning from the following character". Therefore, I understand that on writing, escapechar must always be escaped by itself. If that doesn't happen, when reading it back, escapechar alters the thing that follows it instead of counting as escapechar which is precisely what this bug is about.

[merwok]

Thanks for the patch. The tests look good at first glance. I can’t comment on the C code, I don’t know C. Hopefully someone will do it, otherwise if you don’t get feedback in say four weeks you can ask for a review on python-dev.

[Apathymannequin]

Hi,

I can confirm that this behavior still exists in my current python versions (3.5.2 & 2.7.11). I'm happy to take a look at the code, but considering I made this account specifically to comment on this issue I assume someone else will want to, as well.

If you want to make sure this is still broken, just use any of the docs' reader and writer examples, adding a trailing escape char to the end.

[smontanaro]

The patch looked okay to me, and when applied to the 2.7 source, the new tests pass muster.

I'm not going to pretend I know where this patch should be applied. That's for someone else to pronounce.

[xflr6]

Hi, is there something we can do to get this going? As the issue breaks round-trip, it currently requires work-arounds like this: https://github.com/cldf/csvw/blob/1324550266c821ef32d1e79c124191e93aefbfa8/csvw/dsv.py#L67-L71

[avatarofhope2]

I needed this patch for a project, so I compiled python with the patch applied, and tested my specific use case, and it worked. Thanks for the patch!

[taleinat]

FWIW, though this is arguably fixing a bug, IMO this shouldn't be back-ported to 2.7 or 3.7, but only be in 3.8+, to avoid breaking backwards-compatibility.

That being said, it would be great to get this into 3.8.0!

[AldrianObaja]

I think this issue needs to be escalated, as this is clearly a bug that makes it troublesome to use csv.reader and csv.writer with escapechar.

[taleinat]

New changeset 5c0eed7 by Berker Peksag in branch 'master':
bpo-12178: Fix escaping of escapechar in csv.writer() (GH-13710)
5c0eed7