csv DictWriter's internal _dict_to_list raise error unsupported operand

[schneewe]

BPO	38717
Nosy	@rhettinger, @serhiy-storchaka, @tirkarthi, @schneewe

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-11-06.11:05:04.462>
created_at = <Date 2019-11-06.10:04:48.006>
labels = ['3.7', 'type-bug', 'library']
title = "csv DictWriter's internal _dict_to_list raise error unsupported operand"
updated_at = <Date 2019-11-06.11:05:04.460>
user = 'https://github.com/schneewe'

bugs.python.org fields:

activity = <Date 2019-11-06.11:05:04.460>
actor = 'afflictor'
assignee = 'none'
closed = True
closed_date = <Date 2019-11-06.11:05:04.462>
closer = 'afflictor'
components = ['Library (Lib)']
creation = <Date 2019-11-06.10:04:48.006>
creator = 'afflictor'
dependencies = []
files = []
hgrepos = []
issue_num = 38717
keywords = []
message_count = 6.0
messages = ['356110', '356111', '356113', '356115', '356116', '356118']
nosy_count = 4.0
nosy_names = ['rhettinger', 'serhiy.storchaka', 'xtreak', 'afflictor']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38717'
versions = ['Python 3.7']

[schneewe]

I switched over to Python 3.7 and still using my csv DictWriter.

No I'm receiving an error when calling the writerow() method.
unsupported operand type(s) for -: 'list' and 'list'

Digging deeped in the problem I found out, that the internal _dict_to_list(self, rowdict) method works different compared to Python 2.7
It tries to substract the fieldnames list and the keys of the dict to find keys not included.

But substracting lists is not possible. As a workaround I defined the Dictwriter with the option extrasaction="ignore".

I would be better to use the method like in Python 2.7. I will output my versions of the method:

Python 3.7:
def _dict_to_list(self, rowdict):
if self.extrasaction == "raise":
wrong_fields = rowdict.keys() - self.fieldnames
if wrong_fields:
raise ValueError("dict contains fields not in fieldnames: "
+ ", ".join([repr(x) for x in wrong_fields]))
return (rowdict.get(key, self.restval) for key in self.fieldnames)

Python 2.7:
def _dict_to_list(self, rowdict):
if self.extrasaction == "raise":
wrong_fields = [k for k in rowdict if k not in self.fieldnames]
if wrong_fields:
raise ValueError("dict contains fields not in fieldnames: "
+ ", ".join([repr(x) for x in wrong_fields]))
return [rowdict.get(key, self.restval) for key in self.fieldnames]

[serhiy-storchaka]

I guess you try to write not a dict, but an object whose keys() method returns a list.

[tirkarthi]

Can you please attach a sample script to reproduce the behavior difference? With below script a dictionary is passed then dict.keys() supports subtraction over a list to return the difference. I am not sure how you have received a list with dict.keys() in Python 3.7.

import csv

fields = ["c", "d"]
with open('test.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=fields, extrasaction='raise')
    writer.writeheader()
    writer.writerow({"n": 1})

[schneewe]

Hello all,

This is my dict:
{'rule': 'WAN-2-Web|unknown (Barracuda Networks CloudGen Firewall)', 'April (2018)': '', 'May (2018)': '', 'June (2018)': '', 'July (2018)': '', 'August (2018)': '', 'April (2019)': '', 'May (2019)': '14', 'June (2019)': '', 'July (2019)': '', 'August (2019)': ''}

And this was my fieldnames list:
['rule', 'April (2018)', 'May (2018)', 'June (2018)', 'July (2018)', 'August (2018)', 'April (2019)', 'May (2019)', 'June (2019)', 'July (2019)', 'August (2019)']

when I call the keys Method on this dict, I receive this:
dict_keys(['rule', 'April (2018)', 'May (2018)', 'June (2018)', 'July (2018)', 'August (2018)', 'April (2019)', 'May (2019)', 'June (2019)', 'July (2019)', 'August (2019)'])

[schneewe]

Ok in a new sample Script it is working.

As I used my normal script in a software called Splunk, I saw they are using a custom class OrderedDict.

I will take a look on this first.

[schneewe]

Sorry guys, it was an issue with the cusotm Splunk Library. They treated the keys Method a little different.

Thanks all for the hints.

Will close this