csv module: add header row to DictWriter

[edabraham]

BPO	1537721
Nosy	@smontanaro, @pitrou, @djc, @florentx, @kynan

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/djc'
closed_at = <Date 2010-03-05.09:13:34.511>
created_at = <Date 2006-08-09.22:20:01.000>
labels = ['type-feature']
title = 'csv module: add header row to DictWriter'
updated_at = <Date 2010-08-29.14:14:47.795>
user = 'https://bugs.python.org/edabraham'

bugs.python.org fields:

activity = <Date 2010-08-29.14:14:47.795>
actor = 'kynan'
assignee = 'djc'
closed = True
closed_date = <Date 2010-03-05.09:13:34.511>
closer = 'djc'
components = ['None']
creation = <Date 2006-08-09.22:20:01.000>
creator = 'ed_abraham'
dependencies = []
files = []
hgrepos = []
issue_num = 1537721
keywords = ['patch', 'buildbot']
message_count = 17.0
messages = ['61251', '84796', '86157', '96063', '96069', '96074', '96078', '96089', '96091', '96102', '96103', '96104', '99952', '100404', '100407', '100409', '100455']
nosy_count = 7.0
nosy_names = ['skip.montanaro', 'pitrou', 'ed_abraham', 'djc', 'ivo', 'flox', 'kynan']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1537721'
versions = ['Python 2.7', 'Python 3.2']

[edabraham]

I use the DictWriter class from the csv module, and
have to manually write the header row. A mindless chore
which I would like to see eliminated. Can we have a
writeheader method added to the class? Something like
the following:

def writeheader(self, headernames = {}):
    """Write a header row"""
    if not headernames:
        headernames = dict(zip(self.fieldnames,
self.fieldnames))
        self.writerow(headernames)

This would let you either use the fieldnames directly,
or supply your own pretty header names.

Would be nice to have another keyword argument to
DictWriter, 'header = False'. If header was true, then
the __init__ method could call writeheader().

At the moment I have to write things like

fields = ['a','b','c']
w = csv.DictWriter(fid, fields)
w.writerow(dict(zip(fields, fields)))
for row in rows:
    w.writerow(row)

The proposed changes would let me write the simpler

w = csv.DictWriter(fid, ['a','b','c'], header = True)
for row in rows:
    w.writerow(row)

A problem is that including a new keyword argument
would break code which used position to fill the
keyword arguments and to supply arguments through *args
to the writer class.

[smontanaro]

I don't see a patch. Is there some reason that if you need this
you can't simply subclass DictWriter?

[ivo]

Skip, you were arguing in another csv issue on a NamedTupleReader that
the Reader and Writer should work in concert together.

Certainly, making this default functionality for DictWriter would
definitely make it work more in concert with DictReader.

A sample process of reading and writing might be:

1. DictReader reads in header names, reads in data
2. Pass in header names to DictWriter on init
3. DictWriter writes out modified data back to csv file (headers get
   written automatically or with a method call)
4. DictReader can now reader in the csv file automatically again with
   header names

My feeling is, if DictReader can read in head names, why can't
DictWriter write them back out again? Shouldn't there be a good amount
of symmatry to their function/abilities?

My feeling of how to implement this functionality would be to include a
new init argument, say 'writeheader'. It's default would be True if you
wanted to implement this new feature by default, or False if you wanted
to keep it an option so that older scripts relying on the old
functionality won't break immmediately.
DictWriter would then write the header on the first call to writerow(s),
or perhaps also with an explicit call to writeheader() say.

I certainly am miffed by the fact that DictWriter cannot produce
(normally) a csv file that DictReader can read in automatically.

[djc]

I'd like to commit this, but it would be nice to get a review first:

Index: Lib/csv.py
===================================================================

--- Lib/csv.py	(revision 76697)
+++ Lib/csv.py	(working copy)
@@ -132,6 +132,10 @@
         self.extrasaction = extrasaction
         self.writer = writer(f, dialect, *args, **kwds)
 
+    def writeheader(self):
+        header = dict(zip(self.fieldnames, self.fieldnames))
+        self.writerow(header)
+
     def _dict_to_list(self, rowdict):
         if self.extrasaction == "raise":
             wrong_fields = [k for k in rowdict if k not in self.fieldnames]
Index: Lib/test/test_csv.py
===================================================================
--- Lib/test/test_csv.py	(revision 76697)
+++ Lib/test/test_csv.py	(working copy)
@@ -598,8 +598,10 @@
         fileobj = os.fdopen(fd, "w+b")
         try:
             writer = csv.DictWriter(fileobj, fieldnames = ["f1", "f2",
"f3"])
+            writer.writeheader()
             writer.writerow({"f1": 10, "f3": "abc"})
             fileobj.seek(0)
+            self.assertEqual(fileobj.readline(), "f1,f2,f3\r\n")
             self.assertEqual(fileobj.read(), "10,,abc\r\n")
         finally:
             fileobj.close()

(I think I have commit privileges already.)

[smontanaro]

I'm sorry, but I don't have time to look at this right now. On the one
hand, one person asks for more symmetry. Someone else wants to add a
writeheader method. If you want symmetry shouldn't the DictWriter
simply write the header without being asked? I'm confused by the
various options and feel that someone is going to be disappointed by any
solution. For the time being at least I would prefer that the status
quo remain in place.

S

[djc]

Skip, I agree that it's hard to decide if we should have the class write
the header on __init__(). I figured starting off with a method to make
doing it "manually" is a good start; people can start using that, and if
it's deemed useful we can always add the auto-write later.

[pitrou]

We can't change default behaviour because it will break compatibility,
so an additional method looks ok to me.

[smontanaro]

Antoine> We can't change default behaviour because it will break
Antoine> compatibility, so an additional method looks ok to me.

Why can't default behavior be changed?

S

[pitrou]

Le lundi 07 décembre 2009 à 23:42 +0000, Skip Montanaro a écrit :

Skip Montanaro <skip@pobox.com> added the comment:

Antoine> We can't change default behaviour because it will break
Antoine> compatibility, so an additional method looks ok to me.

Why can't default behavior be changed?

Well, because it will break assumptions about the generated documents?

[smontanaro]

Antoine> We can't change default behaviour because it will break
Antoine> compatibility, so an additional method looks ok to me.

>> Why can't default behavior be changed?

Antoine> Well, because it will break assumptions about the generated documents?

Isn't the alpha period (2.7 and 3.2 in this case) precisely when an API can
change?

Skip

[pitrou]

Isn't the alpha period (2.7 and 3.2 in this case) precisely when an API can
change?

Well, it can, but only if there are compelling reasons to do so. It
should be the exception rather than the rule.
The reasons here seem far from compelling, and moreover we can't detect
whether the user is expecting the new or the old behaviour.

[smontanaro]

> Isn't the alpha period (2.7 and 3.2 in this case) precisely when an
>> API can change?

Antoine> Well, it can, but only if there are compelling reasons to do
Antoine> so. It should be the exception rather than the rule.  The
Antoine> reasons here seem far from compelling, and moreover we can't
Antoine> detect whether the user is expecting the new or the old
Antoine> behaviour.

Fine. Dirkjan, assuming there are the necessary test cases and
documentation changes, feel free to check in your patch and close the
ticket.

Skip

[djc]

Fixed in SVN, r78384.

[florentx]

according to the buidbots, it hurts some platforms: Windows XP, Windows 7 and sparc Solaris10

• sparc solaris10

test_writerows (test.test_csv.Test_Csv) ... ok

======================================================================
FAIL: test_write_simple_dict (test.test_csv.TestDictFields)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home2/buildbot/slave/trunk.loewis-sun/build/Lib/test/test_csv.py", line 607, in test_write_simple_dict
    self.assertEqual(fileobj.read(), "10,,abc\r\n")
AssertionError: 'f1,f2,f3\r\n' != '10,,abc\r\n'

---

Ran 84 tests in 17.335s

FAILED (failures=1)
test test_csv failed -- Traceback (most recent call last):
  File "/home2/buildbot/slave/trunk.loewis-sun/build/Lib/test/test_csv.py", line 607, in test_write_simple_dict
    self.assertEqual(fileobj.read(), "10,,abc\r\n")
AssertionError: 'f1,f2,f3\r\n' != '10,,abc\r\n'

• x86 XP and x86 Windows7

test_csv
test test_csv failed -- Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\trunk.bolen-windows\build\lib\test\test_csv.py", line 604, in test_write_simple_dict
    writer.writerow({"f1": 10, "f3": "abc"})
  File "D:\cygwin\home\db3l\buildarea\trunk.bolen-windows\build\lib\csv.py", line 148, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
IOError: [Errno 0] Error

[djc]

Testing on Windows with this:

Index: Lib/test/test_csv.py
===================================================================

--- Lib/test/test_csv.py	(revision 78430)
+++ Lib/test/test_csv.py	(working copy)
@@ -9,6 +9,7 @@
 import tempfile
 import csv
 import gc
+import io
 from test import test_support
 
 class Test_Csv(unittest.TestCase):
@@ -595,7 +596,7 @@
     ### "short" means there are fewer elements in the row than fieldnames
     def test_write_simple_dict(self):
         fd, name = tempfile.mkstemp()
-        fileobj = os.fdopen(fd, "w+b")
+        fileobj = io.open(fd, 'w+b')
         try:
             writer = csv.DictWriter(fileobj, fieldnames = ["f1", "f2", "f3"])
             writer.writeheader()

[djc]

Committed in r78660 after positive comment from briancurtin re Windows. Hopefully this fixes Solaris, as well.

[djc]

Both the solaris and windows slaves seem to have succeeded this time.