Add named tuple reader to CSV module

[rhettinger]

BPO	1818
Nosy	@smontanaro, @warsaw, @rhettinger, @pitrou, @merwok, @cedk, @asvetlov, @dlenski
Files	ntreader.diff: Proof-of-concept patch ntreader3.diff: namedtuple reader and writer. ntreader4.diff: Includes revision for rename keyword argument named_tuple_write_header2.patch ntreader4_py3_1.diff: Patch against python 3.1a1 ntreader6_py3.diff: updated documentation ntreader6_py27.diff: updated documentation 1818_py35.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/smontanaro'
closed_at = <Date 2020-12-22.02:45:54.546>
created_at = <Date 2008-01-13.22:27:14.884>
labels = ['3.8', 'type-feature', 'library']
title = 'Add named tuple reader to CSV module'
updated_at = <Date 2020-12-22.02:45:54.545>
user = 'https://github.com/rhettinger'

bugs.python.org fields:

activity = <Date 2020-12-22.02:45:54.545>
actor = 'rhettinger'
assignee = 'skip.montanaro'
closed = True
closed_date = <Date 2020-12-22.02:45:54.546>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2008-01-13.22:27:14.884>
creator = 'rhettinger'
dependencies = []
files = ['9151', '12990', '13009', '13188', '13263', '13274', '13275', '39139']
hgrepos = []
issue_num = 1818
keywords = ['patch']
message_count = 42.0
messages = ['59866', '61523', '61532', '81453', '81464', '81518', '81537', '82744', '82745', '82746', '82764', '82765', '82770', '82771', '82778', '82780', '82798', '82799', '82812', '82814', '82819', '83298', '83299', '83310', '83318', '83321', '83332', '83333', '83334', '83340', '102936', '102959', '110598', '111523', '111552', '115348', '235710', '241599', '241601', '242879', '242893', '311182']
nosy_count = 11.0
nosy_names = ['skip.montanaro', 'barry', 'rhettinger', 'pitrou', 'eric.araujo', 'ced', 'jdwhitley', 'rrenaud', 'asvetlov', 'dlenski', 'copper-head']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1818'
versions = ['Python 3.8']

[rhettinger]

Here's a proof-of-concept patch. If approved, will change from
generator form to match the other readers and will add a test suite.

The idea corresponds to what is currently done by the dict reader but
returns a space and time efficient named tuple instead of a dict. Field
order is preserved and named attribute access is supported.

A writer is not needed because named tuples can be feed into the
existing writer just like regular tuples.

[rhettinger]

Barry, any thoughts on this?

[smontanaro]

I'd personally be kind of surprised if Barry had any thoughts on this.
Is there any reason this couldn't be pushed down into the C code and
replace the normal tuple output completely? In the absence of any
fieldnames you could just dream some up, like "field001", "field002",
etc.

Skip

[jdwhitley]

An implementation of a namedtuple reader and writer.

Created a writer for the case where user would like to specify
desired field names and default values on missing field names.

e.g.
mywriter = NamedTupleWriter(f, fieldnames=['f1', 'f2', 'f3'],
restval='missing')

Nt = namedtuple('LessFields', 'f1 f3')
nt = Nt(f1='one', f2=2)

mywriter.writerow(nt) # writes one,missing,2

any thoughts on case where defined fieldname has a leading
underscore? Should there be a flag to silently ignore?

e.g.
if self._ignore_underscores:
fieldname = fieldname.lstrip('_')

Leading underscores may be present in an unsighted csv file,
additionally, spaces and other non alpha numeric characters pose
a problem that does not affect the DictReader class.

Cheers,

[rhettinger]

Consider providing a hook to a function that converts non-conforming
field names (ones with a leading underscore, leading digit, non-letter,
keyword, or duplicate name).

class NamedTupleReader:
    def __init__(self, f, fieldnames=None, restkey=None, restval=None,
                 dialect="excel", fieldnamer=None, *args, **kwds):
                 . . .

I'm going to either post a recipe to do the renaming or provide a static
method for the same purpose. It might work like this:

  >>> renamer(['abc', 'def', '1', '_hidden', 'abc', 'p', 'abc'])
  ['abc', 'x_def', 'x_1', 'x_hidden', 'x_abc', 'p', 'x1_abc']

[rhettinger]

In r69480, named tuples gained the ability to automatically rename
invalid fieldnames.

[jdwhitley]

Updated NamedTupleReader to give a rename=False keyword argument.
rename is passed directly to the namedtuple factory function to enable
automatic handling of invalid fieldnames.

Two new tests for the rename keyword.

Cheers,

[rrenaud]

I am totally new to Python dev. I reinvented a NamedTupleReader
tonight, only to find out that it was created a year ago. My primary
motivation is that DictReader reads headers nicely, but DictWriter
totally sucks at handling them.

Consider doing some filtering on a csv file, like so.

sample_data = [
    'title,latitude,longitude',
    'OHO Ofner & Hammecke Reinigungsgesellschaft mbH,48.128265,11.610848',
    'Kitchen Kaboodle,45.544241,-122.715728',
    'Walgreens,28.339727,-81.596367',
    'Gurnigel Pass,46.731944,7.447778'
    ]

def filter_with_dict_reader_writer():
  accepted_rows = []
  for row in csv.DictReader(sample_data):
    if float(row['latitude']) > 0.0 and float(row['longitude']) > 0.0:
      accepted_rows.append(row)

  field_names = csv.reader(sample_data).next()
  output_writer = csv.DictWriter(open('accepted_by_dict.csv', 'w'),
                                 field_names)
  output_writer.writerow(dict(zip(field_names, field_names)))
  output_writer.writerows(accepted_rows)

You have to work so hard to maintain the headers when you write the file
with DictWriter. I understand this is a limitation of dicts throwing
away the order information. But namedtuples don't have that problem.

NamedTupleReader and NamedTupleWriter should be inverses. This means
that NamedTupleWriter needs to write headers. This should produce
identical output as the dict writer example, but it's much cleaner.

def filter_with_named_tuple_reader_writer():
   accepted_rows = []
   for row in csv.NamedTupleReader(sample_data):
     if float(row.latitude) > 0.0 and float(row.longitude) > 0.0:
       accepted_rows.append(row)

   output_writer = csv.NamedTupleWriter(
       open('accepted_by_named_tuple.csv', 'w'))
   output_writer.writerows(accepted_rows)

I patched on top of the existing NamedTupleWriter patch adding support
for writing headers. I don't know if that's bad style/etiquette, etc.

[rrenaud]

My previous patch could write the header twice. But I am not sure about
about how the writer should handle the fieldnames parameter on one hand,
and the namedtuple._fields on the other.

[rhettinger]

The two latest patches (ntreader4.diff and
named_tuple_write_header.patch) seem like they are going in the right
direction and are getting close.

Barry or Skip, is this something you want in your module?

[smontanaro]

Raymond> Barry or Skip, is this something you want in your module?

Sorry, I haven't really looked at this ticket other than to notice its
presence. I wrote the DictReader/DictWriter functions way back when, so I'm
pretty comfortable using them. I haven't felt the need for any other reader
or writer which manipulates file headers.

Skip

[warsaw]

I think it would be useful to have.