[TDW] Fixing up file and streaming

[afs]

Sec 3.3 (edd6183 : 2014-02-26)

To fix up CSV files in which different lines contain different numbers of values,
additional empty values should be added to the end of lines such that they all contain
the same number of values as the line with the most values.

To know the line with the most values, requires looking through the whole file. Fixes that depend on that means that you can not output any lines before checking for the longest possible row. It stops streaming.

A line longer than the header line, has no column names so again, you'd have to generate the extra columns before doing anything else.

Suggestion:

• pad short lines to the length of the header row with empty fields
• have the suggested API for processing have an "extra" return for lines found longer than the header row