streamCSV

v0.0.2

Read a CSV or TSV file line-by-line, applying headers to each row and guessing the correct type for each entry. Convenient for very large files.

Installation

npm install node-stream-csv

How to use

streamCSV takes three arguments: A filename, a callback for each row, and a callback when it's done. The last is optional.

Examples

These are all examples from test/test.js

var streamCSV = require("node-stream-csv");

streamCSV("state_population.tsv", function(state) {
    console.log(state);
});

/* Input

SUMLEV  REGION  DIVISION    STATE   NAME    POPESTIMATE2013 POPEST18PLUS2013    PCNT_POPEST18PLUS
10  0   0   0   United States   316128839   242542967   76.7
40  3   6   1   Alabama 4833722 3722241 77
40  4   9   2   Alaska  735132  547000  74.4
*/

/* Output:
{ SUMLEV: 10,
  REGION: 0,
  DIVISION: 0,
  STATE: 0,
  NAME: 'United States',
  POPESTIMATE2013: 316128839,
  POPEST18PLUS2013: 242542967,
  PCNT_POPEST18PLUS: 76.7 }
{ SUMLEV: 40,
  REGION: 3,
  DIVISION: 6,
  STATE: 1,
  NAME: 'Alabama',
  POPESTIMATE2013: 4833722,
  POPEST18PLUS2013: 3722241,
  PCNT_POPEST18PLUS: 77 }
{ SUMLEV: 40,
  REGION: 4,
  DIVISION: 9,
  STATE: 2,
  NAME: 'Alaska',
  POPESTIMATE2013: 735132,
  POPEST18PLUS2013: 547000,
  PCNT_POPEST18PLUS: 74.4 }   
*/

Note that the module correctly interpreted the numbers as numbers. If you don't want this behavior, such as when you've got leading zeroes you want to preserve, pass an options object to the first argument with dontguess:

streamCSV(
    {
        filename: "county_population.csv",
        dontguess: true
    },
    function(datum, c) {
        process.stdout.write("Read " + c + " lines from county file.\r");
    },
    function() { // on close 
        cb();
    }       
);

/* Output 
{ SUMLEV: '050',
  REGION: '3',
  DIVISION: '6',
  STATE: '01',
  COUNTY: '001',
  STNAME: 'Alabama',
  CTYNAME: 'Autauga County',
  CENSUS2010POP: '54571'
  // etc etc
}

Options

• dontguess: No type guessing, just import every data point as a string. You can also pass a comma-separated list of fields not to guess on

• noheader: The first line is a regular line of data, not a set of headers. Just give me back arrays of each line.

• sparse: If there is a header but the field value is blank, don't include it in the object for that row

• mode: "csv" or "tsv". If unspecified, guesses from file extension.

Dependencies

• This module uses d3's csv parser on each line.

• It uses a fork of maleck13's readline module to convert the filestream into line-by-line events. The fork allows for the fact that most readline modules, including the core module, often miss the last line of the file. The fork is distributed in this repo without the git files.