Best pattern to prepend and append element to a series while streaming?

[giacecco]

I am using event-stream and request to convert a >2Gb file of records from one JSON 'schema' to another, to bulk upload to CouchDB. The code looks like:

s.createReadStream(inFile, {flags: 'r'})
    .pipe(es.split('\n'))
    .pipe(es.parse())
    .pipe(es.map(function (data, callback) {
            // transform the thing here     
            callback(null, data);       
    }))
    .pipe(es.stringify())
    .pipe(es.join(','))
    .pipe(request({ 
        'url': 'http://localhost:5984/dbname/_bulk_docs',
        'method': 'POST',
        'json': true,
    }, function (err, response, body) { 
        console.log(body);
    }));

The problem is that for the above to work I need to wrap all output in another JSON so that every original item becomes a record in an array called 'docs'. In other words, I need to prepend { "docs": [ and to append ] }, but I have no idea of how to do that. I know I could load the whole file in memory but it is not a good practice for something this big.

How can I achieve that with event-stream? Thanks,

Giacecco

[dominictarr]

It would be easy with through. just make a through stream that emits all the input, but emit one special thing first and last. you may also want to look at the code for JSONStream.stringifyJSONStream

[giacecco]

Thanks, I have solved the problem this way instead:

var out = fs.createWriteStream(argv.in + '.new');
out.write('{ "data":[', 'utf8', function (err) {
    var inStream = s.createReadStream(inFile, {flags: 'r'})
            .pipe(es.split('\n'))
            .pipe(es.parse())
            .pipe(es.map(function (data, callback) {
                    // transform the thing here     
                    callback(null, data);       
            }))
            .pipe(es.stringify())
            .pipe(es.join(','))
            .pipe(out);
    inStream.on('end', function() {
        out.write(']}', 'utf8', function (err) {
            out.end();
        });
    });
});

... but I can't get myself to like this. Then I bumped in other issues created by the size of the file and had to focus on other stuff :-(

G.

[dominictarr]

that works.