es.merge broke my data

[ghost]

I have 12 files to handle, so I want to create 12 ReadStreams and merge to one.

In each file, each line is a JSON string.

My code is like this:

  fileNames = fileNames.filter(Boolean);
  es.merge(fileNames.map(function(item, index) {
               return fs.createReadStream(item);
             }))
    .pipe(es.split())
    .pipe(es.parse())

After run it, I got some JSON.parse error like this:

SyntaxError: Unexpected token

and I found the error json looks like this:

{"ts":1452700832306,"segments":{"sex":1,"age":5,"os":1,"deviceprice":5},"appkey":"560129b88bda20a5270f88da","token":"jXL29pROAnEGng","deviceid":"D4FB58CD-39{"ts":1452700810077,"appkey":"560129b88bda20a5270f88da","token":"jXL29pROAnEGng","deviceid":"B8DA0F64C613E896B412788B9BDB06D4"}

I grep 1452700832306 and 1452700810077, and find they come from two different files:

in file1, test1.log:

{"ts":1452700832306,"segments":{"sex":1,"age":5,"os":1,"deviceprice":5},"appkey":"560129b88bda20a5270f88da","token":"jXL29pROAnEGng","deviceid":"D4FB58CD-39CF-4445-AD26-C85194C0D032"}

in file2, test2.log:

{"ts":1452700810077,"appkey":"560129b88bda20a5270f88da","token":"jXL29pROAnEGng","deviceid":"B8DA0F64C613E896B412788B9BDB06D4"}

So it looks like some of my data has been broken.

Node version: v4.2.3
OS: CentOS Linux release 7.0.1406 (Core)

[dominictarr]

merge doesn't know where the lines in your flies are. when fs reads from the disk, it may break them up anywhere. you need the splits to be at the lines so that es.parse can handle them.

this should work:

  es.merge(fileNames.map(function(item, index) {
               return fs.createReadStream(item)
                  .pipe(es.split())
                  .pipe(es.parse())
             }))

[ghost]

Thank you @dominictarr, you are right! I need to learn about how stream works.