NodeJS streams callback called multiple times

[satishrao84]

• Node.js Version: v10.8.0
• OS: MacOS
• Scope (install, code, runtime, meta, other?): code
• Module (and version) (if relevant): streams

Hello,

The piping of multiple streams works fine in my code as below

stream.on('data',function(data){}).pipe(data_work1).pipe(data_work2).pipe(data_work3).pipe(insertDataIntoDB);

I am trying to unwind an array in a JSON doc. If my source data has array like below:

{
"comp":"company1",
  "campaigns": [
     {
      "name":"yoohoo",
      "age":"43"
     },
     {
      "name":"yolo",
      "age":"21"
     }
   ]
}

If I want to break up the above document into smaller parts

{
"comp":"company1",
"name":"yoohoo",
"age":"43"

}

and

{
"comp":"company1",
"name":"yolo",
"age":"21"

}

		for (var i = 0; i <  campaigns.length; i++){
			
			var doc_insert = data;
			
			doc_insert.campaigns = campaigns[i]
			this.transform.push(doc_insert);
			cb()
		}
		cb()

The problem is I get the below error:

Error [ERR_MULTIPLE_CALLBACK]: Callback called multiple times
at Transform.afterTransform (_stream_transform.js:85:31)
at Object.p16 (/Users/sanupin/Documents/workspace/MongoMigration/util/schema_add.js:360:5)
at Transform.getSchemaAdditions._transform (/Users/sanupin/Documents/workspace/MongoMigration/util/MigrationStream.js:82:24)
at Transform._read (_stream_transform.js:190:10)
at Transform._write (_stream_transform.js:178:12)
at doWrite (_stream_writable.js:410:12)
at writeOrBuffer (_stream_writable.js:394:5)
at Transform.Writable.write (_stream_writable.js:294:11)
at Transform.ondata (_stream_readable.js:663:20)
at Transform.emit (events.js:182:13)
at addChunk (_stream_readable.js:283:12)
at readableAddChunk (_stream_readable.js:264:11)
at Transform.Readable.push (_stream_readable.js:219:10)
at Transform.push (_stream_transform.js:151:32)
at Transform.rulesCheck._transform (/Users/sanupin/Documents/workspace/MongoMigration/util/MigrationStream.js:58:14)
at Transform._read (_stream_transform.js:190:10)
error: worker caught err: Error [ERR_MULTIPLE_CALLBACK]: Callback called multiple times and code: uncaughtException

How can I solve this? That is, How can I send each array element to the next transform stream without invoking the callback multiple times?

[wraugh]

Hi @satishrao84, the cb() function should only be called once on every invocation of your transform handler. It means "I'm done transforming"; no other code shoud run after it.

So to fix your example, remove the cb() in every loop:

		for (var i = 0; i <  campaigns.length; i++){
			
			var doc_insert = data;
			
			doc_insert.campaigns = campaigns[i]
			this.transform.push(doc_insert);
-			cb()
		}
		cb()

[gireeshpunathil]

@satishrao84 - does the solution provided by @wraugh works for you?

[EmiyaYang]

Hi @satishrao84, the cb() function should only be called once on every invocation of your transform handler. It means "I'm done transforming"; no other code shoud run after it.

So to fix your example, remove the cb() in every loop:

		for (var i = 0; i <  campaigns.length; i++){
			
			var doc_insert = data;
			
			doc_insert.campaigns = campaigns[i]
			this.transform.push(doc_insert);
-			cb()
		}
		cb()

Good, this works for me.

[PoojaDurgad]

@satishrao84 , were you able to follow @wraugh 's suggestion ?

[gireeshpunathil]

inactive, but resolution suggested, closing. feel free to reopen if still an issue