Wait to call error callback until currently running parallel tasks complete

[danieldickison]

Use case: I have some parallel tasks that writes files to a temporary directory, and I want a cleanup function to delete the temporary directory after all tasks complete. If one task fails, I'd like to delete the directory immediately. async.parallel fits the bill perfectly, except that I need to wait to delete the directory until all currently running parallel tasks complete, lest the other running tasks try to write files to a nonexistent directory.

I imagine this is a pretty common scenario, and it's rather tricky to do without resorting to running the tasks serially. (Perhaps there's an easy way to do it that I haven't thought of -- would love to hear that.)

It would be useful if, for any parallel tasks, async waited to call the callback function in the case of errors until all concurrently running tasks complete. That is, once a task calls its callback with an error, async should stop starting new tasks and run the final callback once the number of running tasks reaches zero.

[Flackus]

Thumbs up for this.

One more scenario: we have a parallel execution of several backend requests via http. When writing this queue I already know, that data from these requests isn't mandatory for the application in common. An app would, of course, gracefully degrade, but will still work.
So, when getting timeout errors (or any other), I still want all other tasks to complete, and, at the same time, I should still be able to log errors in one common callback.

Current option is to handle such errors in methods, that do such requests, and then pass null as the first argument for the callback. It isn't very convinient.

It would be very nice, if I could pass an option to the async to tell it, that I want all tasks to finish even if there were errors.

[Flackus]

Anyone?

[ifeltsweet]

Really want this feature for async.series, async.foreach and async.auto too!

[shaoshuai0102]

I want this too. A callback should be called after all tasks are completed, even if any of them failed.

[jason-patel-ensighten]

Plus one as well on this issue, our use case:

Need to try and get information from x amount of different sources, all independent, perfect for parallel, except if one task errors the main call back is called, but after the other x-1 tasks finish the main callback is not called. I would still like to get the information i did get back but since the main call back is never called again after all tasks complete if an error has been thrown i can't get the information available.

If we can wait until all the tasks finish to call the main call back so err an array as well that would be perfect

[caolan]

Yes, certainly a valid use case. I'm going to work on an API that will do this and various other tweaks for all functions in the near future.

[bitwiseman]

I'm going to guess you didn't get to this? Still worth doing?

[alizbazar]

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a temporary error. The following address(es) deferred:

alizbazar@gmail.com
Domain alizweb.com has exceeded the max emails per hour (140/120 (116%)) allowed. Message will be reattempted later

------- This is a copy of the message, including all the headers. ------
Received: from github-smtp2-ext8.iad.github.net ([192.30.252.199]:45132 helo=github-smtp2b-ext-cp1-prd.iad.github.net)
by web107.webhotelli.fi with esmtps (TLSv1:DHE-RSA-AES256-SHA:256)
(Exim 4.82)
(envelope-from noreply@github.com)
id 1WTaNI-0003bw-70
for alizbazar@alizweb.com; Fri, 28 Mar 2014 19:15:25 +0200
Date: Fri, 28 Mar 2014 10:15:23 -0700
From: Caolan McMahon notifications@github.com
Reply-To: caolan/async reply@reply.github.com
To: caolan/async async@noreply.github.com
Message-ID: caolan/async/issue/51/issue_event/106554092@github.com
In-Reply-To: caolan/async/issues/51@github.com
References: caolan/async/issues/51@github.com
Subject: Re: [async] Wait to call error callback until currently running
parallel tasks complete (#51)
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="--==_mimepart_5335ae2b48ba_24333fd1fcd332b418146a";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: list
X-GitHub-Recipient: alizbazar
List-ID: caolan/async <async.caolan.github.com>
List-Archive: https://github.com/caolan/async
List-Post: mailto:reply@reply.github.com
List-Unsubscribe: mailto:unsub+i-1593271-bf6ff6ffe838fbf3386711c19738396b40db97c1-736771@reply.github.com,
https://github.com/notifications/unsubscribe/736771__eyJzY29wZSI6Ik5ld3NpZXM6TXV0ZSIsImV4cGlyZXMiOjE3MTE2NDYxMjIsImRhdGEiOnsiaWQiOjEzOTQ4OTF9fQ==--738aff229356c20c115edbcf60f7c74a798919c0
X-Auto-Response-Suppress: All
X-GitHub-Recipient-Address: alizbazar@alizweb.com

----==_mimepart_5335ae2b48ba_24333fd1fcd332b418146a
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 7bit

Closed #51.

---

Reply to this email directly or view it on GitHub:
#51
----==_mimepart_5335ae2b48ba_24333fd1fcd332b418146a
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit

Closed #51.

—
Reply to this email directly or view it on GitHub.

----==_mimepart_5335ae2b48ba_24333fd1fcd332b418146a--

[jfstgermain]

This default behavior has caused us headaches as well. This feature request was closed, is it in the pipeline or is it just discarded?

[aearly]

async.queue is probably a better fit if you need an early exit. If you get an error in the callback to queue.push() you can call queue.pause().

[caolan]

I won't be changing the default behaviour of async.parallel any time soon - but I understand why this would be needed. Open to other suggestions.