Bundle gather calls in as_completed(..., with_results)

[mrocklin]

Currently as_completed(..., with_results=True) can generate many gather calls, which can overwhelm the scheduler a bit. We might instead batch these gather calls if many come in before other gather calls return. This would also help to avoid creating very many file descriptors.

Alternativley this could also be handled at a lower level in gather itself.

[shenker]

Just chiming in that this is a big blocker for me because I keep running into file descriptor limits (was about to raise an issue myself!). Can test this at scale if that's helpful (my typical use case is 300k tasks x 10s each, 250 workers, 1000 cpus; my sysadmin only gives me ulimit -n -H of 4096).

I'd vote for queueing+batching all calls to gather rather than just batching as_completed. If you have multiple streaming pipelines submitting tasks and gathering them back, it'd be difficult/impossible to make sure that all these separate gather-calling operations stay under a global connection limit.