Skip to content
This repository has been archived by the owner on Feb 10, 2023. It is now read-only.

Commit

Permalink
Don't spawn too many consumers
Browse files Browse the repository at this point in the history
If len(to_process) < num_processes, some would not be able to get an
item from the queue, thereby blocking indefinitely. In that case, use
only as many consumers as necessary to process all the files.
  • Loading branch information
mineo committed Apr 16, 2016
1 parent 6059d8e commit 8fe1b37
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion abzer/abzer.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ async def run(self):
if to_process:
tasks = []
tasks.append(self.producer(to_process))
for i in range(0, self.num_processes):
num_tasks = min(len(to_process), self.num_processes)
for i in range(0, num_tasks):
tasks.append(self.consumer())
tasks.append(self.queue.join())
await asyncio.gather(*tasks)
Expand Down

0 comments on commit 8fe1b37

Please sign in to comment.