Skip to content

Commit

Permalink
Fixed problem shutting down threaded doc mapper
Browse files Browse the repository at this point in the history
Synchronization problem with queues. The input reader queue is getting
shut down while the processor is waiting for a document, which was
causing an exception to be raised in the worker process while shutting
down.

This is not a problem: we just have to catch the exception and ignore
it. The stopped flag is set by this point, so we let the process exit
naturally when this happens.
  • Loading branch information
markgw committed Aug 6, 2020
1 parent 097d4f3 commit f6c1544
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions src/python/pimlico/core/modules/map/threaded.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ def run(self):
except Empty:
# Don't worry if the queue is empty: just keep waiting for more until we're shut down
pass
except IOError as e:
if e.args[0] == "handle is closed":
# The queue was closed while we were waiting
# Stopped should have been set by now: we continue and check that
continue
raise
else:
input_buffer.append(tuple([archive, filename] + docs))
if len(input_buffer) >= self.docs_per_batch or self.no_more_inputs.is_set():
Expand Down

0 comments on commit f6c1544

Please sign in to comment.