Skip to content

Commit

Permalink
Fixed critical bug in input feeding
Browse files Browse the repository at this point in the history
Big introduced by 8a8143c, meaning that the doc map processor only
processed half of the documents!
  • Loading branch information
markgw committed Oct 7, 2020
1 parent 899587f commit b291d35
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion src/python/pimlico/core/modules/map/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -618,9 +618,10 @@ def run(self):
if self.record_invalid:
if any(is_invalid_doc(doc) for doc in docs):
self.invalid_docs.put((archive, filename))

batch.append((archive, filename, docs))
if len(batch) < self.feeder_batch_size:
# Don't send this batch yet: get some more documents
batch.append((archive, filename, docs))
continue
# If the queue is full, this will block until there's room to put the next one on
# It also blocks if the queue is closed/destroyed/something similar, so we need to check now and
Expand Down

0 comments on commit b291d35

Please sign in to comment.