Skip to content
Permalink
Browse files

process_parallel was leaking the failures on its internal deferreds. c…

…loses #458

DeferredList implemented cancellation in Twisted 13.2.0 by holding a
reference to the affected deferreds objects, if a deferred errored the
result was propagated to the DeferredList but still referenced by the
original deferred and nobody was consuming it.

The tests started to fail because the reference from DeferredList
prevented the underlining deferred from been collected before the test
finish invalidating the effect of self.flushedLoggedErrors() call.
  • Loading branch information
dangra committed Nov 9, 2013
1 parent 4f7ad36 commit 419a78015958f3b7bc2506881a21559d6231aa40
Showing with 2 additions and 2 deletions.
  1. +2 −2 scrapy/utils/defer.py
@@ -82,8 +82,8 @@ def process_parallel(callbacks, input, *a, **kw):
callbacks
"""
dfds = [defer.succeed(input).addCallback(x, *a, **kw) for x in callbacks]
d = defer.gatherResults(dfds)
d.addErrback(lambda _: _.value.subFailure)
d = defer.DeferredList(dfds, fireOnOneErrback=1, consumeErrors=1)
d.addCallbacks(lambda r: [x[1] for x in r], lambda f: f.value.subFailure)
return d

def iter_errback(iterable, errback, *a, **kw):

0 comments on commit 419a780

Please sign in to comment.
You can’t perform that action at this time.