[MRG +1] LogCounterHandler is never removed from root handlers list, fix that #1259
Conversation
lambda is garbage collected and because receiver is added as weak reference by default - when signals.engine_stopped is fired logging.root.removeHandler is not executed. Fixed that by assigning lambda to a private argument and not by using connect(..., weak=False) because I belive this lambda function should be collected with crawler object
| signals.engine_stopped) | ||
| # lambda is assigned to Crawler attribute because this way it is not | ||
| # garbage collected after leaving __init__ scope | ||
| self.__remove_handler = lambda: logging.root.removeHandler(handler) |
kmike
May 27, 2015
Member
Sounds reasonable. We can also call it '_on_engine_stopped'. Another option is to create a real '_on_engine_stopped' method and keep the reference to a handler: self.loghandler = LogCounterHandler(..). But your method also looks fine.
Sounds reasonable. We can also call it '_on_engine_stopped'. Another option is to create a real '_on_engine_stopped' method and keep the reference to a handler: self.loghandler = LogCounterHandler(..). But your method also looks fine.
nramirezuy
May 27, 2015
Contributor
has to be an attribute? can't be just a variable ?
_remove_handler = lambda: logging.root.removeHandler(handler)
has to be an attribute? can't be just a variable ?
_remove_handler = lambda: logging.root.removeHandler(handler)
kmike
May 27, 2015
Member
it can't - the point of this change is to store a reference to lambda
it can't - the point of this change is to store a reference to lambda
kmike
changed the title
curita
added a commit
that referenced
this pull request
May 27, 2015
…removed [MRG +1] LogCounterHandler is never removed from root handlers list, fix that
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
lambda is garbage collected and because receiver is added as weak reference by default - when
signals.engine_stoppedis firedlogging.root.removeHandleris not executed. Fixed that by assigning lambda to a private argument and not by usingconnect(..., weak=False)because I believe this lambda function should be collected with crawler object and not stay indispatcher.connectionsforever.Tested in ScrapyRT where multiple crawls are running in the same process. Before the fix:
After the fix: