Signal dispatching doesn't work on pypy

[royisme]

There are some exception message when I use pypy1.9 to run scrapy ,but use python2.7 is ok.

% /usr/local/share/pypy/scrapy shell http://www.baidu.com             
zsh: correct 'shell' to 'shells' [nyae]? n
2012-11-09 16:40:06+0800 [scrapy] INFO: Scrapy 0.16.1 started (bot: scrapybot)
2012-11-09 16:40:06+0800 [scrapy] DEBUG: Enabled extensions: TelnetConsole, WebService, CloseSpider, CoreStats, SpiderState
2012-11-09 16:40:06+0800 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2012-11-09 16:40:06+0800 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2012-11-09 16:40:06+0800 [scrapy] DEBUG: Enabled item pipelines: 
2012-11-09 16:40:06+0800 [scrapy] ERROR: Error caught on signal handler: <bound method instance.start_listening of <scrapy.telnet.TelnetConsole instance at 0x00000001063f0bc0>>
    Traceback (most recent call last):
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/core/engine.py", line 75, in start
        yield self.signals.send_catch_log_deferred(signal=signals.engine_started)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 134, in maybeDeferred
        result = f(*args, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: start_listening() got 2 unexpected keyword arguments

2012-11-09 16:40:06+0800 [scrapy] ERROR: Error caught on signal handler: <bound method instance.start_listening of <scrapy.webservice.WebService instance at 0x0000000103f35240>>
    Traceback (most recent call last):
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/core/engine.py", line 75, in start
        yield self.signals.send_catch_log_deferred(signal=signals.engine_started)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 134, in maybeDeferred
        result = f(*args, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: start_listening() got 2 unexpected keyword arguments

2012-11-09 16:40:06+0800 [default] INFO: Spider opened
2012-11-09 16:40:06+0800 [default] ERROR: Error caught on signal handler: <bound method CoreStats.spider_opened of <scrapy.contrib.corestats.CoreStats object at 0x0000000105ff8f38>>
    Traceback (most recent call last):
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/core/engine.py", line 225, in open_spider
        yield self.signals.send_catch_log_deferred(signals.spider_opened, spider=spider)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 134, in maybeDeferred
        result = f(*args, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: spider_opened() got 2 unexpected keyword arguments

2012-11-09 16:40:06+0800 [default] ERROR: Error caught on signal handler: <bound method SpiderState.spider_opened of <scrapy.contrib.spiderstate.SpiderState object at 0x0000000105ff9088>>
    Traceback (most recent call last):
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/core/engine.py", line 225, in open_spider
        yield self.signals.send_catch_log_deferred(signals.spider_opened, spider=spider)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 134, in maybeDeferred
        result = f(*args, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: spider_opened() got 2 unexpected keyword arguments

2012-11-09 16:40:06+0800 [default] ERROR: Error caught on signal handler: <bound method OffsiteMiddleware.spider_opened of <scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware object at 0x0000000104821d38>>
    Traceback (most recent call last):
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 1045, in _inlineCallbacks
        result = g.send(result)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/core/engine.py", line 225, in open_spider
        yield self.signals.send_catch_log_deferred(signals.spider_opened, spider=spider)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/signalmanager.py", line 23, in send_catch_log_deferred
        return signal.send_catch_log_deferred(*a, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/utils/signal.py", line 53, in send_catch_log_deferred
        *arguments, **named)
    --- <exception caught here> ---
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 134, in maybeDeferred
        result = f(*args, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: spider_opened() got 2 unexpected keyword arguments

2012-11-09 16:40:07+0800 [default] DEBUG: Crawled (200) <GET http://www.baidu.com> (referer: None)
2012-11-09 16:40:07+0800 [default] ERROR: Error caught on signal handler: <bound method CoreStats.response_received of <scrapy.contrib.corestats.CoreStats object at 0x0000000105ff8f38>>
    Traceback (most recent call last):
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 464, in _startRunCallbacks
        self._runCallbacks()
      File "/usr/local/Cellar/pypy/1.9/site-packages/twisted/internet/defer.py", line 551, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/core/engine.py", line 200, in _on_success
        response=response, request=request, spider=spider)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/signalmanager.py", line 19, in send_catch_log
        return signal.send_catch_log(*a, **kw)
    --- <exception caught here> ---
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/utils/signal.py", line 22, in send_catch_log
        *arguments, **named)
      File "/usr/local/Cellar/pypy/1.9/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 47, in robustApply
        return receiver(*arguments, **named)
    exceptions.TypeError: response_received() got 4 unexpected keyword arguments

[s] Available Scrapy objects:
[s]   hxs        <HtmlXPathSelector xpath=None data=u'<html><head><meta http-equiv="Content-Ty'>
[s]   item       {}
[s]   request    <GET http://www.baidu.com>
[s]   response   <200 http://www.baidu.com>
[s]   settings   <CrawlerSettings module=None>
[s]   spider     <BaseSpider 'default' at 0x10482b980>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

[pablohoffman]

Thanks for reporting this. Keep in mind that PyPy support is not a priority of the core team right now, so it might take a while for this one to be looked upon, unless there is community interest (and involvement) to propose a solution faster.

[royisme]

Thanks , I'm understand.

[pablohoffman]

@royisme I didn't mean this issue need to be closed, just that it won't be a priority for the moment (it can remain open)

[royisme]

Ok, I got it.

[joehillen]

I was able to fix this issue. Unfortunately in my personal crawling I found PyPy to be twice as slow as CPython. I think this is caused by lxml being twice as slow in PyPy vs CPython.

[dangra]

dupe of #205