Error with Spider arguments sent using Scrapyd

[dfockler]

I figured out the solution to this, but I thought I would put it somewhere for comment and documentation.
I was getting this error when scheduling spider runs through scrapyd with spider arguments.

    TypeError: __init__() got an unexpected keyword argument '_job'

It's not stated in the documentation explicitly, but when using scrapyd your spiders should allow for any number of arguments which I did not realize. scrapyd was sending a _job keyword to the spider, which my spider couldn't handle. So in short your spider __init__ def should look like

    def __init__(self, *args, **kwargs):

and look for the keyword of the argument you want instead of

    def __init__(self, my_keyword="default"):

Hopefully this helps some other python newbies in the future.

[jayzeng]

@dfockler thanks for sharing, mind sending in a PR to update the docs? (https://github.com/scrapy/scrapyd/tree/master/docs)

[emilepetrone]

@dfockler ditto, thanks for sharing. Saved me!

[wangzhen11aaa]

thanks.@dfockler

[sept44]

hi，friends.
i meet a question,i use scrapy-redis,when I run my code,I was getting this error when scheduling spider runs through scrapyd with spider arguments.look:

  File "D:\swipers\snowball_spider\.env\lib\site-packages\scrapy_redis\pipelines.py", line 54, in from_settings
    return cls(**params)
TypeError: __init__() got an unexpected keyword argument 'server'

I thought for a long time, I thought it was a version conflict of redis at first, but I changed a lot of redis versions, or not, my scrapy-redis is 0.6.8, I need your help. Thank you.

[Digenis]

@sept44 rmax/scrapy-redis#59

[re0phimes]

saved my ass!