Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rename spider will load the old left .pyc instead of renamed spider #2181

samson-wang opened this issue Aug 15, 2016 · 6 comments


Copy link

Hi all, I'm not sure if this is an issue.
If got and y.pyc in the spider directory.
Then rename it with mv and update the new
After updates, run scrapy crawl spider will use the old version of spider instead of updates in the
Have to remove y.pyc first.

Copy link

Isn't this just normal Python .pyc problems? Pretty sure it's not scrapy specific. If you're actively moving/editing source code, you should just clear all .pyc files to avoid any issues (they'll get regenerated automatically).

Copy link

@candlejack297 You're right.
However scrapy automatically load the spider code in the spiders directory. I thought scrapy should have handled this situation.

Copy link

djunzu commented Aug 17, 2016

That is just how Python works. (No?) Scrapy is written in Python. That means it is restricted by all constraints that applies to Python scripts/programs.

I think it is not a Scrapy bug.
As a feature request, I would say that it does not worth the needed code.

Copy link

May not be worth the effort, but if it's to be investigated, I believe it'd be around
SpiderLoader and how it uses importlib.import_module in walk_modules
Maybe look at imp.load_source (hint from

Copy link

kmike commented Sep 19, 2016

I think we shouldn't mess with pyc files and implement special handling of them; pyc files are standard Python feature and it'd be too opinionated to assume that .pyc modules shouldn't be loaded - e.g. someone may choose to deploy pyc files instead of .py files.

What I think we should do is to warn if there are several spiders with the same name. It is a more general problem; it is specific to scrapy, not to .pyc files handling.

Copy link

ghost commented Oct 1, 2017

Arrived here searching DuckDuckGo for "There are several spiders with the same name" in quotes.

I'm new to Python and this discussion helped me when I needed to get past the warning mentioned in the title, especially given Atom editor hides files with the pyc extension in its interface by default—I had no idea the compiled files were even there! Thank you!! 🥇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

5 participants