rename spider will load the old left .pyc instead of renamed spider

[samson-wang]

Hi all, I'm not sure if this is an issue.
If got y.py and y.pyc in the spider directory.
Then rename it with mv y.py x.py and update the new x.py.
After updates, run scrapy crawl spider will use the old version of spider instead of updates in the x.py.
Have to remove y.pyc first.
Thanks!

[candlejack297]

Isn't this just normal Python .pyc problems? Pretty sure it's not scrapy specific. If you're actively moving/editing source code, you should just clear all .pyc files to avoid any issues (they'll get regenerated automatically).

[samson-wang]

@candlejack297 You're right.
However scrapy automatically load the spider code in the spiders directory. I thought scrapy should have handled this situation.

[djunzu]

That is just how Python works. (No?) Scrapy is written in Python. That means it is restricted by all constraints that applies to Python scripts/programs.

I think it is not a Scrapy bug.
As a feature request, I would say that it does not worth the needed code.

[redapple]

May not be worth the effort, but if it's to be investigated, I believe it'd be around
SpiderLoader and how it uses importlib.import_module in walk_modules
Maybe look at imp.load_source (hint from http://stackoverflow.com/a/8422549)

[kmike]

I think we shouldn't mess with pyc files and implement special handling of them; pyc files are standard Python feature and it'd be too opinionated to assume that .pyc modules shouldn't be loaded - e.g. someone may choose to deploy pyc files instead of .py files.

What I think we should do is to warn if there are several spiders with the same name. It is a more general problem; it is specific to scrapy, not to .pyc files handling.

[ghost]

Arrived here searching DuckDuckGo for "There are several spiders with the same name" in quotes.

I'm new to Python and this discussion helped me when I needed to get past the warning mentioned in the title, especially given Atom editor hides files with the pyc extension in its interface by default—I had no idea the compiled files were even there! Thank you!! 🥇