Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rename spider will load the old left .pyc instead of renamed spider #2181

Closed
samson-wang opened this issue Aug 15, 2016 · 6 comments
Closed

rename spider will load the old left .pyc instead of renamed spider #2181

samson-wang opened this issue Aug 15, 2016 · 6 comments
Labels
Milestone

Comments

@samson-wang
Copy link

@samson-wang samson-wang commented Aug 15, 2016

Hi all, I'm not sure if this is an issue.
If got y.py and y.pyc in the spider directory.
Then rename it with mv y.py x.py and update the new x.py.
After updates, run scrapy crawl spider will use the old version of spider instead of updates in the x.py.
Have to remove y.pyc first.
Thanks!

@candlejack297
Copy link

@candlejack297 candlejack297 commented Aug 15, 2016

Isn't this just normal Python .pyc problems? Pretty sure it's not scrapy specific. If you're actively moving/editing source code, you should just clear all .pyc files to avoid any issues (they'll get regenerated automatically).

@samson-wang
Copy link
Author

@samson-wang samson-wang commented Aug 16, 2016

@candlejack297 You're right.
However scrapy automatically load the spider code in the spiders directory. I thought scrapy should have handled this situation.

@djunzu
Copy link
Contributor

@djunzu djunzu commented Aug 17, 2016

That is just how Python works. (No?) Scrapy is written in Python. That means it is restricted by all constraints that applies to Python scripts/programs.

I think it is not a Scrapy bug.
As a feature request, I would say that it does not worth the needed code.

@redapple
Copy link
Contributor

@redapple redapple commented Sep 13, 2016

May not be worth the effort, but if it's to be investigated, I believe it'd be around
SpiderLoader and how it uses importlib.import_module in walk_modules
Maybe look at imp.load_source (hint from http://stackoverflow.com/a/8422549)

@kmike
Copy link
Member

@kmike kmike commented Sep 19, 2016

I think we shouldn't mess with pyc files and implement special handling of them; pyc files are standard Python feature and it'd be too opinionated to assume that .pyc modules shouldn't be loaded - e.g. someone may choose to deploy pyc files instead of .py files.

What I think we should do is to warn if there are several spiders with the same name. It is a more general problem; it is specific to scrapy, not to .pyc files handling.

@ghost
Copy link

@ghost ghost commented Oct 1, 2017

Arrived here searching DuckDuckGo for "There are several spiders with the same name" in quotes.

I'm new to Python and this discussion helped me when I needed to get past the warning mentioned in the title, especially given Atom editor hides files with the pyc extension in its interface by default—I had no idea the compiled files were even there! Thank you!! 🥇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.