GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
No description provided.
I would love to see this.
Or any Python distributed crawler.
I'll bump it up in priority. :) Watch this thread--when I clear off some of the more pressing stuff I'm doing, I'll work on this.
@turian Can I ask where you're running into difficulty? What would you like to see in an article? Trying to figure out how to approach this that won't end with us writing an example for every web scraper ever, because that's not really tenable.
Installing scrapy and lxml should do for now. When I test my workers locally they work, when i upload it the build fails with errors:
No package 'libffi' found
removing: _configtest.c _configtest.o
c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory