python-website-scraping

Companion repository of the article "How to scrape websites in 5 minutes ?" in Theodo's blog

Setup

If you don't have virtualenv installed on your computer, install it using the official documentation
Setup a virtualenv with virtualenv venv
Activate your virtualenv with source venv/bin/activate
Install the required libraries with pip install -r requirements.txt

Within your virtual environment with Scrapy installed, you can run each Spider with scrapy runspider selected_spider.py --output output_file.json

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.gitignore		.gitignore
README.md		README.md
article_spider.py		article_spider.py
blog_spider.py		blog_spider.py
page_spider.py		page_spider.py
requirements.txt		requirements.txt