ScrapyDemo is a Scrapy project for the [blog](http://ibloodline.com/articles/2017/12/15/Scrapy-Tutorial.html).
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
scrapydemo
.gitignore
LICENSE
README.md
scrapy.cfg
stackoverflow.jl
tag.jl
user.jl

README.md

ScrapyDemo

ScrapyDemo is a Scrapy project for the blog.It can scrape questions from stackoverflow.

Extracted data

This project extracts questions and users.The extracted data looks like this sample:

{
    "question_content": "How to pass a user defined argument in scrapy spider",
    "user": "L Lawliet"
}

Spiders

This project contains three spiders and you can list them using the list command:

$ scrapy list
stackoverflow
tag
user

Running the spiders

You can run a spider using the scrapy crawl command, such as:

$ scrapy crawl stackoverflow

If you want to save the scraped data to a file, you can pass the -o option:

$ scrapy crawl stackoverflow -o stackoverflow.jl