Skip to content

Scrapy QuotesBot with Hext. Hext is a domain-specific language for extracting structured data from HTML documents.

License

Notifications You must be signed in to change notification settings

html-extract/hext-scrapy-quotesbot

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Scrapy QuotesBot with Hext

This is a fork of Scrapy's QuotesBot. It adds a new spider called toscrape-hext.

$ pip install hext scrapy
$ scrapy list
toscrape-css
toscrape-hext
toscrape-xpath
$ scrapy crawl toscrape-hext -o quotes.json

Original readme:

QuotesBot

This is a Scrapy project to scrape quotes from famous people from http://quotes.toscrape.com (github repo).

This project is only meant for educational purposes.

Extracted data

This project extracts quotes, combined with the respective author names and tags. The extracted data looks like this sample:

{
    'author': 'Douglas Adams',
    'text': '“I may not have gone where I intended to go, but I think I ...”',
    'tags': ['life', 'navigation']
}

Spiders

This project contains two spiders and you can list them using the list command:

$ scrapy list
toscrape-css
toscrape-xpath

Both spiders extract the same data from the same website, but toscrape-css employs CSS selectors, while toscrape-xpath employs XPath expressions.

You can learn more about the spiders by going through the Scrapy Tutorial.

Running the spiders

You can run a spider using the scrapy crawl command, such as:

$ scrapy crawl toscrape-css

If you want to save the scraped data to a file, you can pass the -o option:

$ scrapy crawl toscrape-css -o quotes.json

About

Scrapy QuotesBot with Hext. Hext is a domain-specific language for extracting structured data from HTML documents.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Languages

  • Python 100.0%