Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
A scrapy-based Hacker News crawler.
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
news move /html to /index.html
LICENSE
README.md Merge branch 'master' of github.com:mvanveen/hncrawl
scrapy.cfg

README.md

HNCrawl

A scrapy-based Hacker News crawler.

Introduction

HNCrawl is a tiny, simple scrapy-based crawler which grabs the html content of pages linked to the front page of hacker news.

Examples

Installation

$ pip install scrapy
$ git clone git@github.com:mvanveen/hncrawl.git

Scraping

Note: Please be sure to keep in mind that the Crawl-Delay value set in the HN robots.txt file is set to 30 seconds. Please be sure to avoid using the scraper more than once per 30 seconds!

Scrape the links from the front page of HN

$ scrapy crawl hnspider

Scrape items and return json summary of items scraped into items.json

$ scrapy crawl alias_scrape -o items.json -t json

Output

Here is an example file hierarchy. Folders are a hex digest of the SHA1 hash of the hacker news item url.

 ├── out
 │   ├── 000f86c7547b47a700dee0879a0fe08b4597360f
 │   │   └── index.html
 │   ├── 0190cbad182ab3bc9a92482d169f38e363ca3c57
 │   │   └── index.html
 │   ├── 02bae9642c8dd4b75a593c1c42beff62824ee8fc
 │   │   └── index.html
 │   ├── 05c1460571f0ac45f77bf2ecbd3cba8b85c20621
 │   │   └── index.html
 │   ├── 0b1587a3dbe9996d10a0fd3250f75462ebd59a0b
 │   │   └── index.html
 │   ├── 0c5c67585004e03341e6a87d2db5257b93337b86
 │   │   └── 

The JSON summary of news items look like this:

{'title': u'EFF Wins Protection for Time Zone Database',
 'url': u'https://www.eff.org/press/releases/eff-wins-protection-time-zone-database'}

Dependencies

License

HNCrawl is MIT licensed.

Something went wrong with that request. Please try again.