Skip to content

adipasquale/techcrunch-incremental-scrapy-spider-with-mongodb

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Techcrunch Incremental Scrapy Spider With MongoDB

This project is the support for this blog post : Incremental crawler with Scrapy and MongoDB

Local setup

pip3 install -r requirements.txt

Setup a local MongoDB server. On Mac OS X :

brew install mongodb
brew services start mongodb

Run

scrapy crawl techcrunch -a limit_pages=2

Releases

No releases published

Packages

No packages published

Languages