Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Scrapy Dynamic Configurable

A dynamic configurable news crawler based Scrapy. See the blogs for more detail.

##Requirements

  • Scrapy
  • MySQL
  • Redis
  • SQLAlchemy

##Install in development

Mac OS X, use Homebrew

$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install mysql
$ brew install redis
$ sudo pip install scrapy
$ sudo pip install SQLAlchemy
$ sudo pip install redis

Ubuntu*

$ sudo apt-get install redis-server mysql-server mysql-client
$ pip install scrapy
$ pip install SQLAlchemy
$ pip install redis

and restore datatables from 'db.sql'

It is for scrapy 1.0 . It is for scrapy 1.0 .

About

A dynamic configurable news crawler based Scrapy

Resources

Releases

No releases published

Languages

You can’t perform that action at this time.