Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website.
Switch branches/tags
Nothing to show
Clone or download
Latest commit dc1295b Aug 15, 2018
Failed to load latest commit information.
.vscode update Mar 12, 2018
WebHub fix bug Apr 4, 2018
img Update qrcode Oct 10, 2017
.gitignore format Apr 10, 2017
LICENSE init code Apr 8, 2017 Update Mar 12, 2018 Add instruction of specifying categories Feb 24, 2018
requirements.txt Update requirements.txt Aug 15, 2018

GitHub forks GitHub stars GitHub license

Disclaimer: This project is intended to study the Scrapy Spider Framework and the MongoDB database, it cannot be used for commercial or other personal intentions. If used improperly, it will be the individuals bear.

  • The project is mainly used for crawling a Website, the largest site in the world. In doing so it retrieves video titles, duration, mp4 link, cover url and direct Website`s url.
  • This project crawls quickly, but with a simple structure.
  • This project can crawl up to 5 millon Website`s videos per day, depending on your personal network. Because of my slow bandwith my results are relatively slow.
  • The crawler requests 10 threads at a time, and because of this can achieve the speed mentioned above. If your network is more performant you can request more threads and crawl a larger amount of videos per day. For the specific configuration see [pre-boot configuration]

Environment, Architecture

Language: Python2.7

Environment: MacOS, 4G RAM

Database: MongoDB

  • Mainly uses the scrapy reptile framework.
  • Join to the Spider randomly by extracted from the Cookie pool and UA pool.
  • Start_requests start five Request based on Website`s classification, and crawl the five categories at the same time.
  • Support paging crawl data, and join to the queue.

Instructions for use

Pre-boot configuration

  • Install MongoDB and start without configuration
  • Install Python dependent modules:Scrapy, pymongo, requests or pip install -r requirements.txt
  • Modify the configuration by needed, such as the interval time, the number of threads, etc.

Start up

  • cd WebHub
  • python

Run screenshots

Database description

The table in the database that holds the data is PhRes. The following is a field description:

PhRes table:

video_title:     The title of the video, and as a unique.
link_url:        Video jump to Website`s link
image_url:       Video cover link
video_duration:  The length of the video, in seconds
quality_480p:    Video 480p mp4 download address

For Chinese

  • 关注微信公众号,学习Python开发