Pider

Docmentation: English Chinese

Introduction

   Pider is an elegant,powerful,modulized,templatized spider framework.It aims to fertilize the php community and make easier for PHP deveoper to write a spider. view document in details.

Features

• Crawler (Support)
• Crawler with multi-process (Support)
• Command line interface (Not full support )
• Template crawling(Not support)
• Well debug interface(Not support)
• Data cleaning(Not full support)
• Data visuliazation(Not support)
• Distribution (Support)

Requirments

• PHP >= 7.1
• pthreads (optional for multi-threads support)
• pcntl (optional for multi-processes support)
• xmlreader (optional for XML file processing support)

Installation

Use docker

  There are a out-of-box docker environment for use. You can just pull and use it right away.

docker pull jhbian/pider
git clone https://github.com/duanqiaobb/pider
cd pider
composer install

Install into your laptop (only linux supported currently)

  You can run install.sh under root directory of the project whatever you prefer to install an environment into your laptop.

• Install composer at first.(Details can be pored over on https://getcomposer.org/)

• Set up environment

  git clone https://github.com/duanqiaobb/pider.git
  chmod u+x install.sh
  ./install.sh

Usage

     Hereinafter, I assume that you had set pider up ,not only environment but also the framework itself.

Basic Spider

    This spider crawles categories of product in index page of jd.com

cd pider
mkdir example
cd example && touch JdIndexCategorySpider.php

//In file JdIndexCategorySpider.php
<?php
use Pider\Spider;
use Pider\Http\Response;
    
class JdIndexCategorySpider extends Spider {
    protected $domains = 'www.jd.com';
    protected $urls = [
       'www.jd.com/'];
    
    //Parse data from response of requests
    public function parse(Response $response) {
        $response = $response->outputEncode('utf-8');
        $category_names = $response->xpath("//ul[contains(@class,'cate_menu')]/li/a/text()")->extract();
        var_dump($category_names);
    }
} 

../pider JdIndexCategorySpider.php

array(46) {
  [0] =>
  string(12) "家用电器"
  [1] =>
  string(6) "手机"
  [2] =>
  string(9) "运营商"
  [3] =>
  string(6) "数码"
  [4] =>
  string(6) "电脑"
  [5] =>
  string(6) "办公"
  [6] =>
  string(6) "家居"
  [7] =>
  string(6) "家具"
  ...
}

Spider with proxy

touch JdIndexCategoryWithProxySpider.php

use Pider\Spider;
use Pider\Http\Response;
    
class JdIndexCategoryWithProxySpider extends Spider {
    protected $domains = 'www.jd.com';
    protected $urls = [
       'www.jd.com/'];

    //Generate urls to be crawled
    public function start_requests():array {
        $std_urls = ['www.jd.com'];
        Request::proxy_handler(function(){
            return xxx(); //function return a proxy ip
        });
       return $std_urls; //url or Request object array
    }

    
    //Parse data from response of requests
    public function parse(Response $response) {
        $response = $response->outputEncode('utf-8');
        $category_names = $response->xpath("//ul[contains(@class,'cate_menu')]/li/a/text()")->extract();
        var_dump($category_names);
    }
} 

../pider Examplespider.php

array(46) {
  [0] =>
  string(12) "家用电器"
  [1] =>
  string(6) "手机"
  [2] =>
  string(9) "运营商"
  [3] =>
  string(6) "数码"
  [4] =>
  string(6) "电脑"
  [5] =>
  string(6) "办公"
  [6] =>
  string(6) "家居"
  [7] =>
  string(6) "家具"
  ...
}

Todo

Contribution

     If you have any ideas about this project, please don't hesitate to pull a request.