Skip to content

Commit

Permalink
~readme
Browse files Browse the repository at this point in the history
  • Loading branch information
vantoozz committed Jun 30, 2017
1 parent cd40c84 commit 838ba72
Showing 1 changed file with 120 additions and 3 deletions.
123 changes: 120 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,136 @@ Library for scraping free proxies lists

### Setup

Proxy-scrapper library is built on top of [HTTPlug](http://httplug.io/) and requires a compatible HTTP client. Available clients are listed on Packagist: https://packagist.org/providers/php-http/client-implementation. To use the library you have to install any of them, e.g.:
Proxy-scraper library is built on top of [HTTPlug](http://httplug.io/) and requires a compatible HTTP client. Available clients are listed on Packagist: https://packagist.org/providers/php-http/client-implementation. To use the library you have to install any of them, e.g.:

```bash
composer require php-http/guzzle6-adapter
```

Then install proxy-scrapper library itself:
Then install proxy-scraper library itself:
```bash
composer require vantoozz/proxy-scraper
```

### Usage

### Tests
##### Single scraper
```php
<?php declare(strict_types = 1);

use GuzzleHttp\Client as GuzzleClient;
use Http\Adapter\Guzzle6\Client as GuzzleAdapter;
use Http\Message\MessageFactory\GuzzleMessageFactory;
use Vantoozz\ProxyScraper\HttpClient\HttplugHttpClient;
use Vantoozz\ProxyScraper\Scrapers;

require_once __DIR__ . '/vendor/autoload.php';

$httpClient = new HttplugHttpClient(
new GuzzleAdapter(new GuzzleClient),
new GuzzleMessageFactory
);

$scraper = new Scrapers\FreeProxyListScraper($httpClient);

foreach ($scraper->get() as $proxy) {
echo (string)$proxy . "\n";
}
```

##### Composite scraper
You can easily get data from many scrapers at once
```php
<?php declare(strict_types = 1);

use GuzzleHttp\Client as GuzzleClient;
use Http\Adapter\Guzzle6\Client as GuzzleAdapter;
use Http\Message\MessageFactory\GuzzleMessageFactory;
use Vantoozz\ProxyScraper\HttpClient\HttplugHttpClient;
use Vantoozz\ProxyScraper\Scrapers;

require_once __DIR__ . '/vendor/autoload.php';

$httpClient = new HttplugHttpClient(
new GuzzleAdapter(new GuzzleClient),
new GuzzleMessageFactory
);

$compositeScraper = new Scrapers\CompositeScraper;

$compositeScraper->addScraper(new Scrapers\FreeProxyListScraper($httpClient));
$compositeScraper->addScraper(new Scrapers\MultiproxyScraper($httpClient));
$compositeScraper->addScraper(new Scrapers\ProxyDbScraper($httpClient));
$compositeScraper->addScraper(new Scrapers\SocksProxyScraper($httpClient));
$compositeScraper->addScraper(new Scrapers\SpysMeScraper($httpClient));
$compositeScraper->addScraper(new Scrapers\UsProxyScraper($httpClient));

foreach ($compositeScraper->get() as $proxy) {
echo (string)$proxy . "\n";
}
```

##### Error handling
Sometimes things goes wrong. This example shows how to handle errors while getting data from many scrapers:
```php
<?php declare(strict_types = 1);

use GuzzleHttp\Client as GuzzleClient;
use Http\Adapter\Guzzle6\Client as GuzzleAdapter;
use Http\Message\MessageFactory\GuzzleMessageFactory;
use Vantoozz\ProxyScraper\Exceptions\ScraperException;
use Vantoozz\ProxyScraper\HttpClient\HttplugHttpClient;
use Vantoozz\ProxyScraper\Ipv4;
use Vantoozz\ProxyScraper\Port;
use Vantoozz\ProxyScraper\Proxy;
use Vantoozz\ProxyScraper\Scrapers;

require_once __DIR__ . '/vendor/autoload.php';

$httpClient = new HttplugHttpClient(
new GuzzleAdapter(new GuzzleClient),
new GuzzleMessageFactory
);

$compositeScraper = new Scrapers\CompositeScraper;

// Throws an exception
$compositeScraper->addScraper(new class implements Scrapers\ScraperInterface
{
public function get(): \Generator
{
throw new ScraperException('some error');
}
});

// No exceptions
$compositeScraper->addScraper(new class implements Scrapers\ScraperInterface
{
public function get(): \Generator
{
yield new Proxy(new Ipv4('192.168.0.1'), new Port(8888));
}
});

// Set exception handler
$compositeScraper->handleScraperExceptionWith(function (ScraperException $e) {
echo 'An error occurs: ' . $e->getMessage() . "\n";
});

foreach ($compositeScraper->get() as $proxy) {
echo (string)$proxy . "\n";
}
```
Will output
```
An error occurs: some error
192.168.0.1:8888
```

_Note. Examples uses Guzzle as HTTP client._


### Testing

##### Unit tests
```bash
Expand Down

0 comments on commit 838ba72

Please sign in to comment.