Selenium Based Web Crawler

ReadTheDocs - http://browler.readthedocs.org/en/latest/

Requirements

• redis brew install redis

Install

pip install browler

Uninstall

pip uninstall browler

Configuration

With Selenium Hub

config = {
            "browser": 'remote',
            "remote": {
                "url": 'http://localhost:49044/wd/hub',
                'browser': 'firefox'
            },
            "url": "https://en.wikipedia.org/wiki/Main_Page",
            "limit": 10,
            "processes": 2
         }

Local Firefox

config = {
            "browser": 'firefox',
            "url": "https://en.wikipedia.org/wiki/Main_Page",
            "limit": 10,
            "processes": 2
         }