Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Parker is a Python-based web spider for collecting specific data across a set of configured sites.
Python Other
Failed to load latest commit information.
bin Attempt to get a config path from a PARKER_CONFIG environment variabl…
parker Update mediafile.filename from S3.
puppet Update puppet to conform with changes to User module and bump version…
test
.coveragerc Ensure .coveragerc is used in make test.
.gitignore
CHANGES.rst Bumped version and changes.
LICENSE Add a bunch of files to set this up as a package.
MANIFEST.in Ensure CHANGES.rst is included in manifest.
Makefile Symlink the etc/parker folder to /etc/parker when developing.
README.rst Update documentation, bump the version, remove the needless decode/en…
VERSION Bumped version and changes.
Vagrantfile Add a shell provisioner to run 'make install'.
requirements.txt Add boto to the requirements.
setup.cfg Add a bunch of files to set this up as a package.
setup.py

README.rst

Parker

Parker is a Python-based web spider for collecting specific data across a set of configured sites.

Non-Python requirements:

  • Redis - for task queuing and visit tracking
  • libxml - for HTML parsing of pages

Installation

Install using pip:

$ pip install parker

Configuration

To configure Parker, you will need to install the configuration files in a suitable location for the user running Parker. To do this, use the parker-config script. For example:

$ parker-config ~/.parker

This will install the configuration in your homedir and will output the related environment variable for you to set in your .bashrc.

Something went wrong with that request. Please try again.