No description, website, or topics provided.
Python
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is 5 commits ahead of emendelski:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
visegrad
.gitignore
README.md
private.json.default
requirements.txt
scrapy.cfg
server_cert.pem
vpapi.py

README.md

Installation

You need Python 2.7 and install system requirements.

sudo apt-get install python-dev libxml2-dev libxslt-dev libssl-dev libffi-dev
pip install -r requirements.txt

Usage

Credentials are stored in JSON file private.json in project directory.

All Scrapy commands and settings (e.g. logging or throtlling) can be applied to these scrapers. Docs are here.

Hungary

private.json

{
	"VPAPI_PWD_HU_ORSZAGGYULES": "secret",
	"HU_ORSZAGGYULES_ACCESS_TOKEN": "secret"
}

Command

scrapy crawl parlament.hu

Montenegro

private.json

{
	"VPAPI_PWD_ME_SKUPSTINA": "secret"
}

Command

scrapy crawl skupstina.me

Poland

private.json

{
	"VPAPI_PWD_PL_SEJM": "secret"
}

Command

scrapy crawl mojepanstwo.pl