Scrapes the search result pages of https://zoek.officielebekendmakingen.nl/ using Scrapy, and downloads the XML documents it may find along it's way.
Python
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
officielebekendmakingen
.gitignore
README.mdown
scrapy-ctl.py

README.mdown

Officiële bekendmakingen scraper

Author: Justin van Wees (justin@vwees.net)
Date: 2010-06-21

Officiële bekendmakingen scraper scrapes the search result pages of https://zoek.officielebekendmakingen.nl/ and downloads the XML documents it may find along it's way.

Requirements

Installation and configuration

After you've made sure that all the required Python packages are installed, please edit "officielebekendmakingen/settings.py". The settings should be self explanatory.

Running Officielebekendmakingen scraper

Run python scrapy-ctl.py crawl zoek.officielebekendmakingen.nl

You can monitor the Scrapy process by visiting http://[HOSTNAME]:6080 or by opening a Telnet session to port 6023 (the "stats" object contains information about the current run)