DISCONTINUED Easily setup a ScraperWiki scraper for local python development.
Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
scraperwiki_local_python
.gitignore
README.md
examples.py

README.md

ScraperWiki Local Python

Use this if you want to locally develop and test a Python ScraperWiki scraper. The code in the scraperwiki_local_python/scraperwiki directory is a subset of the ScraperWiki repository.

Using the ScraperWiki source to emulate the functions was the main goal of this project.

Features

Usage

Download the scraperwiki_local_python folder into your working directory and, in your scraper, replace the import scraperwiki line with import scraperwiki_local_python as scraperwiki (be sure to revert this change and replace import scraperwiki_local_python as scraperwiki with import scraperwiki before saving your scraper at ScraperWiki)

The SQLite database and the scraped page cache will be saved to a folder in your working directory. E.g. Running python examples.py will create an examples_data folder with page cache folder and a defaultdb.sqlite database inside of it.

License

GPLv3 Same as ScraperWiki