Permalink
Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
20 lines (16 sloc) 715 Bytes
europeana-crawler is a simple proof of concept script for extracting
rdfa metadata from record pages using the sitemap they make available
for search engine crawlers. The triples for each resource are persisted
as a file to the filesystem using a pairtree to evenly distribute the
files across subdirectories.
To run the crawler you'll need to install a few dependencies. You might
want to do this with a virtualenv, or globally on your system. The
instructions here are for using a virtualenv:
1. virtualenv --no-site-packages ENV
2. source ENV/bin/activate
3. pip install -r requirements.pip
4. ./crawl.py
5. tail -f crawl.log
6. ./aggregate.py > europeana.nt
Questions, comments:
Ed Summers <ehs@pobox.com>