Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

48 lines (42 sloc) 1.532 kb

Have you ever wanted a machine readable list of URLs for library websites around the world? It's pretty unlikely, but if you have, read on.

This screen scraper will crawl LIBWEB and extract information about library websites around the world and persist it as JSON. The libraries.json file that is committed here has name, location and the URL for 7990 libraries, kinda like:

[
  {
    "url": "http://libraries.adelphi.edu/", 
    "name": "Adelphi University", 
    "location": "Garden City NY"
  }, 
  {
    "url": "http://www.sunyacc.edu/Library/", 
    "name": "Adirondack Community College", 
    "location": "Queensbury NY"
  }, 
  {
    "url": "http://www.acp.edu/academics_library.html", 
    "name": "Albany College of Pharmacy", 
    "location": "Albany NY"
  }, 
  {
    "url": "http://www.albertus.edu/academic/library.shtml", 
    "name": "Albertus Magnus College", 
    "location": "New Haven CT"
  }, 
  {
    "url": "http://www.albright.edu/library/", 
    "name": "Albright College", 
    "location": "Reading PA"
  }, 
  {
    "url": "http://www.herr.alfred.edu/", 
    "name": "Alfred University", 
    "location": "Alfred NY"
  }, 
  ...
]

To re-crawl LIBWEB you will need to install Python, lxml and then run libraries.py

License: CCO

Jump to Line
Something went wrong with that request. Please try again.