NYPL Map Warper
This software has been developed by Topomancy LLC with the support and collaboration of the New York Public Library.
You can see this code in action at http://maps.nypl.org/.
Consult the wiki for for more documentation including a report detailing the changes in this version compared with the previous version. https://github.com/NYPL/nypl-warper/wiki
The NYPL Map Warper is an open source map geo-rectification, warping and georeferencing application. It enables a user to browse and explore the NYPL's historical map collection, and by placing control points on a reference map and the image, to warp it, to stretch it to fit.
The application can be seen in use at http://maps.nypl.org
The application is a web based crowdsourced geospatial project that enables people and organisations to collaboratively publish images of maps online and digitize and extract vector information from them.
Users rectify, warp or stretch images of historical maps with a reference basemap, assigning locations on image and map that line up with each other. Often these historical maps were in big paper books, and so for the first time they can be stitched together and shown as a whole, in digital format.
Users can crop around the maps, and join them together into layers.
By georeferencing the images, they can be warped or georectified to match the locations in space, and used in GIS software and other services. One such use of these warped maps is an application that that helps people digitize, that is, trace over the maps to extract information from them. For example, buildings in 18th Century Manhattan, details changing land use, building type etc. This application is called the Digitizer.
The application runs as a Ruby on Rails application using a number of open source geospatial libraries and technologies, including PostGIS, Mapserver, Geoserver, and GDAL tools.
The resulting maps can be exported as a PNG, GeoTIFF, WMS, Tiles, and KML for use in many different applications.
- Find and search maps by geography
- Adding control points to maps side by side
- Crop maps
- User commenting on maps
- Align maps from similar
- Login via Github / Twitter / etc
- OR signup with email and password
- Export as GeoTiff, PNG, WMS, Tile, KML etc
- Preview in Google Earth and Google Maps
- Map Favourites
- Social media sharing
- Bibliographic metatadata creation and export support
- Multiple georectfication options
- Control point from files import
- Admin tools include
- User statistics
- Activity monitoring
- User administration, disabling
- Roles management (editor, developer, admin etc)
- Batch Imports
- User Flags
- User Statistics
- Revisions and Rollbacks
Note on code and branches
Unmaintained branches exist for older systems and setups
- Rails 2.3 and Ruby 1.8.4 - See the rails2 branch
Ruby & Rails
- Rails 4.2
- Ruby 1.9+
- Postgresql 9.1 (or 9.1, 8.4)
- Postgis 2 (may work with 1.5)
Check out the Vagrant section lower down in the readme if you want to get started quickly.
on Ubuntu 14.04 LTS
apt-get install -y ruby ruby-dev postgresql-9.3-postgis-2.1 postgresql-server-dev-all postgresql-contrib build-essential git-core libxml2-dev libxslt-dev imagemagick libmapserver1 gdal-bin libgdal-dev ruby-mapscript nodejs
Due to a bug with the gdal gem, you may need to disable a few flags from your ruby rbconfig.rb see https://github.com/zhm/gdal-ruby/issues/4 for more information
Then install the gem files using bundler
Create and configure the following files
In addition have a look in
config/initializers/application_config.rb for some other paths and variables, and
config/initializers/devise.rb for devise and omniauth and also
config/environments/development.rb if you rather not use the
/warper relative path.
Create a postgis database
sudo -u postgres createdb mapwarper_development psql mapwarper_development -c "create extension postgis;" RAILS_ENV=development bundle exec rake db:migrate
Creating a new user
user = User.new user.login = "super" user.email = "email@example.com" user.password = "your_password" user.password_confirmation = "your_password" user.confirmed_at = Time.now user.save role = Role.find_by_name('super user') permission = Permission.new permission.role = role permission.user = user permission.save role = Role.find_by_name('administrator') permission = Permission.new permission.role = role permission.user = user permission.save
to get and install the virtual machine - this will also install the libraries and depencies and ruby gems for mapwarper into the virtual machine. See
/lib/vagrant/provision.sh for more details about this process.
After that runs, type
vagrant ssh to login and then you can:
cd /srv/mapwarper rails c
Create a user in the console, as shown above and then
exit. Then start the server in a relative url root use
RAILS_ENV=development bundle exec thin --prefix=/warper start
and then browse to
In non-Vagrant circumstances you may want to run it just using
Import a test map
This codebase only supports maps from NYPL via the Digital Collections API (use that link to get an account and token). Fortunately, many of our maps are Public Domain so you can test the warper locally (non PD maps are not downloadable in high-res from the API). The warper imports maps (subject to throttling limits) based on the
UUID visible in the item page (
98b7f36a-3d7a-6c65-e040-e00a18064b00 in this case):
- add the generated API token to
- start the server and log in with a super user
- go to
Admin / Imports
Add New Import
Mapand type in the
This will take a few minutes to pull the map. You now have a map you can work with and test functionality!
The system can use capistrano for deployment.
Making Code Changes and Deployment
There are two main changes to the system. Changes to checked in code, and changes to configuration files. Some configuration files (
/config/*) are not stored in the repository. To change these you need to ssh into the server and change the file. Some
.example files are provided for reference.
Changes to checked in code.
Develop, patch the code locally and get it running. Probably the easiest way to start developing is via Vagrant, see above for more details. Once you have Vagrant running, these are some useful commands to make sure updates are reflected:
If assets had been changed (images, JS, CSS etc.) or if you just want to be safe:
RAILS_ENV=production rake assets:clobber
RAILS_ENV=production rake assets:precompile
If necessary to run rails console commands:
RAILS_ENV=production rails console
To restart the server:
To restart the Apache server (not needed in almost all cases):
sudo service apache2 restart
Changes to config file:
- ssh into the server
- Find the desired config file
- Edit the file
- Restart the warper:
README_API.md for API details