MrGeo is a geospatial toolkit designed to provide raster-based geospatial capabilities that can be performed at scale. MrGeo is built upon Apache Spark and the Hadoop ecosystem to leverage the storage and processing of hundreds of commodity computers. See the wiki for more details.
Java Scala Python Shell Jupyter Notebook Scheme
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
color-scales Fixed green-yellow-red-quantiles Nov 21, 2017
distribution Initial commit of Mapbox vector tiles vector data provider Dec 14, 2017
logo Add logo and sources May 5, 2016
mrgeo-cmd Initial commit of Mapbox vector tiles vector data provider Dec 14, 2017
mrgeo-core Support for MB tiles data provider, including tests Jan 3, 2018
mrgeo-dataprovider Address problems after rasterizing a road network. Jan 8, 2018
mrgeo-mapalgebra Initial commit of Mapbox vector tiles vector data provider Dec 14, 2017
mrgeo-python Make sure mapops are in the proper objects Oct 25, 2017
mrgeo-services Clean up color scale code Nov 21, 2017
scripts Support CDH 5.14.2 Apr 16, 2018
src/main/findbugs Initial commit of Mapbox vector tiles vector data provider Dec 14, 2017
.gitignore Update .gitignore Dec 5, 2016
Jenkinsfile Create Jenkinsfile Dec 14, 2016
Jenkinsfile.geoint removed sudo from fpm command Feb 13, 2017
LICENSE Initial commit Nov 21, 2014
NOTICE Update NOTICE Feb 11, 2015
README.md Update README.md Dec 5, 2016
about.yml Add files via upload Oct 13, 2016
build noop commit to trigger Jenkins Mar 16, 2015
clean initial checkin Nov 25, 2014
deploy changes to allow "-yarn" to be added to build type, not just "-y" Apr 11, 2016
eclipse initial checkin Nov 25, 2014
integration initial checkin Nov 25, 2014
pom.xml Support CDH 5.14.2 Apr 16, 2018
rebuild initial checkin Nov 25, 2014
shade initial checkin Nov 25, 2014
test initial checkin Nov 25, 2014
verify initial checkin Nov 25, 2014

README.md

Apache
Version 2.6.0 2.7.1
Status Build Status Build Status
CDH
Version 5.5.2 5.6.0 5.7.1
Status Build Status Build Status Build Status
RPM pyMrGeo
Version 1.1.0 0.0.7
Status Build Status Build Status
Amazon EMR
Version 4.7.1 5.0.0
Status Build Status Build Status

##Origin

Join the chat at https://gitter.im/ngageoint/mrgeo Hex.pm

MrGeo was developed at the National Geospatial-Intelligence Agency (NGA) in collaboration with DigitalGlobe. The government has "unlimited rights" and is releasing this software to increase the impact of government investments by providing developers with the opportunity to take things in new directions. The software use, modification, and distribution rights are stipulated within the Apache 2.0 license.

###Pull Requests

All pull request contributions to this project will be released under the Apache 2.0 license.

Software source code previously released under an open source license and then modified by NGA staff is considered a "joint work" (see 17 USC 101); it is partially copyrighted, partially public domain, and as a whole is protected by the copyrights of the non-government authors and must be released according to the terms of the original open source license.

###MrGeo in Action See YouTube explainer

###MrGeo in the News NGA Press Release

DigitalGlobe Press Release

MrGeo got a mention in an article on NGA Opensourcing from Reuters

###MrGeo Overview

MrGeo (pronounced "Mister Geo") is an open source geospatial toolkit designed to provide raster-based geospatial processing capabilities performed at scale. MrGeo enables global geospatial big data image processing and analytics.

MrGeo is built upon the Apache Spark distributed processing frarmework to leverage the storage and processing of 100’s of commodity computers. Functionally, MrGeo stores large raster datasets as a collection of individual tiles stored in Hadoop to enable large-scale data and analytic services. The co-location of data and analytics offers the advantage of minimizing the movement of data in favor of bringing the computation to the data; a more favorable compute method for Geospatial Big Data. This framework has enabled the servicing of terabyte scale raster databases and performed terrain analytics on databases exceeding 100’s of gigabytes in size.

MrGeo has been fully deployed and tested in Amazon EMR.

See Wiki for detailed documentation

Unique features/solutions of MrGeo:

  • Scalable storage and processing of raster data
  • Application ready data: data is stored in MrGeo in a format that is ready for computation, eliminating several data pre-processing steps from production workflows.
  • A suite of robust Spark analytics that that include algebraic math operations, focal operations (e.g. slope and gaussian)
  • A third generation data storage model that
    • Maintains data locality via spatial indexing.
    • An abstraction layer between the analytics and storage methods to enables a diverse set of cloud storage options such as HDFS, Accumulo, HBASE etc.
  • A Map algebra interface that enables the development of custom algorithms in a simple scripting API
  • A plugin architecture that facilitates a modular software development and deployment strategies
  • Data and Analytic capabilities provisioned by OGC and REST service end points

Exemplar MrGeo Use Cases:

  • Raster Storage and Provisioning: MrGeo has been used to store, index, tile, and pyramid multi-terabyte scale image databases. Once stored, this data is made available through a simple Tiled Map Services (TMS) and Web Mapping Services (WMS) and can be made available through GeoServer via a MrGeo plugin.
  • Large Scale Batch Processing and Serving: MrGeo has been used to pre-compute global 1 ArcSecond (nominally 30 meters) elevation data (300+ GB) into derivative raster products : slope, aspect, relative elevation, terrain shaded relief (collectively terabytes in size), and Tobler and Pingel friction surfaces
  • Global Computation of Cost Distance: Given all pub locations in OpenStreetMap, compute 2 hour drive times from each location. The full resolution is 1 ArcSecond (30 meters nominally)