Skip to content
A viewer for GDE data
Python HTML JavaScript CSS Mako
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
api_example_scripts
assets
flask_app
frontend
geoprocessing
requirements
tilestache
.gitignore
README.md
requirements.txt
set_local_vars.sh.template

README.md

Groundwater dependent ecosystems (GDE) web application

Introduction

This repository contains the web application to publish the results of TNC's GDE-pulse project. The project has four parts (represented by directories within this repository).

  • geoprocessing: Scripts to generate data from upstream data sources

  • flask_app: The web app (backend or API) using Flask

  • frontend: A single page app (SPA) using ESRI's Javascript API

  • tilestache: Tilestache configuration to server raster data to the application

Documents and background materials

Requirements document: https://docs.google.com/document/d/1jq-g7XsYepUNjvrSz5AIJ9YYU5oY6tVGretLFCLgcjw/edit?usp=sharing

Mockup: https://docs.google.com/presentation/d/1NWLJaaxO4azQ2t7fw1SD6Y7M0bjmVHgNMiR8uqtyKK4/edit?usp=sharing

Project planning document: https://docs.google.com/presentation/d/17ZtkSpF1tX0vzya-B9GG0aXiWbjVh0eBxp1ebXmBbkg/edit?usp=sharing

https://github.com/earthlab/tutorials/blob/master/documentation/intro-google-earth-engine-python-api.md

Proposed hosting site: https://groundwaterresourcehub.org/

Manuskript folder https://drive.google.com/drive/u/0/folders/1Kw55i0C4ZHQtUu9paMT03iORP70KRubC

Ian's Earth Engine code: https://github.com/ihousman/TNC

Prototype by Ian Housman

https://github.com/ihousman/TNC.git

API Documentation

Assets stored in ArcGIS online

The application relies on two hosted feature services stored in ArcgisOnline:

Groundwater dependent ecosystems: https://services.arcgis.com/F7DSX1DSNSiWmOqh/arcgis/rest/services/gdes/FeatureServer

Measurement wells locations and attributes: https://services.arcgis.com/F7DSX1DSNSiWmOqh/arcgis/rest/services/wells/FeatureServer

Assets served by TileStache

Tiled raster layer are served by our own tileserver:

NDVI trend for California: https://tiles.codefornature.org/tiles/ndvitrend/

The same proudct masked by the GDE layer: https://tiles.codefornature.org/tiles/ndvitrend-masked/

The schema for integration into web maps is: https://tiles.codefornature.org/tiles/ndvitrend{-masked}/{z}/{x}/{y}.png

Multiple subdomain support for quicker loading is not implemented yet.

Data served by Flask API

GDE NDVI, NDMI, precipitation

The API endpoint for GDE related data extracted from GEE is:

https://gde-api.codefornature.org/api/gdes/{polygon_id}/

e.g. https://gde-api.codefornature.org/api/gdes/22234/

This will return a CSV file (text/csv) containing annual data bewteen 1985 and 2018:

  • polygon_id (int): Unique identifier of GDEs
  • year (int): Observation year
  • month (int): Month, 14 identifies annual summaries
  • ndvi (float): Normalized difference vegetation index, medoid of observations between day 190 and 250 of the year, derived from Landsat 5-7 composite, 30m data, zonal mean over polygon
  • ndmi (float): Normalized difference moisture index, medoid of observations between day 190 and 250 of the year, derived from Landsat 5-7 composite, 30m data, zonal mean over polygon
  • precip (int): Annual sum of preciptation from GridMet (2.5 degree data), zonal mean over polygon

Well Measurements

The API endpoint for groundwater levels measured at wells is

https://gde-api.codefornature.org/api/measurements/{well_id}/

e.g. http://gde-api.codefornature.org/api/measurements/48758/

This will return a CSV file (text/csv) containing annual data bewteen 1985 and 2018 whenever it has been measured:

  • stn_id (int): Unique identifier of observation wells
  • site_code (str): A code containing the location of the well
  • date (datetime): Measurement date
  • wse (float): Water surface elevation (over/under sea level)
  • gse_wse (float): Ground surface elevation - water surface elevation = depth to groundwater at the well location
  • quality_desc (str): Quality description
  • accuracy_desc (str): Accuracy description
  • comments (str)

See also https://github.com/tnc-ca-geo/gde/files/2866286/Well_measurement_schema.xlsx

Build directions

Please follow the build instructions to fully bootstrap the system. I am sure, I forgot a couple of steps and that ambiguities and different requirements for different systems will occur. Please file a GitHub issues or contact @postfalk if help needed.

Install system dependencies

Install GDAL 2 on your system:

Ubuntu >= 16: $ sudo apt install libgdal

OSX: $ brew install gdal

Skip the GDAL step if you use Anaconda or Miniconda for Python deps.

You also need to install Postgresql and PostGIS:

Create a role, a database and the PostGIS extension in Postgresql

$ psql postgres -c 'CREATE ROLE gde_user SUPERUSER LOGIN;'
$ psql postgres -c 'CREATE DATABASE groundwater OWNER gde_user;'
$ psql groundwater -c 'CREATE EXTENSION postgis;'

Install Python dependencies

On Windows systems it is highly recommended to use Conda. However, there is currently no conda_environment.yml in the repo due to the lack of time to test or maintain. Just follow along the pip requirements.txt file manually.

The build information below applies to unix-like systems in particular to OSX and Ubuntu.

If you do not use Conda, check your GDAL version:

$ gdalinfo --version
GDAL 2.4.0, released 2018/12/14

You need this information to manually install GDAL's Python-bindings like this (in our case 2.4 because of the info above):

$ pip install GDAL==2.4

You might compile Shapely and Fiona on your system to fully work with your GDAL installation, for this reason install your deps like this:

$ pip install --no-binary shapely,fiona  -r requirements.txt

Build Mapnik

Use Python 2.7 for mapnik since the Python 3 install is still a big pain. Don't even try! (TODO: add build instructions)

Bootstrap data

After activating your virtual environment and installing all dependencies, change into the geoprocessing directory.

All information to bootstrap the project is stored in the config.py file which also defines the workflow for generating the data.

The script

python evaluate_sources.py

will check whether a defined data source exists and attempt to create it if not. I am sure that this script will not run smoothly on the first attempt.

Following requisits need to be met:

  • You need a Google EarthEngine enabled Google account and your computer need to be authorized to interact with GEE, run earthengine authenticate

  • You need to be able to connect to the so-called "N Drive" or the CaroGIS server. The path where the drive is mounted to needs to be specified (unless it is N:). On OSX using Samba, e.g. export CAROGISROOT=/Volumes/GIS/. On OSX use Finder/Go/Connect to server to mount.

  • You need valid ESRI credentials to interact with ArcGIS online.

  • See set_local_vars.sh.template for environment variables to configure.

If you don't meet one or any of these requirements you can add the files defined in config.py from different sources manually.

The bootstrap process will take some time (2 hours) do to download speed and GEE rate limits.

Create ArcGIS Online assets

In order to move the processed data to AGO, create and style services, from the geoprocessing directory run

$ python publish_gde_layer.py
$ python publish_well_layer.py
$ python style_gde_layer.py
$ python style_well_layer.py

Copy GEE assets

In order to perform the data extraction from EE analysis we need to push our assets, the GDE shapes, to GEE. Since the limit for FeatureCollections on GEE is 5000 our original file has been broken up in 40 pieces that will be processed separately.

Google Cloud Storage needs to be activated within the same account used to interact with GEE. The default bucket name is gde_data and write access to this bucket is required. You can also change the bucket in the config.py file.

From the geoprocessing directory run

$ python copy_assets

Extract EE data

With the GEE assets in place and still from the geoprocessing folder run

$ python extract.py

Which will create the CSV files that can be loaded in the project database.

Load the data into the database (FLASK)

Switch to the flask_app folder and make sure that the DATABASE_URL and the FLASK_APP environmental variables are set:

$ export DATABASE_URL='postgresql://localhost:5432/groundwater'
$ export FLASK_DEBUG=True
$ export FLASK_APP=app.py
$ export FLASK_ENV=development

Now you should be able to run:

$ flask load_data
$ flask load_measurements

This will be slow and has only little feedback

Possible future developments

This is very interesting for building tileservers in the future https://github.com/azavea/lambda-geotrellis-tile-server/

Data dumps

echo "gde,year,ndvi,ndmi,precip" > gde-data-1.csv
psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde<=32000 ORDER BY gde,year;" >> gde-data-1.csv
echo "gde,year,ndvi,ndmi,precip" > gde-data-2.csv
psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde>32000 AND gde<=64000 ORDER BY gde,year;" >> gde-data-2.csv
echo "gde,year,ndvi,ndmi,precip" > gde-data-3.csv
psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde>64000 AND gde<=112000 ORDER BY gde,year;" >> gde-data-3.csv
echo "gde,year,ndvi,ndmi,precip" > gde-data-4.csv
psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde>112000  ORDER BY gde,year;" >> gde-data-4.csv
You can’t perform that action at this time.