Groundwater dependent ecosystems (GDE) web application
This repository contains the web application to publish the results of TNC's GDE-pulse project. The project has four parts (represented by directories within this repository).
geoprocessing: Scripts to generate data from upstream data sources
flask_app: The web app (backend or API) using Flask
tilestache: Tilestache configuration to server raster data to the application
Documents and background materials
Project planning document: https://docs.google.com/presentation/d/17ZtkSpF1tX0vzya-B9GG0aXiWbjVh0eBxp1ebXmBbkg/edit?usp=sharing
Proposed hosting site: https://groundwaterresourcehub.org/
Ian's Earth Engine code: https://github.com/ihousman/TNC
Prototype by Ian Housman
Assets stored in ArcGIS online
The application relies on two hosted feature services stored in ArcgisOnline:
Groundwater dependent ecosystems: https://services.arcgis.com/F7DSX1DSNSiWmOqh/arcgis/rest/services/gdes/FeatureServer
Measurement wells locations and attributes: https://services.arcgis.com/F7DSX1DSNSiWmOqh/arcgis/rest/services/wells/FeatureServer
Assets served by TileStache
Tiled raster layer are served by our own tileserver:
NDVI trend for California: https://tiles.codefornature.org/tiles/ndvitrend/
The same proudct masked by the GDE layer: https://tiles.codefornature.org/tiles/ndvitrend-masked/
The schema for integration into web maps is:
Multiple subdomain support for quicker loading is not implemented yet.
Data served by Flask API
GDE NDVI, NDMI, precipitation
The API endpoint for GDE related data extracted from GEE is:
This will return a CSV file (
text/csv) containing annual data bewteen 1985 and 2018:
- polygon_id (int): Unique identifier of GDEs
- year (int): Observation year
- month (int): Month, 14 identifies annual summaries
- ndvi (float): Normalized difference vegetation index, medoid of observations between day 190 and 250 of the year, derived from Landsat 5-7 composite, 30m data, zonal mean over polygon
- ndmi (float): Normalized difference moisture index, medoid of observations between day 190 and 250 of the year, derived from Landsat 5-7 composite, 30m data, zonal mean over polygon
- precip (int): Annual sum of preciptation from GridMet (2.5 degree data), zonal mean over polygon
The API endpoint for groundwater levels measured at wells is
This will return a CSV file (
text/csv) containing annual data bewteen 1985 and 2018 whenever it has been measured:
- stn_id (int): Unique identifier of observation wells
- site_code (str): A code containing the location of the well
- date (datetime): Measurement date
- wse (float): Water surface elevation (over/under sea level)
- gse_wse (float): Ground surface elevation - water surface elevation = depth to groundwater at the well location
- quality_desc (str): Quality description
- accuracy_desc (str): Accuracy description
- comments (str)
Please follow the build instructions to fully bootstrap the system. I am sure, I forgot a couple of steps and that ambiguities and different requirements for different systems will occur. Please file a GitHub issues or contact @postfalk if help needed.
Install system dependencies
Install GDAL 2 on your system:
Ubuntu >= 16:
$ sudo apt install libgdal
$ brew install gdal
Skip the GDAL step if you use Anaconda or Miniconda for Python deps.
You also need to install Postgresql and PostGIS:
Create a role, a database and the PostGIS extension in Postgresql
$ psql postgres -c 'CREATE ROLE gde_user SUPERUSER LOGIN;' $ psql postgres -c 'CREATE DATABASE groundwater OWNER gde_user;' $ psql groundwater -c 'CREATE EXTENSION postgis;'
Install Python dependencies
On Windows systems it is highly recommended to use Conda. However, there is currently no conda_environment.yml in the repo due to the lack of time to test or maintain. Just follow along the pip requirements.txt file manually.
The build information below applies to unix-like systems in particular to OSX and Ubuntu.
If you do not use Conda, check your GDAL version:
$ gdalinfo --version GDAL 2.4.0, released 2018/12/14
You need this information to manually install GDAL's Python-bindings like this (in our case 2.4 because of the info above):
$ pip install GDAL==2.4
You might compile Shapely and Fiona on your system to fully work with your GDAL installation, for this reason install your deps like this:
$ pip install --no-binary shapely,fiona -r requirements.txt
Use Python 2.7 for mapnik since the Python 3 install is still a big pain. Don't even try! (TODO: add build instructions)
After activating your virtual environment and installing all dependencies, change into the geoprocessing directory.
All information to bootstrap the project is stored in the
which also defines the workflow for generating the data.
will check whether a defined data source exists and attempt to create it if not. I am sure that this script will not run smoothly on the first attempt.
Following requisits need to be met:
You need a Google EarthEngine enabled Google account and your computer need to be authorized to interact with GEE, run
You need to be able to connect to the so-called "N Drive" or the CaroGIS server. The path where the drive is mounted to needs to be specified (unless it is N:). On OSX using Samba, e.g.
export CAROGISROOT=/Volumes/GIS/. On OSX use Finder/Go/Connect to server to mount.
You need valid ESRI credentials to interact with ArcGIS online.
See set_local_vars.sh.template for environment variables to configure.
If you don't meet one or any of these requirements you can add the files defined in config.py from different sources manually.
The bootstrap process will take some time (2 hours) do to download speed and GEE rate limits.
Create ArcGIS Online assets
In order to move the processed data to AGO, create and style services, from the geoprocessing directory run
$ python publish_gde_layer.py $ python publish_well_layer.py $ python style_gde_layer.py $ python style_well_layer.py
Copy GEE assets
In order to perform the data extraction from EE analysis we need to push our assets, the GDE shapes, to GEE. Since the limit for FeatureCollections on GEE is 5000 our original file has been broken up in 40 pieces that will be processed separately.
Google Cloud Storage needs to be activated within the same account used to interact with GEE. The default bucket name is gde_data and write access to this bucket is required. You can also change the bucket in the config.py file.
From the geoprocessing directory run
$ python copy_assets
Extract EE data
With the GEE assets in place and still from the geoprocessing folder run
$ python extract.py
Which will create the CSV files that can be loaded in the project database.
Load the data into the database (FLASK)
Switch to the flask_app folder and make sure that the DATABASE_URL and the FLASK_APP environmental variables are set:
$ export DATABASE_URL='postgresql://localhost:5432/groundwater' $ export FLASK_DEBUG=True $ export FLASK_APP=app.py $ export FLASK_ENV=development
Now you should be able to run:
$ flask load_data $ flask load_measurements
This will be slow and has only little feedback
Possible future developments
This is very interesting for building tileservers in the future https://github.com/azavea/lambda-geotrellis-tile-server/
echo "gde,year,ndvi,ndmi,precip" > gde-data-1.csv psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde<=32000 ORDER BY gde,year;" >> gde-data-1.csv echo "gde,year,ndvi,ndmi,precip" > gde-data-2.csv psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde>32000 AND gde<=64000 ORDER BY gde,year;" >> gde-data-2.csv echo "gde,year,ndvi,ndmi,precip" > gde-data-3.csv psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde>64000 AND gde<=112000 ORDER BY gde,year;" >> gde-data-3.csv echo "gde,year,ndvi,ndmi,precip" > gde-data-4.csv psql -d groundwater -t -A -F "," -c "SELECT gde,year,ndvi,ndmi,precip FROM gde_data WHERE gde>112000 ORDER BY gde,year;" >> gde-data-4.csv