Challenge #3 - Open source tool for GIS-based dissolve functionalities

[jwagemann]

Challenge 3

Open source tool for GIS-based dissolve functionalities for the European Flood Awareness System (EFAS)

Software development.

Goal: To make the process of adding shapefiles from partners of EFAS more efficient

Mentors	@calumbaugh , Damien Decremer, @latimau
Skills required	- GIS–knowledge of working with OSGeo compliant vector data - Python – handling geospatial vector data via osgeo/shapely modules

---

Challenge description

What data/system(s) should be used?

The European Flood Awareness System (EFAS) is run at ECMWF on behalf of the Copernicus Emergency Management Service. Flood forecasts are available to those hydro-meteorological/civil protection agencies who are partners, currently there are 74 EFAS partner institutions. When a new partner is added, ECMWF receives a geospatial polygon, currently in ESRI Shapefile format, which delineates their area of interest. This is intersected with a shapefile of river basins to show when flood notifications need to be sent to this new partner. Finally the new polygon is merged into a web layer which shows the partner coverage of EFAS.

What is the current problem/limitation?

The problem is that the above procedure is very manual, time consuming and is difficult to train staff members to utilise it. Each step is performed separately using a combination of QGIS and Python. The most time consuming step is the merging of the polygon into the web layer. QGIS’ dissolve function is temperamental and frequently fails. Sometimes due to the presence of geometric errors in the new polygon. The manual corrections may often take hours. Sometimes, the dissolve tool does not seamlessly combine polygon boundaries, requiring even more manual editing which can take up to a day. Even then, the dissolve tool still doesn’t always work and we resort to sending a copy to our colleagues at the Joint Research Centre who use the equivalent function in ArcGIS which works fine. ECMWF does not have ArcGIS, so we need an open source equivalent of a reliable Dissolve tool.

What could be the solution?

The solution would be a tool which takes the input partner area polygon, automatically intersects with the river basin network and then merges the polygon with the web layer.

Possible idea for the implementation:

A tool to clean the input partner polygon to remove any geometric errors e.g. using osgeo or shapely modules in Python.
Intersect the new partner polygon with a shapefile of river basins. Identify which river basins are intersected, select the EFAS computational river cells within these basins and assign the new partner ID to these cells. Again, this can be done using osgeo/shapely in Python, existing scripts at ECMWF could be recycled.
Merge the new partner polygon into the EFAS web layer which shows the spatial coverage of partners. Create a Dissolve tool which dissolves based on an attribute value. The tool will need to correct for any remaining geometric errors but also detect and correct for any gaps or overlaps which appear at boundaries. Osgeo/Shapely could again be used. The resulting data will be saved in ESRI Shapefile format.

[varmar05]

We @lutraconsulting are interested in participating in the challenge and have two possible directions for solution. One could be a web application with simple UI to upload input layer, while processing happens at server and output is already stored in database, with option to export result to different format. Another one could be desktop application based on QGIS where user however needs to maintain the latest version of all layers of interest. The latter will thus involve more manual steps.

Could you please add some more comments about your overall workflow (where data comes from, whether there is centralized source of 'river basin network' and what would be the final use of output layer, e.g. publishing in webmaps) so we can decide which will better fit your needs?
Thank you.

[calumbaugh]

Hi @varmar05 thank you for getting in touch about the project. There are two main workflows that are involved

1. Dissolving new partner region into the web layer which displays all current EFAS partner regions. This is stored as an ESRI shapefile on a webserver which is then used to display the partner regions on the EFAS website using mapserver. So to update this file I:

• retrieve a copy of the shapefile from the webserver onto my desktop machine
• open it up within QGIS along with a shapefile of the new partner region.
• attempt to use the dissolve function based on shared attribute values within both shapefiles
• perform any necessary manual corrections using QGIS vector editor
• save the updated partner regions shapefile and re-upload it to the webserver

2. Intersect with new partners with river basins. We have shapefiles of the river basins, EFAS partner regions and 5km river channel points (we run a grid based river model at 5km resolution, this shapefile represents the location of every 5km river grid cell) which are stored on a shared drive. These are then queried by our operational flood forecast system twice a day (00 and 12 UTC). The update workflow is:

• Within QGIS add the new partner region polygon into the shapefile with all the current EFAS partner regions (note, this is different to the one used for the website because we don't dissolve the boundaries)
• Intersect the new partner region polygon with the river basins polygon and select all river basins which the partner region falls within
• Use these selected river basins to select all the 5km channel points which lie within them
• Update the selected 5km channel points - there is an attribute field which stores the ID of the relevant EFAS partner, so we add the new partner ID to this attribute.
• Save all the shapefiles and make sure they are updated on the shared drive
  I have been able to put most of this workflow into a Python script using osgeo and/or shapely. One thing to point out is that we're in the process of replacing the shapefile of 5km river channel points with a GeoSpatial database in postgres, so I don't know if this would affect any potential future workflow.

I hope the above information helps, please let me know if you have any further questions.

[jwagemann]

REMINDER: Deadline to register and submit your proposal is upcoming Sunday, 21 April at 23:59 GMT!

Application process is a 2-step process:

• You have to first register here, and then
• Submit a proposal under "Call for abstract" here.

Applications without a submitted proposal will not be taken under consideration!
We are looking forward to your proposal!

[calebjuma27]

Hallo,
Do we have to submit the tool alongside the proposal beforehand and if so which channel should be used?
Thank you

[jwagemann]

Hi @calebjuma27 ,
the idea is that you develop the tool during the coding period between May and August. The proposal should contain your idea and how you would like to develop the tool. You can of course link to any developed tool / applications within your proposal.
HTH,
Julia