Allow Mass Restoration of TR-55 Projects #3530

rajadain · 2022-04-21T18:04:18Z

Overview

We realized that the migration in #3528 did not fix the issue it was attempting to. This was because while we cleared the results and modification_hash fields, the aoi_census and modification_censuses were not, and with their cached values continued to have the same issue as before.

This PR includes an updated version of that migration that also clears those fields. Unfortunately, once those are gone, we realized that the code goes down a path not often trod:

model-my-watershed/src/mmw/apps/modeling/views.py

Lines 645 to 656 in afebb07

    
           polygons = [aoi] + [m['shape']['geometry'] for m in pieces] 
        
           geop_input = {'polygon': [json.dumps(p) for p in polygons]} 
        
           # Use WKAoI only if there are no pieces to modify the AoI 
        
           wkaoi = wkaoi if not pieces else None 
        
           job_chain.insert( 
        
               0, 
        
               geoprocessing.run.s('nlcd_soil', 
        
                                   geop_input, 
        
                                   wkaoi, 
        
                                   layer_overrides=layer_overrides)) 
        
           job_chain.append(tasks.run_tr55.s(aoi, model_input))

This code path is only exercised when neither the AoI census nor the modification censuses are defined, which almost never happens in regular user interaction. Usually, the AoI census is calculated when the Current Conditions scenario is initialized, and then copied over to New Scenarios. When adding modifications, only one is added at a time, and previously cached modification censuses are used otherwise. This is important because the current geoprocessing service only works on one polygon at a time, and if multiple polygons are provided, it unifies them into a single one and returns a combined result. The TR-55 code does expect multiple items, one for each polygon:

model-my-watershed/src/mmw/apps/modeling/tasks.py

Lines 254 to 259 in afebb07

    
           # Modification/BMP fragments and their censuses 
        
           # The original modifications are not POSTed. We only 
        
           # send the altered modifications/modification pieces. 
        
           modification_pieces = model_input.get('modification_pieces') 
        
           modification_censuses = (censuses[1:] if cached_aoi_census is None 
        
                                    else censuses[0:])

but our geoprocessing summary returned the single result wrapped in a simple list:

model-my-watershed/src/mmw/apps/modeling/tasks.py

Lines 225 to 228 in afebb07

    
           return [{ 
        
               'cell_count': total_count, 
        
               'distribution': dist, 
        
           }]

which works correctly as long as only a single polygon is queried. However, when multiple polygons are queried, the result is a combined histogram, whereas we want individualized ones.

Hence, in WikiWatershed/mmw-geoprocessing#101, we add a new operation RasterGroupedCountMany, which will be included in the upcoming 5.3.0 release of the geoprocessing service, that returns individualized histograms for each input polygon.

This PR switches TR-55 to use that new operation.

Notes

While this should be tested ASAP, this cannot be merged until WikiWatershed/mmw-geoprocessing#101 is merged and a 5.3.0 version of the geoprocessing service is tagged and released.

Testing Instructions

Preparation

The following is no longer needed as of the 5.3.0 release.

~~In a local install of mmw-geoprocessing, check out Add new operation supporting many input polygons mmw-geoprocessing#101, then run ./scripts/cibuild~~

~~Copy the generated JAR file to your model-my-watershed directory:~~

cp mmw-geoprocessing/api/target/scala-2.11/api-assembly-5.2.0.jar model-my-watershed/

~~Go into your worker VM, and replace the local JAR with this one:~~

vagrant ssh worker

sudo mv /vagrant/api-assembly-5.2.0.jar /opt/geoprocessing/mmw-geoprocessing-5.2.0.jar
sudo service mmw-geoprocessing stop && sudo service mmw-geoprocessing start

Instead, just reprovision the worker to get the latest mmw-geoprocessing JAR:

vagrant provision worker

Evaluation

On master, ensure you are logged in so that your work is saved
Make a TR-55 project with this shape: rivera.geojson.txt
Draw some modifications on it
- Ensure it fails
Check out this branch
Run migrations
```
./scripts/manage migrate
```

Restart Celery

vagrant ssh worker -c 'sudo service celeryd restart'

Go back to your saved project
- Ensure it succeeds

rajadain

Left some comments.

rajadain · 2022-04-21T18:08:47Z

src/mmw/mmw/settings/base.py

@@ -525,6 +525,19 @@ def get_env_setting(setting):
                'zoom': 0
            }
        },
+        'nlcd_soil_tr55': {


Decided to make a new one instead of changing the existing one because I wasn't sure if that was used elsewhere.

rajadain · 2022-04-21T18:12:32Z

src/mmw/apps/modeling/tasks.py

+    if 'error' in results:
+        raise Exception(f'[nlcd_soil_tr55] {results["error"]}')
+
+    return [nlcd_soil(result) for result in results]


This is just a simple wrapper around the old one, which runs it for every result in the output of RasterGroupedCountMany.

rajadain · 2022-04-21T18:12:59Z

deployment/ansible/group_vars/all

@@ -44,7 +44,7 @@ docker_compose_version: "1.26.*"
 geop_host: "localhost"
 geop_port: 8090

-geop_version: "5.2.0"
+geop_version: "5.3.0"


This may cause build failures until 5.3.0 is tagged and released for mmw-geoprocessing.

rajadain · 2022-04-21T18:13:25Z

src/mmw/apps/modeling/migrations/0040_clear_nlcd2019_tr55_results.py

-        modification_hash=''
+        modification_hash='',
+        aoi_census='{}',
+        modification_censuses='{}',


These need to be cleared as well for the scenarios to correctly re-run.

Since we're switching out the land cover layer, any cached censuses are invalid, and should be cleared out as well.

This ensures that multiple shapes can be analyzed at once, and their histograms are reported individually, rather than as a group, as was the case with RasterGroupedCount.

This operation takes multiple polygons and returns histograms for each of them. This is needed for TR-55, where multiple spatial modifications can exist on top of the area of interest, and need their own histograms.

rajadain · 2022-04-21T18:29:22Z

I've added 4b022a9 to get the CI to pass until the 5.3.0 release is available. Once it is, we should drop that commit before merging.

rajadain · 2022-04-25T20:37:44Z

I just released 5.3.0: https://github.com/WikiWatershed/mmw-geoprocessing/releases/tag/5.3.0. Going to remove that final commit.

Also, with this new release being available, all the steps in the Preparation section can be replaced with this one line command:

vagrant provision worker

emilyhu0106

I tested the fix between master and this branch. All modifications applied to the shape works well. Nice work!

rajadain · 2022-04-27T15:49:06Z

Thank you! Merging this now, will test on staging shortly.

rajadain added the NSF Funding Source: National Science Foundation label Apr 21, 2022

rajadain assigned emilyhu0106 Apr 21, 2022

rajadain requested a review from emilyhu0106 April 21, 2022 18:04

rajadain force-pushed the tt/switch-tr55-to-nlcd-2011 branch from d93dcb8 to 2f62bf6 Compare April 21, 2022 18:11

rajadain commented Apr 21, 2022

View reviewed changes

rajadain added 3 commits April 21, 2022 18:28

Also clear stored censuses for existing projects

f2f3376

Since we're switching out the land cover layer, any cached censuses are invalid, and should be cleared out as well.

Switch TR-55 to use RasterGroupedCountMany

8f0dd02

This ensures that multiple shapes can be analyzed at once, and their histograms are reported individually, rather than as a group, as was the case with RasterGroupedCount.

Update geoprocessing to support RasterGroupedCountMany

153d9b9

This operation takes multiple polygons and returns histograms for each of them. This is needed for TR-55, where multiple spatial modifications can exist on top of the area of interest, and need their own histograms.

rajadain force-pushed the tt/switch-tr55-to-nlcd-2011 branch from 2f62bf6 to 4b022a9 Compare April 21, 2022 18:28

rajadain force-pushed the tt/switch-tr55-to-nlcd-2011 branch from 4b022a9 to 153d9b9 Compare April 25, 2022 20:37

emilyhu0106 approved these changes Apr 27, 2022

View reviewed changes

rajadain merged commit 2671182 into hotfix/1.33.7 Apr 27, 2022

rajadain deleted the tt/switch-tr55-to-nlcd-2011 branch April 27, 2022 15:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow Mass Restoration of TR-55 Projects #3530

Allow Mass Restoration of TR-55 Projects #3530

rajadain commented Apr 21, 2022 •

edited

Loading

rajadain left a comment

rajadain Apr 21, 2022

rajadain Apr 21, 2022

rajadain Apr 21, 2022

rajadain Apr 21, 2022

rajadain commented Apr 21, 2022

rajadain commented Apr 25, 2022

emilyhu0106 left a comment

rajadain commented Apr 27, 2022

	polygons = [aoi] + [m['shape']['geometry'] for m in pieces]
	geop_input = {'polygon': [json.dumps(p) for p in polygons]}
	# Use WKAoI only if there are no pieces to modify the AoI
	wkaoi = wkaoi if not pieces else None

	job_chain.insert(
	0,
	geoprocessing.run.s('nlcd_soil',
	geop_input,
	wkaoi,
	layer_overrides=layer_overrides))
	job_chain.append(tasks.run_tr55.s(aoi, model_input))

	# Modification/BMP fragments and their censuses
	# The original modifications are not POSTed. We only
	# send the altered modifications/modification pieces.
	modification_pieces = model_input.get('modification_pieces')
	modification_censuses = (censuses[1:] if cached_aoi_census is None
	else censuses[0:])

Allow Mass Restoration of TR-55 Projects #3530

Allow Mass Restoration of TR-55 Projects #3530

Conversation

rajadain commented Apr 21, 2022 • edited Loading

Overview

Notes

Testing Instructions

Preparation

Evaluation

rajadain left a comment

Choose a reason for hiding this comment

rajadain Apr 21, 2022

Choose a reason for hiding this comment

rajadain Apr 21, 2022

Choose a reason for hiding this comment

rajadain Apr 21, 2022

Choose a reason for hiding this comment

rajadain Apr 21, 2022

Choose a reason for hiding this comment

rajadain commented Apr 21, 2022

rajadain commented Apr 25, 2022

emilyhu0106 left a comment

Choose a reason for hiding this comment

rajadain commented Apr 27, 2022

rajadain commented Apr 21, 2022 •

edited

Loading