From 2fede45d404b0f647c2d18bc0dba2d3682912915 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Tue, 7 May 2024 13:05:55 -0400 Subject: [PATCH 01/10] Fix bug and raster resolution --- src/delineate_hydros_and_produce_HAND.sh | 2 +- src/run_unit_wb.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/delineate_hydros_and_produce_HAND.sh b/src/delineate_hydros_and_produce_HAND.sh index 05667dd90..7f5a5a578 100755 --- a/src/delineate_hydros_and_produce_HAND.sh +++ b/src/delineate_hydros_and_produce_HAND.sh @@ -242,7 +242,7 @@ if [ -f $tempHucDataDir/osm_bridges_subset.gpkg ]; then -t $tempCurrentBranchDataDir/rem_zeroed_masked_$current_branch_id.tif \ -r 10 else - -e $startDiv"No applicable bridge data for $hucNumber" + echo -e $startDiv"No applicable bridge data for $hucNumber" fi diff --git a/src/run_unit_wb.sh b/src/run_unit_wb.sh index 03129bfa3..9505de749 100755 --- a/src/run_unit_wb.sh +++ b/src/run_unit_wb.sh @@ -119,7 +119,7 @@ echo -e $startDiv"Clipping rasters to branches $hucNumber $branch_zero_id" [ ! -f $tempCurrentBranchDataDir/dem_meters.tif ] && \ gdalwarp -cutline $tempHucDataDir/wbd_buffered.gpkg -crop_to_cutline -ot Float32 -r bilinear -of "GTiff" \ -overwrite -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "TILED=YES" -co "COMPRESS=LZW" \ - -co "BIGTIFF=YES" -t_srs $huc_CRS $input_DEM $tempHucDataDir/dem_meters.tif + -co "BIGTIFF=YES" -t_srs $huc_CRS -tr 10 10 $input_DEM $tempHucDataDir/dem_meters.tif Tcount From 280e471353026d505413ffc0768294859c099247 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Tue, 7 May 2024 13:29:51 -0400 Subject: [PATCH 02/10] Update CHANGELOG.md --- docs/CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 754885541..f8972e254 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,6 +1,20 @@ All notable changes to this project will be documented in this file. We follow the [Semantic Versioning 2.0.0](http://semver.org/) format. +## v4.5.x.x - 2024-05-07 - [PR#1150](https://github.com/NOAA-OWP/inundation-mapping/pull/1150) + +Fixes two bugs discovered in v4.4.5.0: +1. `echo` missing in bash command +2. raster resolution of `dem_meters.tif` has now been explicitly set to 10 m in `gdalwarp`. + +### Changes + +- `src/` + - `delineate_hydros_and_produce_HAND.sh`: added missing `echo` + - `run_unit_wb.sh`: `-tr` added to gdalwarp when generating `dem_meters.tif` + +

+ ## v4.5.0.0 - 2024-05-06 - [PR#1122](https://github.com/NOAA-OWP/inundation-mapping/pull/1122) This PR includes 2 scripts to add Open Street Map bridge data into the HAND process: a script that pulls data from OSM and a script that heals those bridges in the HAND grids. Both scripts should be run as part of a pre-processing step for FIM runs. They only need to be run if we think OSM data has changed a lot or for any new FIM versions. From 5c7b1f22505c7e8914d9d850ac636bb74eb82a1e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CRobHanna-NOAA=E2=80=9D?= <“Robert.Hanna@NOAA.gov”> Date: Tue, 7 May 2024 18:34:00 +0000 Subject: [PATCH 03/10] update PR template --- .github/PULL_REQUEST_TEMPLATE.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index f9265baef..3bcbdf0fa 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -12,11 +12,11 @@ _How does the changes affect the product?_ - [ ] Code only? -- [ ] Require new or adjusted data inputs. Does it have start, end and duration code (in UTC)? -- [ ] If new or updated data updates, has FIM related code been updated and tested? +- [ ] Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)? +- [ ] If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)? - [ ] Require new pre-clip set? - [ ] Has new or updated python packages? -- [ ] If applicable, is the deployment plan be calculated and work with Deployment person/team? +- [ ] If applicable, has a deployment plan be created with the deployment person/team? ### Issuer Checklist (For developer use) From 0407f974891cf0023c9852af5c335077996635f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Tue, 7 May 2024 14:38:03 -0400 Subject: [PATCH 04/10] Add res environment variable --- docs/CHANGELOG.md | 5 +++-- src/bash_variables.env | 5 ++++- src/delineate_hydros_and_produce_HAND.sh | 2 +- src/run_unit_wb.sh | 4 ++-- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f8972e254..c33688e03 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -5,13 +5,14 @@ We follow the [Semantic Versioning 2.0.0](http://semver.org/) format. Fixes two bugs discovered in v4.4.5.0: 1. `echo` missing in bash command -2. raster resolution of `dem_meters.tif` has now been explicitly set to 10 m in `gdalwarp`. +2. raster resolution of `dem_meters.tif` has now been explicitly set in `gdalwarp`. ### Changes - `src/` + - `bash_variables.env`: added `res` environment variable for default raster cell size - `delineate_hydros_and_produce_HAND.sh`: added missing `echo` - - `run_unit_wb.sh`: `-tr` added to gdalwarp when generating `dem_meters.tif` + - `run_unit_wb.sh`: added `-tr` to gdalwarp when generating `dem_meters.tif`

diff --git a/src/bash_variables.env b/src/bash_variables.env index ff242a9cc..42513fad8 100644 --- a/src/bash_variables.env +++ b/src/bash_variables.env @@ -1,8 +1,11 @@ ## Define inputs -# NOTE: $inputsDir is defined in Dockerfile +export res=10 # Default raster cell size in meters + export DEFAULT_FIM_PROJECTION_CRS=EPSG:5070 export ALASKA_CRS=EPSG:3338 + +# NOTE: $inputsDir is defined in Dockerfile export input_DEM=${inputsDir}/3dep_dems/10m_5070/fim_seamless_3dep_dem_10m_5070.vrt export input_DEM_Alaska=${inputsDir}/3dep_dems/10m_South_Alaska/23_11_07/FIM_3dep_dem_South_Alask_10m.vrt export input_DEM_domain=${inputsDir}/3dep_dems/10m_5070/HUC6_dem_domain.gpkg diff --git a/src/delineate_hydros_and_produce_HAND.sh b/src/delineate_hydros_and_produce_HAND.sh index 7f5a5a578..379761f1a 100755 --- a/src/delineate_hydros_and_produce_HAND.sh +++ b/src/delineate_hydros_and_produce_HAND.sh @@ -240,7 +240,7 @@ if [ -f $tempHucDataDir/osm_bridges_subset.gpkg ]; then -s $tempHucDataDir/osm_bridges_subset.gpkg \ -p $tempCurrentBranchDataDir/bridge_lines_raster_$current_branch_id.tif \ -t $tempCurrentBranchDataDir/rem_zeroed_masked_$current_branch_id.tif \ - -r 10 + -r $res else echo -e $startDiv"No applicable bridge data for $hucNumber" fi diff --git a/src/run_unit_wb.sh b/src/run_unit_wb.sh index 9505de749..c54333934 100755 --- a/src/run_unit_wb.sh +++ b/src/run_unit_wb.sh @@ -119,13 +119,13 @@ echo -e $startDiv"Clipping rasters to branches $hucNumber $branch_zero_id" [ ! -f $tempCurrentBranchDataDir/dem_meters.tif ] && \ gdalwarp -cutline $tempHucDataDir/wbd_buffered.gpkg -crop_to_cutline -ot Float32 -r bilinear -of "GTiff" \ -overwrite -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "TILED=YES" -co "COMPRESS=LZW" \ - -co "BIGTIFF=YES" -t_srs $huc_CRS -tr 10 10 $input_DEM $tempHucDataDir/dem_meters.tif + -co "BIGTIFF=YES" -t_srs $huc_CRS -tr $res $res $input_DEM $tempHucDataDir/dem_meters.tif Tcount ## GET RASTER METADATA echo -e $startDiv"Get DEM Metadata $hucNumber $branch_zero_id" -read fsize ncols nrows ndv xmin ymin xmax ymax cellsize_resx cellsize_resy\ +read fsize ncols nrows ndv xmin ymin xmax ymax cellsize_resx cellsize_resy \ <<<$($srcDir/getRasterInfoNative.py $tempHucDataDir/dem_meters.tif) ## RASTERIZE NLD MULTILINES ## From 088fe352efa9c40f3225f345de3f2ecd34dd7b5e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CRobHanna-NOAA=E2=80=9D?= <“Robert.Hanna@NOAA.gov”> Date: Tue, 7 May 2024 18:55:31 +0000 Subject: [PATCH 05/10] udpate changelog --- docs/CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f8972e254..6c00a7164 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,9 +1,9 @@ All notable changes to this project will be documented in this file. We follow the [Semantic Versioning 2.0.0](http://semver.org/) format. -## v4.5.x.x - 2024-05-07 - [PR#1150](https://github.com/NOAA-OWP/inundation-mapping/pull/1150) +## v4.5.0.1 - 2024-05-07 - [PR#1150](https://github.com/NOAA-OWP/inundation-mapping/pull/1150) -Fixes two bugs discovered in v4.4.5.0: +Fixes two bugs discovered in v4.5.0.0: 1. `echo` missing in bash command 2. raster resolution of `dem_meters.tif` has now been explicitly set to 10 m in `gdalwarp`. From fcf5fdfbd3ed4af919163d9aa3e98551457f19ed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Wed, 8 May 2024 09:22:06 -0400 Subject: [PATCH 06/10] Ignore max_order if less than output_flows --- src/add_crosswalk.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/add_crosswalk.py b/src/add_crosswalk.py index 2331e4c65..84bf74ff3 100755 --- a/src/add_crosswalk.py +++ b/src/add_crosswalk.py @@ -244,6 +244,16 @@ def add_crosswalk( (output_flows.From_Node == to_node) & (output_flows['order_'] == max_order) ]['HydroID'].item() + # output_flows has a higher order than the max_order + elif output_flows.loc[(output_flows.From_Node == to_node), 'order_'].max() > max_order: + update_id = output_flows.loc[ + (output_flows.From_Node == to_node) + & ( + output_flows['order_'] + == output_flows.loc[(output_flows.From_Node == to_node), 'order_'].max() + ) + ]['HydroID'].values[0] + # Get the first one # Same stream order, without drainage area info it is hard to know which is the main channel. else: From 47c6c438de8c6019721fd434d258a71142c465b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Wed, 8 May 2024 10:48:30 -0400 Subject: [PATCH 07/10] Ignore max_order if no output_flows of that order --- src/add_crosswalk.py | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/add_crosswalk.py b/src/add_crosswalk.py index 84bf74ff3..ddc9444df 100755 --- a/src/add_crosswalk.py +++ b/src/add_crosswalk.py @@ -257,9 +257,19 @@ def add_crosswalk( # Get the first one # Same stream order, without drainage area info it is hard to know which is the main channel. else: - update_id = output_flows.loc[ - (output_flows.From_Node == to_node) & (output_flows['order_'] == max_order) - ]['HydroID'].values[0] + if max_order in output_flows.loc[output_flows.From_Node == to_node, 'order_'].values: + update_id = output_flows.loc[ + (output_flows.From_Node == to_node) & (output_flows['order_'] == max_order) + ]['HydroID'].values[0] + + else: + update_id = output_flows.loc[ + (output_flows.From_Node == to_node) + & ( + output_flows['order_'] + == output_flows.loc[output_flows.From_Node == to_node, 'order_'].max() + ) + ]['HydroID'].values[0] # no upstream segments; single downstream segment elif len(output_flows.loc[output_flows.From_Node == to_node]['HydroID']) == 1: From c2eb61e4748de2812e9959cbc65652fd24899e47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Wed, 8 May 2024 13:04:20 -0400 Subject: [PATCH 08/10] Remove extraneous Tcount --- src/run_unit_wb.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/src/run_unit_wb.sh b/src/run_unit_wb.sh index c54333934..348b5518e 100755 --- a/src/run_unit_wb.sh +++ b/src/run_unit_wb.sh @@ -121,7 +121,6 @@ gdalwarp -cutline $tempHucDataDir/wbd_buffered.gpkg -crop_to_cutline -ot Float32 -overwrite -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -co "TILED=YES" -co "COMPRESS=LZW" \ -co "BIGTIFF=YES" -t_srs $huc_CRS -tr $res $res $input_DEM $tempHucDataDir/dem_meters.tif -Tcount ## GET RASTER METADATA echo -e $startDiv"Get DEM Metadata $hucNumber $branch_zero_id" From 5901cdf7026674f783e4d2f174a6287b77474d3f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CMattLuck-NOAA=E2=80=9D?= Date: Wed, 8 May 2024 16:22:07 -0400 Subject: [PATCH 09/10] Fix raster resolution and number of rows/columns --- docs/CHANGELOG.md | 3 ++- src/heal_bridges_osm.py | 8 ++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 14c4b4a7e..a8caace58 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -12,7 +12,8 @@ Fixes two bugs discovered in v4.5.0.0: - `src/` - `bash_variables.env`: added `res` environment variable for default raster cell size - `delineate_hydros_and_produce_HAND.sh`: added missing `echo` - - `run_unit_wb.sh`: added `-tr` to gdalwarp when generating `dem_meters.tif` + - `heal_bridges_osm.py`: fixed raster resolution and number of rows/columns + - `run_unit_wb.sh`: added `-tr` to gdalwarp when generating `dem_meters.tif`; removed extraneous `Tcount`

diff --git a/src/heal_bridges_osm.py b/src/heal_bridges_osm.py index bd72a481d..288572fc5 100644 --- a/src/heal_bridges_osm.py +++ b/src/heal_bridges_osm.py @@ -52,8 +52,8 @@ def process_bridges_in_huc( ########### setup new raster to save bridge max hand values ############# bbox = hand_grid.bounds xmin, ymin, xmax, ymax = bbox - w = (xmax - xmin) // resolution - h = (ymax - ymin) // resolution + w = hand_grid.shape[1] + h = hand_grid.shape[0] out_meta = { "driver": "GTiff", @@ -74,7 +74,7 @@ def process_bridges_in_huc( shapes = ((geom, value) for geom, value in zip(osm_gdf.geometry, osm_gdf.max_hand)) # burn in values to any pixel that's touched by polygon and add nodata fill value burned = features.rasterize( - shapes=shapes, fill=-999999, out=out_arr, transform=out.transform, all_touched=True + shapes=shapes, fill=-999999, out=out_arr, transform=hand_grid.transform, all_touched=True ) out.write_band(1, burned) @@ -138,7 +138,7 @@ def process_bridges_in_huc( help='OPTIONAL: Resolution of HAND grid. Default value is 10m', required=False, default=10, - type=int, + type=float, ) args = vars(parser.parse_args()) From 0c1edbfc2b38174596b8960c12a82ff14cdf3068 Mon Sep 17 00:00:00 2001 From: Rob Hanna - NOAA <90854818+RobHanna-NOAA@users.noreply.github.com> Date: Thu, 9 May 2024 10:44:40 -0600 Subject: [PATCH 10/10] Update CHANGELOG.md --- docs/CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a8caace58..c904fef02 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,7 +1,7 @@ All notable changes to this project will be documented in this file. We follow the [Semantic Versioning 2.0.0](http://semver.org/) format. -## v4.5.0.1 - 2024-05-07 - [PR#1150](https://github.com/NOAA-OWP/inundation-mapping/pull/1150) +## v4.5.0.1 - 2024-05-09 - [PR#1150](https://github.com/NOAA-OWP/inundation-mapping/pull/1150) Fixes two bugs discovered in v4.5.0.0: 1. `echo` missing in bash command @@ -10,6 +10,7 @@ Fixes two bugs discovered in v4.5.0.0: ### Changes - `src/` + - `add_crosswalk.py`: fixed stream order if max > `max_order` - `bash_variables.env`: added `res` environment variable for default raster cell size - `delineate_hydros_and_produce_HAND.sh`: added missing `echo` - `heal_bridges_osm.py`: fixed raster resolution and number of rows/columns