[6pt] HUC 4,6,8 parameterization #4
Comments
BrianAvant
pushed a commit
that referenced
this issue
Sep 15, 2020
-slope raster is now derived using dem_meters.tif instead of burn_filled_dem.tif -fixes bug in split_flows.py -related to issue #4
|
This should be split into three tasks:
|
BrianAvant
changed the title
This was referenced Sep 22, 2020
Closed
|
There are two objectives with this task and both may not necessarily need to be completed immediately.
With the recent decrease in processing time to run the entire NWM domain on a HUC 8 scale, the first objective may be less important going forward. There are several issues that need to be addressed to complete these objectives. Objective 1
Objective 2
|
fernando-aristizabal
added a commit
that referenced
this issue
Sep 25, 2020
commit d8bca1b Author: fernando-aristizabal <fernando.aristizabal@noaa.gov> Date: Thu Sep 17 15:38:33 2020 +0000 Adds help instructions for preprocessing. Added preprocessing usage and USGS NHDPlusHR reference One last try for indentation on Readme. Need to find a tool to preview locally. My bad for extra commits Patches indentation on README Patches to Readme file Updates to Readme * Additional usage * Testing information * Credits and References Patches non-zero exit code in 18050002 * Duplicate NHDPlusID found in this HUC * Yields error in snap_and_clip... module * Now catching and handling this edge case error Github Issue #8 HUC input handling * now searches HUC inputs for membership in included_huc*.lst files * throws error if not present Github Issue #8 Patches issue where slope raster was derived from burn filled dem -slope raster is now derived using dem_meters.tif instead of burn_filled_dem.tif -fixes bug in split_flows.py -related to issue #4
nickchadwick-noaa
pushed a commit
that referenced
this issue
Sep 25, 2020
commit c5674cf Author: fernando-aristizabal <fernando-aristizabal@noaa.gov> Date: Thu Sep 17 15:38:33 2020 +0000 Adds help instructions for preprocessing. Added preprocessing usage and USGS NHDPlusHR reference One last try for indentation on Readme. Need to find a tool to preview locally. My bad for extra commits Patches indentation on README Patches to Readme file Updates to Readme * Additional usage * Testing information * Credits and References Patches non-zero exit code in 18050002 * Duplicate NHDPlusID found in this HUC * Yields error in snap_and_clip... module * Now catching and handling this edge case error Github Issue #8 HUC input handling * now searches HUC inputs for membership in included_huc*.lst files * throws error if not present Github Issue #8 Patches issue where slope raster was derived from burn filled dem -slope raster is now derived using dem_meters.tif instead of burn_filled_dem.tif -fixes bug in split_flows.py -related to issue #4
This was referenced Sep 30, 2020
Merged
Merged
nickchadwick-noaa
pushed a commit
that referenced
this issue
Oct 8, 2020
* aggregate_nhd_hr_streams.py creates CONUS NWM headwaters and NHDplus HR layers
* snap_and_clip_to_nhd.py adjusted to read in CONUS input layers
* filter_catchments_...py uses HUC8 fossid to select catchments; used to define HUC boundary
* enforces int dtype for feature_id and HydroID in add_crosswalk.py
* updates output filenames in run_by_unit.sh, test_run_case.py, and aggregate_fim_outputs.sh
* some issues remain with incoming streams #58
This addresses Github issue #42; Related to #4
BrianAvant
added a commit
that referenced
this issue
Oct 9, 2020
-slope raster is now derived using dem_meters.tif instead of burn_filled_dem.tif -fixes bug in split_flows.py -related to issue #4
This was referenced Dec 15, 2020
nickchadwick-noaa
pushed a commit
that referenced
this issue
Jan 21, 2021
* headwater stream segments geometries are adjusted to align with with NWM streams * incoming streams are selected using intersection points between NWM streams and HUC4 boundaries * clip_vectors_to_wbd.py handles local headwaters * removes NHDPlus features categorized as coastline and underground conduit * added streams layer to production whitelist * fixed progress bar in lib/acquire_and_preprocess_inputs.py * added getDriver to shared functions.py * cleaned up variable names and types This addresses Github issues #58, #12, #118, #73, and partial resolution of #4
nickchadwick-noaa
pushed a commit
that referenced
this issue
Feb 4, 2021
Additions * viz outputs aggregated to HUC6 scale; saves outputs to aggregate_fim_outputs folder Changes * split_flows.py now splits streams at HUC8 boundaries to ensure consistent catchment boundaries along edges * aggregate_fim_outputs.sh has been depreciated but remains in the repo for potential FIM 4 development * replaced geopandas driver arg with getDriver throughout repo * organized parameters in environment files by group * cleaned up variable names in split_flows.py and build_stream_traversal.py * build_stream_traversal.py is now assigning HydroID by midpoint instead centroid * cleanup of clip_vectors_to_wbd.py resolves #161 and #4
CarsonPruitt-NOAA
added a commit
that referenced
this issue
Apr 14, 2023
* Fixed Typo Fixed small typo in add_crosswalk.py * Modifications to build and run Docker image more reliably Modifications to build and run Docker image more reliably. Closes #178. Changes Changed to noninteractive install of grass. Fixes #182. Notes aggregate_vector_inputs.py doesn't work yet. Need to externally download required data to run fim_run.sh * hotfix to address AHPSs mapping errors (#200) * handle dytpe and leading zeros for hydrotable * exit inundation.py when all stream segments are lakes * Update README.md Added to reference list * Update CHANGELOG.md * Hotfix for handling nodata value in rasterized levee lines - Resolves bug for HUCs where $ndv > 0 (Great Lakes region) - Initialize the nld_rasterized_elev.tif using a value of -9999 instead of $ndv * Changed the directory where the included_huc*.lst files are being read from. Changed the directory where the `included_huc*.lst` files are being read from. * Preprocess MS and FR stream networks * headwater stream segments geometries are adjusted to align with with NWM streams * incoming streams are selected using intersection points between NWM streams and HUC4 boundaries * clip_vectors_to_wbd.py handles local headwaters * removes NHDPlus features categorized as coastline and underground conduit * added streams layer to production whitelist * fixed progress bar in lib/acquire_and_preprocess_inputs.py * added getDriver to shared functions.py * cleaned up variable names and types This addresses Github issues #58, #12, #118, #73, and partial resolution of #4 * Initial Version of the FIM API Addition of an API service to schedule, run and manage fim_run jobs through a user-friendly web interface. * Added api folder that contains all the codebase for the new service. * Updated point of contact on README.md * Updated README.md credits Updated README.md with Esri Arc Hydro credit to acknowledge the knowledge gained and discoveries made through collaboration with Esri. * Post-process to aggregate FIM outputs to HUC6 scale Additions * viz outputs aggregated to HUC6 scale; saves outputs to aggregate_fim_outputs folder Changes * split_flows.py now splits streams at HUC8 boundaries to ensure consistent catchment boundaries along edges * aggregate_fim_outputs.sh has been depreciated but remains in the repo for potential FIM 4 development * replaced geopandas driver arg with getDriver throughout repo * organized parameters in environment files by group * cleaned up variable names in split_flows.py and build_stream_traversal.py * build_stream_traversal.py is now assigning HydroID by midpoint instead centroid * cleanup of clip_vectors_to_wbd.py resolves #161 and #4 * Minor bug fixes Changes - corrected variable name in fim_run.sh - acquire_and_preprocess_inputs.py now creates huc_lists folder and updates file file path Fixes #252 * New python script "wrappers" for using inundation.py. - Created inundation_wrapper_nwm_flows.py to produce inundation outputs using NWM recurrence flows: 1.5 year, 5 year, 10 year. - Created inundation_wrapper_custom_flow.py to produce inundation outputs with user-created flow file. - Created new tools parent directory to store inundation_wrapper_nwm_flows.py and inundation_wrapper_custom_flow.py. This resolves #220. * Update README.md * Update README.md * Updated MS Crosswalk method to address gaps in FIM. - Fixed typo in stream midpoint calculation in split_flows.py and add_crosswalk.py. - add_crosswalk.py now restricts the MS crosswalk to NWM MS catchments. - add_crosswalk.py now performs a secondary MS crosswalk selection by nearest NWM MS catchment. This resolves #232. * Addresses issue when running on HUC6 scale * src should be fixed and slightly smaller by removing whitespace * rasters are about the same size as running fim as huc6 (compressed and tiled; aggregated are slightly larger) * naming convention and feature id attribute are only added to the aggregated hucs (this bullet and the next one assume that Viz will not use the huc8 layer outputs) * HydroIDs are different for huc6 vs aggregated huc8s mostly due to forced split at huc boundaries (so long we use consistent workflow it shouldn't matter) * known issue where sometimes an incoming stream is not included in the final selection (resolves #238) will affect aggregate outputs * Modified rem.py with a new function to output HAND reference elev - Function "make_catchment_hydroid_dict" creates a df of pixel catchment ids and overlapping hydroids - Merge hydroid df and thalweg minimum elevation df - Produces new output containing all catchment ids and min thalweg elevation value → “hand_ref_elev_table.csv” - Overwrites the demDerived_reaches_split.gpk layer --> adding additional attribute: “Min_Thal_Elev_meters” to view the elevation value for each hydroid * Rating curves for short stream segments are replaced with rating curves from upstream/downstream segments. - Short stream segments are identified and are reassigned the channel geometry from upstream/downstream segment. - fossid renamed to fimid and the attribute's starting value is now 1000 to avoid HydroIDs with leading zeroes. - Addresses issue where HydroIDs were not included in final hydrotable. - Added import sys to inundation.py (missing from previous feature branch). - Variable names and general workflow are cleaned up. This resolves #100. * Enhancements to allow for evaluation at AHPS sites, the generation of a query-optimized metrics CSV, and the generation of categorical FIM. Enhancements to allow for evaluation at AHPS sites, the generation of a query-optimized metrics CSV, and the generation of categorical FIM. This merge requires that the `/test_cases` directory be updated for all machines performing evaluation. - `generate_categorical_fim.py` was added to allow production of NWS Flood Categorical HAND FIM (CatFIM) source data. More changes on this script are to follow in subsequent branches. - `ble_autoeval.sh` and `all_ble_stats_comparison.py` were deleted because `synthesize_test_cases.py` now handles the merging of metrics. - The code block in `run_test_case.py` that was responsible for printing the colored metrics to screen has been commented out because of the new scale of evaluations (formerly in `run_test_case.py`, now in `shared_functions.py`) - Remove unused imports from inundation wrappers in `/tools`. - Updated `synthesize_test_cases.py` to allow for AHPS site evaluations. - Reorganized `run_test_case.py` by moving more functions into `shared_functions.py`. - Created more shared variables in `shared_variables.py` and updated import statements in relevant scripts. This resolves #258 and #259. * Removed "_new" from test_cases dir * Update CHANGELOG.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Adding HAND SRC datum elev values to hydroTable.csv output (#272) - Adding HAND SRC datum elev values to hydroTable.csv output - Renamed hydroid attribute to "Median_Thal_Elev_m" * Bug fixes to new evaluation code and README.md cleanup - Fixed a bug in synthesize_test_cases.py where the extent (MS/FR) was not being written to merged metrics file properly. - Fixed a bug in synthesize_test_cases.py where only BLE test cases were being written to merged metrics file. - Removed unused imports from inundation.py. - Updated README.md This resolves #270. * Update CHANGELOG.md * Enhancement that creates metric plots and summary statistics. Enhancement that creates metric plots and summary statistics using metrics compiled by synthesize_test_cases.py. Additions - Added eval_plots.py, which produces: - Boxplots of CSI, FAR, and POD/TPR - Barplot of aggregated CSI scores - Scatterplot of CSI comparing two FIM versions - CSV of aggregated statistics (CSI, FAR, POD/TPR) - CSV of analyzed data and analyzed sites This resolves #70. * Restructured the repository Restructured the repository. This has no impact on hydrological code and simply moves files and renaming directories. - Moved the contents of the lib folder to a new folder called src. - Moved the contents of the tests folder to the tools folder. - Changed any instance of lib or libDir to src or srcDir This resolves #283. * Renaming of benchmark data and associated variables in evaluation scripts Renamed benchmark layers in test_cases and updated variable names in evaluation scripts. - Updated run_test_case.py with new benchmark layer names. - Updated run_test_case_calibration.py with new benchmark layer names. This resolves #284. * Update README.md Updates the move from tests/ to tools/synthesize_test_cases.py * Convert CatFIM pipeline to open source Refactored CatFIM pipeline to open source. - Added VIZ_PROJECTION to shared_variables.py. - Added missing library referenced in inundation.py. - Cleaned up and converted evaluation scripts in generate_categorical_fim.py to open source. - Removed util folders under tools directory. This resolves #279. * [1pt] PR: Patch import in tools_shared_functions.py (#302) Minor patch to tools_shared_functions.py where tools_shared_functions.py was not imported correctly. * Enhancements to post-processing for Viz-related use-cases. - Aggregate grids are projected to Web Mercator during -v runs in fim_run.sh. - HUC6 aggregation is parallelized. - Aggregate grid blocksize is changed from 256 to 1024 for faster postprocessing. This resolves #294 and resolves #295. * Additional Enhancements for the FIM API Enhancements to FIM API. - fim_run.sh can now be run with jobs in parallel. - Viz post-processing can now be selected in API interface. - Jobs table shows jobs that end with errors. - HUC preset lists can now be selected in interface. - Better output_handler file writing. - Overall better restart and retry handlers for networking problems. - Jobs can now be canceled in API interface. - Both FR and MS configs can be selected for a single job. Resolves issues #231 and #265 * Preprocessing of flow files for Categorical FIM. - Generate Categorical FIM flow files for each category (action, minor, moderate, major). - Generate point shapefile of Categorical FIM sites. - Generate csv of attribute data in shapefile. - Aggregate all shapefiles and csv files into one file in parent directory. - Add flood of record category. - Stability fixes to generate_categorical_fim.py. This resolves #243 and resolves #274. * Patch to synthesize_test_cases.py Bug fix to synthesize_test_cases.py to allow comparison between testing version and official versions. * Improvements to CatFIM service source data generation Improvements to CatFIM service source data generation. - Renamed generate_categorical_fim.py to generate_categorical_fim_mapping.py. - Updated the status outputs of the nws_lid_sites layer and saved it in the same directory as the merged catfim_library layer. - Additional stability fixes (such as improved compatability with WRDS updates). - Added generate_categorical_fim.py to wrap generate_categorical_fim_flows.py and generate_categorical_fim_mapping.py. - Create new nws_lid_sites shapefile located in same directory as the catfim_library shapefile. This resolves #305, resolves #321, and resolves #322. * Update CHANGELOG.md * Adds detail and more information to plotting capabilities Add detail/information to plotting capabilities. - Merge plot_functions.py into eval_plots.py and move eval_plots.py into the tools directory. - Remove plots subdirectory. - Optional argument to create barplots of CSI for each individual site. - Create a csv containing the data used to create the scatterplots. This resolves #325. * Update README.md * Update spatial option when performing eval plots Removes file dependencies from spatial option. Does require the WBD layer which should be specified in .env file. - Produces outputs in a format consistent with requirements needed for publishing. - Preserves leading zeros in huc information for all outputs from eval_plots.py. - Creates fim_performance_points.shp: this layer consists of all evaluated ahps points (with metrics). Spatial data retrieved from WRDS on the fly. - Creates fim_performance_polys.shp: this layer consists of all evaluated huc8s (with metrics). Spatial data retrieved from WBD layer. This resolves #325. * Update CHANGELOG.md * [1pt] Tool to compare synthetic rating curve with benchmark rating curve (sierra test) (#332) * `usgs_gage_crosswalk.py`: generates `usgs_elev_table.csv` in run_by_unit.py with elevation at USGS gages * `rating_curve_comparison.py`: post-processing script to plot and calculate metrics between FIM and USGS rating curve data * updates `aggregate_fim_outputs.py` call argument in `fim_run.sh` from 4 jobs to 6 jobs (optimizing API performance) * reroutes median elevation data from `add_crosswalk.py` and `rem.py` to new file (depreciating `hand_ref_elev_table.csv`) *- adds new files to `viz_whitelist` in `output_cleanup.py` * Create tool to retrieve rating curves from USGS sites and convert to elevation (NAVD88). Intended to be used as part of the Sierra Test. - Modify usgs_gage_crosswalk.py to: - Look for location_id instead of site_no attribute field in usgs_gages.gpkg file. - Filter out gages that do not have rating curves included in the usgs_rating_curves.csv - Modify rating_curve_comparison.py to perform a check on the age of the user specified usgs_rating_curves.csv and alert user to the age of the file and recommend updating if file is older the 30 days. - Add rating_curve_get_usgs_curves.py. This script will generate the following files: - usgs_rating_curves.csv: A csv file that contains rating curves (including converted to NAVD88 elevation) for USGS gages in a format that is compatible with rating_curve_comparisons.py. As it is is currently configured, only gages within CONUS will have rating curve data. - log.csv: A log file that records status for each gage and includes error messages. - usgs_gages.gpkg: A geospatial layer (in FIM projection) of all active USGS gages that meet a predefined criteria. Additionally, the curve attribute indicates whether a rating curve is found in the usgs_rating_curves.csv. This spatial file is only generated if the all option is passed with the -l argument. This resolves #289. * Implementing a prototype technique to estimate the missing bathymetric component in the HAND-derived synthetic rating curves. Implementing a prototype technique to estimate the missing bathymetric component in the HAND-derived synthetic rating curves. The new Bathymetric Adjusted Rating Curve (BARC) function is built within the fim_run.sh workflow and will ingest bankfull geometry estimates provided by the user to modify the cross section area used in the synthetic rating curve generation. - add_crosswalk.py outputs the stream order variables to src_full_crosswalked.csv and calls the new bathy_rc_adjust.py if bathy env variable set to True and extent=MS. - run_by_unit.sh includes a new csv outputs for reviewing BARC calculations. - params_template.env & params_calibrated.env contain new BARC function input variables and on/off toggle variable. eval_plots.py now includes additional AHPS eval sites in the list of "bad_sites" (flagged issues with MS flowlines). - Adds bathy_rc_adjust.py - Imports the existing synthetic rating curve table and the bankfull geometry input data (topwidth and cross section area per COMID). - Performs new synthetic rating curve calculations with bathymetry estimation modifications. - Flags issues with the thalweg-notch artifact. This resolves #306, resolves #328, resolves #309, resolves #311, and resolves #330. * Add mainstems attribute for sierra test Sierra test considered all USGS gage locations to be mainstems even though many actually occurred with tributaries. This resulted in unrealistic comparisons as incorrect gages were assigned to mainstems segments. This feature branch identifies gages that are on mainstems via attribute field. - Modifies usgs_gage_crosswalk.py to filter out gages from the usgs_gages.gpkg layer such that for a "MS" run, only consider gages that contain rating curve information (via curve attribute) and are also mainstems gages (via mainstems attribute). - Modifies usgs_gage_crosswalk.py to filter out gages from the usgs_gages.gpkg layer such that for a "FR" run, only consider gages that contain rating curve information (via curve attribute) and are not mainstems gages (via mainstems attribute). - Modifies how mainstems segments are determined by using the nwm_flows_ms.gpkg as a lookup to determine if the NWM segment specified by WRDS for a gage site is a mainstems gage. - Adds a mainstem attribute field to usgs_gages.gpkg that indicates whether a gage is located on a mainstems river. - Adds NWM_FLOWS_MS variable to the .env and .env.template files. - Adds the extent argument specified by user when running fim_run.sh to usgs_gage_crosswalk.py. * Fixes production list in output_cleanup.py. Fixes production list in output_cleanup.py. * Preprocess NHDPlus HR rasters for consistent projections, nodata values, and convert from cm to meters. Preprocess NHDPlus HR rasters for consistent projections, nodata values, and convert from cm to meters. - preprocess_rasters.py reprojects raster, converts to meters, and updates nodata value to -9999. - Cleaned up log messages from bathy_rc_adjust.py and usgs_gage_crosswalk.py. - Outputs paths updated in generate_categorical_fim_mapping.py and generate_categorical_fim.py. - update_raster_profile cleans up raster crs, blocksize, nodata values, and converts elevation grids from cm to meters. - reproject_dem.py imports gdal to reproject elevation rasters because an error was occurring when using rasterio. - burn_in_levees.py replaces the gdal_calc.py command to resolve inconsistent outputs with burned in levee values. This resolves #300. * Closing multiprocessing pool objects Updating repo to close all multiprocessing Pool objects to fix memory issues * Update plotting script to not error out if FR or MS are not analyzed when using spatial argument update eval_plots.py to correctly produce spatial datasets if version is not analyzed. For example, if no FR versions are analyzed, instead of throwing an error, it will check to make sure there exists a FR analyzed datasets, if one exists then create spatial data for FR datasets (ble). * Implement CatFIM threshold flows to Sierra test and add AHPS benchmark preprocessing scripts. Implement CatFIM threshold flows to Sierra test and add AHPS benchmark preprocessing scripts. - Produce CatFIM flows file when running rating_curve_get_usgs_gages.py. - Several scripts to preprocess AHPS benchmark data. Requires numerous file dependencies not available through Cahaba. - Modify rating_curve_comparison.py to ingest CatFIM threshold flows in calculations. - Modify eval_plots.py to save all site specific bar plots in same parent directory instead of in subdirectories. - Add variables to env.template for AHPS benchmark preprocessing. This resolves #341, resolves #361, and resolves #357. * Refactor synthesize_test_case.py to handle exceptions during multiprocessing Refactor synthesize_test_case.py to handle exceptions during multiprocessing. Resolves issue #351 Changes - refactored inundation.py and run_test_case.py to handle exceptions without using sys.exit(). This isn't the ideal way to handle exceptions when using multiprocessing because the child process is expecting a return. * Refactor NHDPlus HR preprocessing workflow Refactor NHDPlus HR preprocessing workflow. - Consolidate NHD streams, NWM catchments, and headwaters MS and FR layers with mainstem column. - HUC8 intersections are included in the input headwaters layer. - clip_vectors_to_wbd.py removes incoming stream segment from the selected layers. This resolves #238. * Generate nws_lid.gpkg Generate nws_lid.gpkg. - Generate nws_lid.gpkg with attributes indicating if site is a headwater nws_lid as well as if it is co-located with another nws_lid which is referenced to the same nwm_feature_id segment. This resolves #368. * Remove Great Lakes from wbd buffer - The gl_water_polygons.gpkg layer is used to mask out Great Lakes boundaries and remove NHDPlus HR coastline segments. These segments are causing issues later in run_by_unit.sh and unnecessarily increasing total processing time. Resolves issue #374 * Added fuctionality for new "Release" feature in FIM API New "Release" feature added to the FIM API. This feature will allow for automated FIM, CatFIM, and relevant metrics to be generated when a new FIM Version is released. See #373 for more detailed steps that take place in this feature. Additions - Added new window to the UI in api/frontend/gui/templates/index.html. - Added new job type to api/node/connector/connector.py to allow these release jobs to run. - Added additional logic in api/node/updater/updater.py to run the new eval and CatFIM scripts used in the release feature. Changes - Updated api/frontend/output_handler/output_handler.py to allow for copying more broad ranges of file paths instead of only the /data/outputs directory. Resolves Issues #264 #278 #307 and #373 * Updated Eventlet package to 0.31.0 to patch a vulnerability The current version of Eventlet used in the Connector module of the FIM API is outdated and vulnerable. This update bumps the version to the patched version. Using new nhd inputs. * Bug fixes in preprocessing and run_test_case.py and update eval_plots Modifications and fixes to run_test_case.py, eval_plots.py, and AHPS preprocessing scripts. - Comment out return statement causing run_test_case.py to skip over sites/hucs when calculating contingency rasters. - Move bad sites list and query statement used to filter out bad sites to the tools_shared_variables.py. - Add print statements in eval_plots.py detailing the bad sites used and the query used to filter out bad sites. - Update AHPS preprocessing scripts to produce a domain shapefile. - Change output filenames produced in ahps preprocessing scripts. - Update workarounds for some sites in ahps preprocessing scripts. This resolves #381, resolves #369, resolves #326, resolves #385, resolves #377. * Update CHANGELOG.md * Alpha test bug fixes and improvements Enhancements and bug fixes to synthesize_test_cases.py. - Addresses a bug where AHPS sites without benchmark data were receiving a CSI of 0 in the master metrics CSV produced by synthesize_test_cases.py. - Includes a feature enhancement to synthesize_test_cases.py that allows for the inclusion of user-specified testing versions in the master metrics CSV. - Removes some of the print statements used by synthesize_test_cases.py. This resolves #354 and resolves #386. * BARC updates to cap xsec area and allow user to choose input bankfull geometry BARC updates to cap the bathy calculated xsec area in bathy_rc_adjust.py and allow user to choose input bankfull geometry. - Added new env variable to control which input file is used for the bankfull geometry input to bathy estimation workflow. - Modified the bathymetry cross section area calculation to cap the additional area value so that it cannot exceed the bankfull cross section area value for each stream segment (bankfull value obtained from regression equation dataset). - Modified the rating_curve_comparison.py plot output to always put the FIM rating curve on top of the USGS rating curve (avoids USGS points covering FIM). - Created a new aggregate csv file (aggregates for all hucs) for all of the usgs_elev_table.csv files (one per huc). - Evaluate the FIM Bathymetry Adjusted Rating Curve (BARC) tool performance using the estimated bankfull geometry dataset derived for the NWM route link dataset. This resolves #342, resolves #343, resolves #313, resolves #401, and resolves #402. * Update CHANGELOG.md * Correct headwater assignment nws_lid layer Bug fix to the generate_nws_lid.py script - Fixes incorrectly assigned attribute field "is_headwater" for some sites in the nws_lid.gpkg layer. - Updated agg_nhd_headwaters_adj.gpkg, agg_nhd_streams_adj.gpkg, nwm_flows.gpkg, and nwm_catchments.gpkg input layers using latest NWS LIDs. This resolves #383 and resolves #397. * Fim Run Memory Profiling To help analyze the memory consumption of the Fim Run process, the python module memory-profiler has been added to give insights into where peak memory usage is with in the codebase. In addition, the Dockerfile was previously broken due to the Taudem dependency removing the version that was previously being used by FIM. To fix this, and allow new docker images to be built, the Taudem version has been updated to the newest version on the Github repo and thus needs to be thoroughly tested to determine if this new version has affected the overall FIM outputs. Resolves #392 and #403 Additions - Added memory-profiler to Pipfile and Pipfile.lock. - Added mprof (memory-profiler cli utility) call to the time_and_tee_run_by_unitsh to create overall memory usage graph location in the /logs/{HUC}_memory.png of the outputs directory. - Added @Profile decorator to all functions within scripts used in the run_by_unit.sh script to allow for memory usage tracking, which is then recorded in the /logs/{HUC}.log file of the outputs directory. Changes - Changed the Taudem version in Dockerfile.dev to 98137bb6541a0d0077a9c95becfed4e56d0aa0ac. - Changed all calls of python scripts in run_by_unit.sh to be called with the -m memory-profiler argument to allow scripts to also track memory usage. * Feature to evaluate alternative CatFIM technique performance Added eval_catfim_alt.py to evaluate performance of alternative CatFIM techniques. This resolves #414. * Thalweg Profile Tool and lateral Thalweg Adjustment Threshold Adding a thalweg profile tool to identify significant drops in thalweg elevation. Also setting lateral thalweg adjustment threshold in hydroconditioning. - thalweg_drop_check.py checks the elevation along the thalweg for each stream path downstream of MS headwaters within a HUC. - Removing dissolveLinks arg from clip_vectors_to_wbd.py. - Cleaned up code in split_flows.py to make it more readable. - Refactored reduce_nhd_stream_density.py and adjust_headwater_streams.py to limit MS headwater points in agg_nhd_headwaters_adj.gpkg. - Fixed a bug in adjust_thalweg_lateral.py lateral elevation replacement threshold; changed threshold to 3 meters. - Updated aggregate_vector_inputs.py to log intermediate processes. This resolves #408, resolves #18, and resolves #409. * Allowance of evaluation at IFC sites and updating of bad eval sites list Updates to evaluation scripts to allow for Alpha testing at Iowa Flood Center (IFC) sites. Also, BAD_SITES variable updates to omit sites not suitable for evaluation from metric calculations. - The BAD_SITES list in tools_shared_variables.py was updated and reasons for site omission are documented. - Refactored run_test_case.py, synthesize_test_cases.py, tools_shared_variables.py, and eval_plots.py to allow for IFC comparisons. This resolves #424, resolves #412, and resolves #428. * Update synthesize_test_cases.py Bug fix to if/else statement block * Update CHANGELOG.md * Updating logging and fixing bug in vector preprocessing Updating logging and fixing bug in vector preprocessing. - fim_completion_check.py adds message to docker log to log any HUCs that were requested but did not finish run_by_unit.sh. - Adds input_data_edits_changelog.txt to the inputs folder to track any manual or version/location specific changes that were made to data used in FIM 3. - Provides unique exit codes to relevant domain checkpoints within run_by_unit.sh. - Bug fixes in reduce_nhd_stream_density.py, 'mprof plot` call. - Improved error handling in add_crosswalk.py. This resolves #416 and resolves #427. * Updating Readme (#435) * Adds two QAQC scripts to tools directory This merge adds two new scripts into /tools/ for use in QAQC. Adds inundate_nation.py to produce inundation maps for the entire country for use in QAQC. Adds check_deep_flooding.py to check for depths of inundation greater than a user-supplied threshold at specific areas defined by a user-supplied shapefile. This resolves #439 and resolves #438. * Documentation of various scripts This merge better documents the following scripts: - inundate_nation.py - synthesize_test_cases.py - adjust_thalweg_lateral.py - rem.py This resolves #441. * Adding check for NWM stream segments within HUC boundaries This merge modifies clip_vectors_to_wbd.py to check for relevant input data. - clip_vectors_to_wbd.py now checks that there are NWM stream segments within the buffered HUC boundary. - included_huc8_ms.lst has several additional HUC8s. * Cahaba Codebase Cleanup - Small Items General repository cleanup, made memory-profiling an optional flag, API's release feature now saves outputs. - Remove Dockerfile.prod, rename Dockerfile.dev to just Dockerfile, and remove ``.dockerignore`. - Clean up Dockerfile and remove any unused* packages or variables. - Remove any unused* Python packages from the Pipfile. - Move the CHANGELOG.md, SECURITY.md, and TERMS.md files to the /docs folder. - Remove any unused* scripts in the /tools and /src folders. - Move tools/preprocess scripts into tools/. - Ensure all scripts in the /src folder have their code in functions and are being called via a __main__ function (This will help with implementing memory profiling fully). - Changed memory-profiling to be an option flag -m for fim_run.sh. - Updated FIM API to save all outputs during a "release" job. This resolves #432, resolves #426, and resolves #434. * [1pt] PR: Function to extract HAND and hydroid values at point locations for SRC adjustment tool (#444) This merges in the script, adjust_rc_with_feedback.py, that will be expanded in future issues. The primary function that performs the HAND value and hydroid extraction is ingest_points_layer() but this may change as the overall process evolves. - Added adjust_rc_with_feedback.py with ingest_points_layer(), a function to extract HAND and hydroid values for use in an automatic synthetic rating curve updating mechanism. This resolves #437. * Patch params_template.env Changes max_split_distance_meters in params_template.env to 1500. This resolves #446. * Update CHANGELOG.md * Updated README with ESIP/AWS information * Update README.md * Update README.md * Update README.md * Update README.md * Disable BARC for specified stream order This merge addresses an issues with the bathymetry adjusted rating curve (BARC) calculations exacerbating single-pixel inundation issues for the lower Mississippi River. This fix allows the user to specify a stream order value that will be ignored in BARC calculations (reverts to using the original/default rating curve). If/when the "thalweg notch" issue is addressed, this change may be unmade. - Added new env variable ignore_streamorders set to 10. - Added new BARC code to set the bathymetry adjusted cross-section area to 0 (reverts to using the default SRC values) based on the streamorder env variable. This resolves #454. * Fix two CatFIM bugs This fixes a bug in the get_metadata() function in /tools/tools_shared_functions.py that arose because of a WRDS update. Previously the metadata_source response was returned as independent variables, but now it is are returned a list of strings. Another issue was observed where the EVALUATED_SITES_CSV variable was being misdefined (at least on the development VM) through the OS environmental variable setting. - Changed parsing of WRDS metadata_sources to account for new list type. - Changed the way the EVALUATED_SITES_CSVpath is defined from OS environmental setting to a relative path that will work within Docker container. This resolves #445. * Update CHANGELOG.md * Update CHANGELOG.md * Rename BARC variables Renames the BARC modified variables that are exported to src_full_crosswalked.csv to replace the original variables. The default/original variables are renamed with orig_ prefix. This change is needed to ensure downstream uses of the src_full_crosswalked.csv are able to reference the authoritative version of the channel geometry variables (i.e. BARC-adjust where available). - In src_full_crosswalked.csv, default/original variables are renamed with orig_ prefix and SA_div is renamed to SA_div_flag. This resolves #418. * Update README.md * Update README.md * Update CHANGELOG.md * Update README.md * Evaluation patch Patches an issue where only certain benchmark categories were being used in evaluation. - In tools/tools_shared_variables.py, created a variable MAGNITUDE_DICT to store benchmark category magnitudes. - synthesize_test_cases.py imports MAGNITUDE_DICT and uses it to assign magnitudes. * Adjust SRC with user observed feedback points See Changelog 3.0.22.6 for details. This resolves #450 and resolves #436. * Bump pillow from 8.3.0 to 8.3.2 Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.3.0 to 8.3.2. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst) - [Commits](python-pillow/Pillow@8.3.0...8.3.2) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Approx bankfull stage and apply composite roughness in synthetic rating curves See Changelog for 3.0.22.7 * adding tool that allows the codebase to be linted and standardized (#470) * Fixing reversed flow on MS River See Changelog for 3.0.22.8 for more details. * Combine SRC modification modules in fim_run.sh post-processing step See CHANGELOG.md, version 3.0.23.0 for details. This resolves #466. * Patch updater.py after BARC merge Change API for release runs to use params_template instead of params_calibrated * Fix for null discharge values from composite roughness calcs See Changelog for 3.0.32.2 for more details. * Additional hydraulic properties to hydroTable and additional files in viz whitelist See CHANGELOG v3.0.23.3 for more details. * Adds full resolution and mainstem inundation composite capability This resolves #476. See Changelog 3.0.24.0 for full details. * Patch for inundation composite See Changelog for version 3.0.24.1 for full details. * [1pt] PR: Hotfix for .csv file delete in SRC post-processing (#486) Full details in Changelog for version 3.0.24.2. This resolves #485. * Update pip package to fix projection issue See Changelog for version 3.0.23.3. This resolves #483. * FIM version 4 See Changelog v4.0.0.0 for full details. This resolves #523. * Adds basic unit test capabilities See Changelog v4.0.1.0 for full details. * Preliminary readme fixes and deletion of old Changelog * Unit tests improvements * Adds unit test templates to make it easier to create new unit tests * Bump ipython from 7.29.0 to 7.31.1 (#510) Bumps [ipython](https://github.com/ipython/ipython) from 7.29.0 to 7.31.1. - [Release notes](https://github.com/ipython/ipython/releases) - [Commits](ipython/ipython@7.29.0...7.31.1) --- updated-dependencies: - dependency-name: ipython dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Suppress unit tests warnings which made outputs harder to read * New Feature to remove stream orders 1 and 2 if desired via command arg See Changelog v4.0.2.0 for full details. This addresses #552. * FIM 4 USGS gage crosswalk and Sierra Test See Changelog v4.0.3.0 for full details. This resolves #539. * Bug fixes for the Alpha Test to run in FIM 4 See Changelog 4.0.3.1 for full details. This addressed #551. * Bump pillow from 9.0.0 to 9.0.1 (#563) Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.0.0 to 9.0.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@9.0.0...9.0.1) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Stream order selection bug fixes, new logging features See Changelog 4.0.4.0 for full details. This resolves #579 * [1pt] PR: New tools and adjustments for bulk gms processing for alpha tests (#587) * A couple of new tools to create a list of hucs that have test case data and copy fim 3/4 folders tht match that huc list to another folder. * Upgrade output prints for better traceability, plus changed aggregate_branch_lists.py to be re-usable as a stand alone command line tool that the copy_test_case_folders.py can use when isolating specific hucs for alpha tests. * Update CHANGELOG.md * Update README.md (#589) * [1pt] PR: Hotfixes and updates to FIM4 alpha eval and eval plots (#594) * Hotfixes to resolve issues or missing eval metrics - Updated IFC magnitudes list variable (previously not used) - Copied FIM3 version of the workflow to assign FR, MS, COMP resolution variable - Added PND metric to calculations and output csv file - Added 'ras2fim' benchmark variable to list of data to evaluate * Updated magnitude dictionary lookup code - Added 'ras2fim' magnitude list to `tools_shared_variables.py` * Added re import * Copied Carson's eval_plots.py changes * Replaced deprecated parameter name for matplotlib grid() * Added Carson's latest eval_plots.py revisions to hotfix * Added PND plots to eval_plots.py * Updated changelog * Update to fix error related to old version of flask (#599) * [1pt] Mosaic inundation maps for FIM3 FR and FIM4 GMS SO 3+ (#605) * WIP: basic inundation work, next is compositing * WIP: (t5) inundation and mosaic of a single huc fully operational. Now on to multi hucs * WIP: Fix masking error * WIP: Most is working, just adding multi processing to hucs * Completed: composite_inundation operational, with various adjustment to secondary files like inundate_gms.py and others * updated pathing for the unit test file * Update CHANGELOG.md * [1pt] PR: 'Branch zero' that utilizes the full stream network (#611) * move DEM processing to unit level * created to run common processing between and * finalize branch zero * branch zero code standardization and cleanup * fixed gitignore * Docker pathing issue resulted in not being able to see the new deny_gms_branch_zero.lst file. This is a temp work around. * small bug fixes for branch zero * bug fix for branch zero feature_id crosswalk errors * remove redundant files * remove tests directory * branch errors folder bug * update CHANGELOG.md Co-authored-by: Rob Hanna <Robert.Hanna@NOAA.gov> * [1pt] PR: Alpha Test Refactor (#612) * move DEM processing to unit level * created to run common processing between and * finalize branch zero * updated alpha test to use test_case class from dev-fim3 * small bug fixes for the alpha test * updated verbose printing, fix exception handing issue inside multi processing, created a generic re-usable verbose printing method and misc styling upgrades * fix issue with multi-proc not working * Update CHANGELOG.md Co-authored-by: Rob Hanna <Robert.Hanna@NOAA.gov> Co-authored-by: Rob Hanna - NOAA <90854818+RobHanna-NOAA@users.noreply.github.com> * added version to pipenv (#634) * Updated unit tests based on latest code and adjusted some pathing (#622) * Simplify usage of fim4 via gms_pipeline.sh (#641) * Bug fix for missing `testing_versions` folder (#642) * Removing special characters from CHANGELOG (#649) * Adding step to trim DEM derived streamlines if they extend past branch streamlines (#614) * Prune branches that fail with Exit status: 61 in split_flows.py (#655) * Adds USFIMR Data to Benchmark Capability (#639) * Update docker python packages for calibration db (#652) * FIM4 synthetic rating curve calibration functionality (#657) * Patches for FIM4 alpha test (#658) * Hotfix for addressing missing input variable in gms_run_branch.sh (#665) * Update Inundate_nation.py for FIM4 (#654) * Prune branches in waterbodies (#671) * Update derive level paths to use stream order (#672) * Add tool to push data to S3 for Hydroviz (#664) * v4.0.9.2 Bug fixes for pruning lake branches and stream order prioritizing (#678) * Cleanup CHANGELOG.md (#688) * v4.0.9.3 Download USGS 3DEP Dems (#681) * v4.0.9.4 Cleanup Branch Zero output at the end of a processing run (#691) * v4.0.9.5 HOTFIX: fixes for calib system, landSea deny list and params (#696) * v4.0.9.6 Inundate-nation bug fixes and some minor upgrades (#711) * v4.0.9.7 Add fim3 s3 whitelist and rename the fim s3 whitelist (#703) * v4.0.9.8 Added calibration database engine to FIM4 (#701) * v4.0.10.0 Implementing USGS 3DEP DEMs into the FIM 4 workflow (#697) * v4.0.10.1 Hotfix for FIM4 sierra test plots with branch zero (#695) * v4.0.10.2 Runs branch 0 on HUCs without other branches after stream order filtering (#723) * v4.0.10.3 Port thalweg notch fix from FIM3 (#718) * v4.0.10.4 Create HydroID - feature_id crosswalk table (#727) * v4.0.10.5 Add cblend (buffer) to acquire 3dep downloads (#720) * v4.0.11.0 Mask levee-protected areas from Relative Elevation Model (#690) * v4.0.11.1 Upgrade jupyter core (#732) * v4.0.11.2 Fim4 branch zero override (#737) * v4.0.11.3 Add tool to preprocess downloaded levee prot areas (#739) * v4.0.11.4 Adds capability to produce single FIM4 rating curve comparison plots for each gage (#709) * v4.0.11.5 Skip `usgs_gage_unit_setup.py` if no level paths (#746) * v4.0.12.0 Channel/overbank subdivision framework (#736) * v4.0.12.1 Cleanup a few deny list files (#751) * v4.0.12.2 Hotfix: Stop writing `gms_inputs_removed.csv` if empty (#754) * v4.0.13.0 Replace NHD headwaters with NWM-derived headwaters and remove NHD references (#744) * V4.0.13.1 fim4 alpha by catchments (#743) * Bump certifi from 2022.9.24 to 2022.12.7 (#757) * Bug fix to inundation.py ignoring nodata * Mask levee-protected areas from DEM * Clip WBD to 3DEP DEM domain * Update and renamed whitelists for hydrovis S3 put * Enhancements to CatFIM, rating curve generation script, and other files * Make GMS HAND post-processing independent * Add rsync to docker image * Bug fix to Stage-Based CatFIM * Remove `other` folder from `test_cases` * Cleanup REM masking * v4.0.17.4 Update hydrovis S3 push whitelist (#781) * v4.0.18.0 Clip WBD and branch buffer polygons to DEM domain (#780) * v4.0.18.1 Stacked Bar Eval Plots (#760) * v4.0.18.2 Remove Great Lakes clipping (#790) * v.4.0.19.0 Change the projection of HAND processing to EPSG:5070 (#782) * v4.0.19.1 Bug fix: combine_crosswalk_tables.py fails if only one HUC is run (#796) * v4.0.19.2 Consolidate bash environment variables into a single file (#797) * v4.0.19.3 Remove fim3 files and refs (#794) * v4.0.19.4 branch trimming for lake segments (#802) * v4.0.19.5 Fix error msg of stats object not referenced (#801) * v4.1.0.0 One HUC processes its own branches (#806) * v4.1.1.0 Updated 1-foot interval CatFIM with restart capabilities (#809) * v4.1.1.0 Update CHANGELOG.md (#817) * v4.1.2.0 Refactor `/unit_tests` to use Pytest. (#808) * v4.1.3.0 Update for calib db pathing to work in AWS (#812) * Delete check_unit_errors_params.json (#819) * Delete README.md (#818) * Delete derive_level_paths_params.json (#820) * v4.2.0.0 Remove FIM3 and most references to GMS (#816) * v4.2.0.1 Fix for missing branch_id.lst file (#827) * v4.2.1.0 Fix broken unit tests (#829) * v4.3.0.0 Replace GRASS with Whitebox (#814) * v4.3.0.1 Delete intermediate agreedem files (#841) * v4.3.1.0 Change all occurances of `/data/outputs` to `/outputs` (#834) * 4.3.2.0: Revise masking levee-protected areas from inundation (#845) See Changelog 4.3.2.0 for full details. * 4.3.3.0: Fix for CatFIM sites sharing LID and workaround for WRDS slowdown (#835) See Changelog 4.3.3.0 for full details. * v4.3.3.1 Bug fix -- KeyError in `src/associate_levelpaths_with_levees.py` (#855) * v4.3.3.2 Bug fix - Change .split() to os.path.splitext() (#851) * v4.3.3.3 Fixed datatype issue in coord accuracy list (#854) * v4.3.3.4 Hotfix for inundate nation projection error (#849) --------- Co-authored-by: Greg Cocks <greg.cocks@noaa.gov> Co-authored-by: MattLuck-NOAA <matthew.luck@gmail.com> Co-authored-by: brian.avant <brian.avant@noaa.gov> Co-authored-by: FernandoSalas-NOAA <fernando.salas@noaa.gov> Co-authored-by: NickChadwick-NOAA <nick.chadwick@noaa.gov> Co-authored-by: RyanSpies-NOAA <ryan.spies@noaa.gov> Co-authored-by: Brad <bradford.bates@noaa.gov> Co-authored-by: MattLuck-NOAA <matt.luck@noaa.gov> Co-authored-by: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rob Hanna - NOAA <90854818+RobHanna-NOAA@users.noreply.github.com> Co-authored-by: Rob Hanna <Robert.Hanna@NOAA.gov> Co-authored-by: JimJam <89595253+JamesColl-NOAA@users.noreply.github.com> Co-authored-by: EricMyskowski-NOAA <108021418+EricMyskowski-NOAA@users.noreply.github.com> Co-authored-by: CalebOliven-NOAA <97476664+CalebOliven-NOAA@users.noreply.github.com> Co-authored-by: LauraKeys-NOAA <70279180+LauraKeys-NOAA@users.noreply.github.com> Co-authored-by: Rob Gonzalez-Pita <rob.g.pita@noaa.gov>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Investigate how the HUC 4,6,8 parameterization affects the fim_run outputs especially the inundation inputs: rem_.tif, gw_, and hydro-table. These affect inundation maps and subsequent scoring. Example: fim_run outputs for 120903 and 12090301 are mostly the same if not identical except for the rating curves. This affects the maps and the scoring with 12090301 getting worse results.
The text was updated successfully, but these errors were encountered: