Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
commit 7befb39 Author: Brad <bradford.bates@noaa.gov> Date: Thu Jul 1 10:16:49 2021 -0500 Allowance of evaluation at IFC sites and updating of bad eval sites list Updates to evaluation scripts to allow for Alpha testing at Iowa Flood Center (IFC) sites. Also, BAD_SITES variable updates to omit sites not suitable for evaluation from metric calculations. - The BAD_SITES list in tools_shared_variables.py was updated and reasons for site omission are documented. - Refactored run_test_case.py, synthesize_test_cases.py, tools_shared_variables.py, and eval_plots.py to allow for IFC comparisons. This resolves #424, resolves #412, and resolves #428. commit ac4ae06 Author: Brian Avant <brian.avant@noaa.gov> Date: Mon Jun 21 07:46:35 2021 -0500 Thalweg Profile Tool and lateral Thalweg Adjustment Threshold Adding a thalweg profile tool to identify significant drops in thalweg elevation. Also setting lateral thalweg adjustment threshold in hydroconditioning. - thalweg_drop_check.py checks the elevation along the thalweg for each stream path downstream of MS headwaters within a HUC. - Removing dissolveLinks arg from clip_vectors_to_wbd.py. - Cleaned up code in split_flows.py to make it more readable. - Refactored reduce_nhd_stream_density.py and adjust_headwater_streams.py to limit MS headwater points in agg_nhd_headwaters_adj.gpkg. - Fixed a bug in adjust_thalweg_lateral.py lateral elevation replacement threshold; changed threshold to 3 meters. - Updated aggregate_vector_inputs.py to log intermediate processes. This resolves #408, resolves #18, and resolves #409. commit ca8cb7e Author: Brad <bradford.bates@noaa.gov> Date: Thu Jun 10 10:54:32 2021 -0500 Feature to evaluate alternative CatFIM technique performance Added eval_catfim_alt.py to evaluate performance of alternative CatFIM techniques. This resolves #414. commit 129f2d5 Author: NickChadwick-NOAA <nick.chadwick@noaa.gov> Date: Wed Jun 9 10:38:19 2021 -0500 Fim Run Memory Profiling To help analyze the memory consumption of the Fim Run process, the python module memory-profiler has been added to give insights into where peak memory usage is with in the codebase. In addition, the Dockerfile was previously broken due to the Taudem dependency removing the version that was previously being used by FIM. To fix this, and allow new docker images to be built, the Taudem version has been updated to the newest version on the Github repo and thus needs to be thoroughly tested to determine if this new version has affected the overall FIM outputs. Resolves #392 and #403 Additions - Added memory-profiler to Pipfile and Pipfile.lock. - Added mprof (memory-profiler cli utility) call to the time_and_tee_run_by_unitsh to create overall memory usage graph location in the /logs/{HUC}_memory.png of the outputs directory. - Added @Profile decorator to all functions within scripts used in the run_by_unit.sh script to allow for memory usage tracking, which is then recorded in the /logs/{HUC}.log file of the outputs directory. Changes - Changed the Taudem version in Dockerfile.dev to 98137bb6541a0d0077a9c95becfed4e56d0aa0ac. - Changed all calls of python scripts in run_by_unit.sh to be called with the -m memory-profiler argument to allow scripts to also track memory usage. commit a001d75 Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Fri Jun 4 15:25:56 2021 -0500 Correct headwater assignment nws_lid layer Bug fix to the generate_nws_lid.py script - Fixes incorrectly assigned attribute field "is_headwater" for some sites in the nws_lid.gpkg layer. - Updated agg_nhd_headwaters_adj.gpkg, agg_nhd_streams_adj.gpkg, nwm_flows.gpkg, and nwm_catchments.gpkg input layers using latest NWS LIDs. This resolves #383 and resolves #397. commit cca68e1 Author: Brad <bradford.bates@noaa.gov> Date: Fri Jun 4 14:03:05 2021 -0500 Update CHANGELOG.md commit b9167d5 Author: RyanSpies-NOAA <ryan.spies@noaa.gov> Date: Fri Jun 4 14:00:06 2021 -0500 BARC updates to cap xsec area and allow user to choose input bankfull geometry BARC updates to cap the bathy calculated xsec area in bathy_rc_adjust.py and allow user to choose input bankfull geometry. - Added new env variable to control which input file is used for the bankfull geometry input to bathy estimation workflow. - Modified the bathymetry cross section area calculation to cap the additional area value so that it cannot exceed the bankfull cross section area value for each stream segment (bankfull value obtained from regression equation dataset). - Modified the rating_curve_comparison.py plot output to always put the FIM rating curve on top of the USGS rating curve (avoids USGS points covering FIM). - Created a new aggregate csv file (aggregates for all hucs) for all of the usgs_elev_table.csv files (one per huc). - Evaluate the FIM Bathymetry Adjusted Rating Curve (BARC) tool performance using the estimated bankfull geometry dataset derived for the NWM route link dataset. This resolves #342, resolves #343, resolves #313, resolves #401, and resolves #402. commit ac25186 Author: Brad <bradford.bates@noaa.gov> Date: Fri May 21 13:31:07 2021 -0500 Alpha test bug fixes and improvements Enhancements and bug fixes to synthesize_test_cases.py. - Addresses a bug where AHPS sites without benchmark data were receiving a CSI of 0 in the master metrics CSV produced by synthesize_test_cases.py. - Includes a feature enhancement to synthesize_test_cases.py that allows for the inclusion of user-specified testing versions in the master metrics CSV. - Removes some of the print statements used by synthesize_test_cases.py. This resolves #354 and resolves #386. commit f2d65ee Author: Brad <bradford.bates@noaa.gov> Date: Tue May 18 13:34:26 2021 -0500 Update CHANGELOG.md commit 8fbf4b6 Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Tue May 18 13:26:56 2021 -0500 Bug fixes in preprocessing and run_test_case.py and update eval_plots Modifications and fixes to run_test_case.py, eval_plots.py, and AHPS preprocessing scripts. - Comment out return statement causing run_test_case.py to skip over sites/hucs when calculating contingency rasters. - Move bad sites list and query statement used to filter out bad sites to the tools_shared_variables.py. - Add print statements in eval_plots.py detailing the bad sites used and the query used to filter out bad sites. - Update AHPS preprocessing scripts to produce a domain shapefile. - Change output filenames produced in ahps preprocessing scripts. - Update workarounds for some sites in ahps preprocessing scripts. This resolves #381, resolves #369, resolves #326, resolves #385, resolves #377. commit cbc360f Author: NickChadwick-NOAA <nick.chadwick@noaa.gov> Date: Tue May 11 10:42:27 2021 -0500 Updated Eventlet package to 0.31.0 to patch a vulnerability The current version of Eventlet used in the Connector module of the FIM API is outdated and vulnerable. This update bumps the version to the patched version. Using new nhd inputs. commit b5da4b4 Author: NickChadwick-NOAA <nick.chadwick@noaa.gov> Date: Fri May 7 14:45:39 2021 -0500 Added fuctionality for new "Release" feature in FIM API New "Release" feature added to the FIM API. This feature will allow for automated FIM, CatFIM, and relevant metrics to be generated when a new FIM Version is released. See #373 for more detailed steps that take place in this feature. Additions - Added new window to the UI in api/frontend/gui/templates/index.html. - Added new job type to api/node/connector/connector.py to allow these release jobs to run. - Added additional logic in api/node/updater/updater.py to run the new eval and CatFIM scripts used in the release feature. Changes - Updated api/frontend/output_handler/output_handler.py to allow for copying more broad ranges of file paths instead of only the /data/outputs directory. Resolves Issues #264 #278 #307 and #373 commit 22ba0df Author: Brian Avant <brian.avant@noaa.gov> Date: Fri May 7 12:51:19 2021 -0500 Remove Great Lakes from wbd buffer - The gl_water_polygons.gpkg layer is used to mask out Great Lakes boundaries and remove NHDPlus HR coastline segments. These segments are causing issues later in run_by_unit.sh and unnecessarily increasing total processing time. Resolves issue #374 commit 25c3ec1 Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Mon May 3 10:07:19 2021 -0500 Generate nws_lid.gpkg Generate nws_lid.gpkg. - Generate nws_lid.gpkg with attributes indicating if site is a headwater nws_lid as well as if it is co-located with another nws_lid which is referenced to the same nwm_feature_id segment. This resolves #368. commit 9518cbe Author: Brian Avant <brian.avant@noaa.gov> Date: Fri Apr 30 15:41:29 2021 -0500 Refactor NHDPlus HR preprocessing workflow Refactor NHDPlus HR preprocessing workflow. - Consolidate NHD streams, NWM catchments, and headwaters MS and FR layers with mainstem column. - HUC8 intersections are included in the input headwaters layer. - clip_vectors_to_wbd.py removes incoming stream segment from the selected layers. This resolves #238. commit 199ab6e Author: Brian Avant <brian.avant@noaa.gov> Date: Wed Apr 28 11:32:11 2021 -0500 Refactor synthesize_test_case.py to handle exceptions during multiprocessing Refactor synthesize_test_case.py to handle exceptions during multiprocessing. Resolves issue #351 Changes - refactored inundation.py and run_test_case.py to handle exceptions without using sys.exit(). This isn't the ideal way to handle exceptions when using multiprocessing because the child process is expecting a return. commit 99c197b Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Tue Apr 27 12:42:59 2021 -0500 Implement CatFIM threshold flows to Sierra test and add AHPS benchmark preprocessing scripts. Implement CatFIM threshold flows to Sierra test and add AHPS benchmark preprocessing scripts. - Produce CatFIM flows file when running rating_curve_get_usgs_gages.py. - Several scripts to preprocess AHPS benchmark data. Requires numerous file dependencies not available through Cahaba. - Modify rating_curve_comparison.py to ingest CatFIM threshold flows in calculations. - Modify eval_plots.py to save all site specific bar plots in same parent directory instead of in subdirectories. - Add variables to env.template for AHPS benchmark preprocessing. This resolves #341, resolves #361, and resolves #357. commit c311317 Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Thu Apr 22 15:32:10 2021 -0500 Update plotting script to not error out if FR or MS are not analyzed when using spatial argument update eval_plots.py to correctly produce spatial datasets if version is not analyzed. For example, if no FR versions are analyzed, instead of throwing an error, it will check to make sure there exists a FR analyzed datasets, if one exists then create spatial data for FR datasets (ble). commit 6387384 Author: Brian Avant <brian.avant@noaa.gov> Date: Tue Apr 20 09:54:37 2021 -0500 Closing multiprocessing pool objects Updating repo to close all multiprocessing Pool objects to fix memory issues commit a304cf3 Author: Brian Avant <brian.avant@noaa.gov> Date: Mon Apr 19 12:14:47 2021 -0500 Preprocess NHDPlus HR rasters for consistent projections, nodata values, and convert from cm to meters. Preprocess NHDPlus HR rasters for consistent projections, nodata values, and convert from cm to meters. - preprocess_rasters.py reprojects raster, converts to meters, and updates nodata value to -9999. - Cleaned up log messages from bathy_rc_adjust.py and usgs_gage_crosswalk.py. - Outputs paths updated in generate_categorical_fim_mapping.py and generate_categorical_fim.py. - update_raster_profile cleans up raster crs, blocksize, nodata values, and converts elevation grids from cm to meters. - reproject_dem.py imports gdal to reproject elevation rasters because an error was occurring when using rasterio. - burn_in_levees.py replaces the gdal_calc.py command to resolve inconsistent outputs with burned in levee values. This resolves #300. commit 1a15cd8 Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Fri Apr 16 14:09:21 2021 -0500 Fixes production list in output_cleanup.py. Fixes production list in output_cleanup.py. commit 5a8ebcd Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Wed Apr 14 08:47:58 2021 -0500 Add mainstems attribute for sierra test Sierra test considered all USGS gage locations to be mainstems even though many actually occurred with tributaries. This resulted in unrealistic comparisons as incorrect gages were assigned to mainstems segments. This feature branch identifies gages that are on mainstems via attribute field. - Modifies usgs_gage_crosswalk.py to filter out gages from the usgs_gages.gpkg layer such that for a "MS" run, only consider gages that contain rating curve information (via curve attribute) and are also mainstems gages (via mainstems attribute). - Modifies usgs_gage_crosswalk.py to filter out gages from the usgs_gages.gpkg layer such that for a "FR" run, only consider gages that contain rating curve information (via curve attribute) and are not mainstems gages (via mainstems attribute). - Modifies how mainstems segments are determined by using the nwm_flows_ms.gpkg as a lookup to determine if the NWM segment specified by WRDS for a gage site is a mainstems gage. - Adds a mainstem attribute field to usgs_gages.gpkg that indicates whether a gage is located on a mainstems river. - Adds NWM_FLOWS_MS variable to the .env and .env.template files. - Adds the extent argument specified by user when running fim_run.sh to usgs_gage_crosswalk.py. commit c9161b8 Author: RyanSpies-NOAA <ryan.spies@noaa.gov> Date: Thu Apr 8 11:11:56 2021 -0500 Implementing a prototype technique to estimate the missing bathymetric component in the HAND-derived synthetic rating curves. Implementing a prototype technique to estimate the missing bathymetric component in the HAND-derived synthetic rating curves. The new Bathymetric Adjusted Rating Curve (BARC) function is built within the fim_run.sh workflow and will ingest bankfull geometry estimates provided by the user to modify the cross section area used in the synthetic rating curve generation. - add_crosswalk.py outputs the stream order variables to src_full_crosswalked.csv and calls the new bathy_rc_adjust.py if bathy env variable set to True and extent=MS. - run_by_unit.sh includes a new csv outputs for reviewing BARC calculations. - params_template.env & params_calibrated.env contain new BARC function input variables and on/off toggle variable. eval_plots.py now includes additional AHPS eval sites in the list of "bad_sites" (flagged issues with MS flowlines). - Adds bathy_rc_adjust.py - Imports the existing synthetic rating curve table and the bankfull geometry input data (topwidth and cross section area per COMID). - Performs new synthetic rating curve calculations with bathymetry estimation modifications. - Flags issues with the thalweg-notch artifact. This resolves #306, resolves #328, resolves #309, resolves #311, and resolves #330. commit 40631f4 Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Mon Apr 5 13:45:09 2021 -0500 Create tool to retrieve rating curves from USGS sites and convert to elevation (NAVD88). Intended to be used as part of the Sierra Test. - Modify usgs_gage_crosswalk.py to: - Look for location_id instead of site_no attribute field in usgs_gages.gpkg file. - Filter out gages that do not have rating curves included in the usgs_rating_curves.csv - Modify rating_curve_comparison.py to perform a check on the age of the user specified usgs_rating_curves.csv and alert user to the age of the file and recommend updating if file is older the 30 days. - Add rating_curve_get_usgs_curves.py. This script will generate the following files: - usgs_rating_curves.csv: A csv file that contains rating curves (including converted to NAVD88 elevation) for USGS gages in a format that is compatible with rating_curve_comparisons.py. As it is is currently configured, only gages within CONUS will have rating curve data. - log.csv: A log file that records status for each gage and includes error messages. - usgs_gages.gpkg: A geospatial layer (in FIM projection) of all active USGS gages that meet a predefined criteria. Additionally, the curve attribute indicates whether a rating curve is found in the usgs_rating_curves.csv. This spatial file is only generated if the all option is passed with the -l argument. This resolves #289. commit 44b6333 Author: brian.avant <brian.avant@noaa.gov> Date: Thu Apr 1 13:40:49 2021 -0500 [1pt] Tool to compare synthetic rating curve with benchmark rating curve (sierra test) (#332) * `usgs_gage_crosswalk.py`: generates `usgs_elev_table.csv` in run_by_unit.py with elevation at USGS gages * `rating_curve_comparison.py`: post-processing script to plot and calculate metrics between FIM and USGS rating curve data * updates `aggregate_fim_outputs.py` call argument in `fim_run.sh` from 4 jobs to 6 jobs (optimizing API performance) * reroutes median elevation data from `add_crosswalk.py` and `rem.py` to new file (depreciating `hand_ref_elev_table.csv`) *- adds new files to `viz_whitelist` in `output_cleanup.py` commit 4df6d1d Author: Brad <bradford.bates@noaa.gov> Date: Thu Apr 1 12:27:10 2021 -0500 Update CHANGELOG.md commit 385ae0f Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Thu Apr 1 12:24:05 2021 -0500 Update spatial option when performing eval plots Removes file dependencies from spatial option. Does require the WBD layer which should be specified in .env file. - Produces outputs in a format consistent with requirements needed for publishing. - Preserves leading zeros in huc information for all outputs from eval_plots.py. - Creates fim_performance_points.shp: this layer consists of all evaluated ahps points (with metrics). Spatial data retrieved from WRDS on the fly. - Creates fim_performance_polys.shp: this layer consists of all evaluated huc8s (with metrics). Spatial data retrieved from WBD layer. This resolves #325. commit 391b547 Author: Brad <bradford.bates@noaa.gov> Date: Wed Mar 31 10:00:34 2021 -0500 Update README.md commit a60e6ee Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Tue Mar 30 16:17:02 2021 -0500 Adds detail and more information to plotting capabilities Add detail/information to plotting capabilities. - Merge plot_functions.py into eval_plots.py and move eval_plots.py into the tools directory. - Remove plots subdirectory. - Optional argument to create barplots of CSI for each individual site. - Create a csv containing the data used to create the scatterplots. This resolves #325. commit 2b26065 Author: Brad <bradford.bates@noaa.gov> Date: Thu Mar 25 09:08:06 2021 -0500 Update CHANGELOG.md commit 34123dd Author: TrevorGrout-NOAA <69653333+TrevorGrout-NOAA@users.noreply.github.com> Date: Thu Mar 25 08:51:04 2021 -0500 Improvements to CatFIM service source data generation Improvements to CatFIM service source data generation. - Renamed generate_categorical_fim.py to generate_categorical_fim_mapping.py. - Updated the status outputs of the nws_lid_sites layer and saved it in the same directory as the merged catfim_library layer. - Additional stability fixes (such as improved compatability with WRDS updates). - Added generate_categorical_fim.py to wrap generate_categorical_fim_flows.py and generate_categorical_fim_mapping.py. - Create new nws_lid_sites shapefile located in same directory as the catfim_library shapefile. This resolves #305, resolves #321, and resolves #322. commit d43bd8b Author: Brad <bradford.bates@noaa.gov> Date: Wed Mar 24 10:21:04 2021 -0500 Patch to synthesize_test_cases.py Bug fix to synthesize_test_cases.py to allow comparison between testing version and official versions.
- Loading branch information
1 parent
13e32cb
commit 2e13a30