Pi 6 ngwpc 6402 #30
Merged
Pi 6 ngwpc 6402 #30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author
|
@taddyb The last line which reads the inundation file can be developed more in terms of better represents the results. |
taddyb
approved these changes
Jun 3, 2025
taddyb
left a comment
taddyb
left a comment
There was a problem hiding this comment.
Pushing changes to remove the tracebacks from the github script. other than that, looks good!
taddyb
added a commit
that referenced
this pull request
Aug 29, 2025
* Adds step to resync foreign tables to wrds-db-sync Refs NOAA-OWP#1050 * Updates AEP AK mapx to match AEP CONUS mapx * Converts KCFS to CFS appropriately in RnR setup Refs NOAA-OWP#1100 * Updates AEP AK mapx to match AEP CONUS mapx (pt 2) * initial commit for WIP * Set feature_serivce = true in AEP service ymls * Table Name Change * Updates during 3/21 tag up w/Lorne * Modified code with event and env changes * update with s3 lifecyle policies * making S3 lifecycle rules dynamic per bucket * Critical Infrastructure Point Service EGIS DB Ingestion and Static Service Creation Revision (NOAA-OWP#1046) Created Jupyter Notebook to document the ingestion of critical infrastructure point data into the EGIS database. Mapx and yml files were created for a new critical infrastructure point static service. * Created new FIMPact to Critical Infrastructure service as a sublayer in the SRF 18hr Max Inundation Extent service. * Merge pull request NOAA-OWP#1119 from NOAA-OWP/fimpact_crit_infra_points_wide_release FIMPact to Critical Infrastructure wider release * Tweaked SRF 18hr Mapx - Added Crit FIMPact to RFC Based 5day max inundation. * Tweaks * TraceMalloc Tweak * Modified to dissolve * 5922: Integrate HML Reader into Hydrovis (#25) * moved message queue from hml_reader/ to Ingest/ * updating to reflect PR review from Quercus * Make botocore an optional dependency. * Updates to formalize ripple as standalone process * Tweak as necessary to get working in the pipeline * Ripple ana * Made uploading tracking data off by default for UAT/PRD * Use generate_file_list shared function when processing ingest groups. * deployment related fixes * Adding Ripple Services * New Service mapx * Move viz-db-ingest to image-based lambda (NOAA-OWP#1121) Create an image-based lambda for viz-db-ingest. Terraform config should match the other image-based lambdas. Refactored some of the code to achieve the following: 1. Use the new viz_database and viz_s3 modules. 2. Simplify netcdf manipulation code 3. Refactor SQL code for better simplicity. * Geometry Needs to be MULTIPOLYGON * Docker setup refined * Modified code to focus on control method only. * Adjusted for small reach file sizes * first pass, working * More updates * Add README * Add AWS login step to README * Fixes * Another fix * Logic tweak * Update to Dev\configuration.yml * Updated README * Marking Lambda and StepFunction contents as sensitive so they don't show up in Terraform plan * Fixed publish lambda mapx uploads * removing unused file * Small fixes to new Lambda workflow * Updating Lambda Dockerfiles to pull base images from ghcr instead of Dockerhub * updating to new Lambda restructure * Fixes to latest merge * Put the ripple step function back where it was suppose to be. * Fixes to last commit * Re-structured zonal-coastal-fim to be more compliant with new format. Added mapx/sql/yml files * Updated yml service text * Remove the "non-0 branches for AK" HAND FIM filter * Began working on mrf nbm 10day product config yml * Experimental Change - Added a Sublayer to ANA High Flow Magnitude Service that uses 2.0 instead of 2.5 for High Water Threshold for Cluster 7 (NOAA-OWP#1132) Following a Texas water event, it was observed locally that many streams that experienced high water events were not flagged as such in our services. As an initial adjustment, we've added an additional sublayer that represents the ana high flow magnitude based on a new conus recurrence flow table. This new table is identical to the old version, except the cluster 7 reaches have been redirected to use the 2.0 year recurrence flow value for their high water threshold instead of the previous 2.5 year. This adjustment should results in streams in cluster 7 represented by the ana high flow magnitude service more accurately reflecting reality. ## Additions -New experimental sublayer in the ana high flow magnitude conus service based on updated recurrence flow table, where cluster 7 reaches use 2.0 year recurrence flows instead of 2.5. Co-authored-by: katherine powell <katherine.powell@noaa.gov> Co-authored-by: CatherineFitzpatrick-NOAA <49403615+cfitzpat479@users.noreply.github.com> Co-authored-by: Catherine <catherine.fitzpatrick@noaa.gov> Co-authored-by: Nick Chadwick <nick.chadwick@noaa.gov> Co-authored-by: ShawnCrawley-NOAA <shawn.crawley@noaa.gov> Co-authored-by: George Rought <1061699810113305@mil> * Hand fixes for aep and catchments (NOAA-OWP#1135) When attempting to create HAND FIM polygons for AEP/Catchments using the latest HAND FIM version (4.6.1.4), we kept finding that no polygons were being created. Turns out that line 284 of the hand_fim_processing lambda_function.py was dropping all of the polygons due to the recently added `flood_area_above_expected_coeff` column being `nan` for all rows in the case of static/reference services. This fixes that by dropping that column completely prior to the `join().dropna()`, but also swaps out the `join().dropna()` for an inner join, since that seems to be what was intended there. * Repurpose apocalyptic testing for general events * Finalize terraform and enable Dev resources * Fix references * [WIP] adds most terraform infrastructure Loose ends left hanging to pivot to cleaning up and standardizing the docker image stuff * Update environment.yml with corrected up to date contents * Added fsspec environment.yml * Update lambda_function.py Replaced boto3 with fsspec updated error handling moved SQLAlechmy create engine to the global level instead of doing it internally in helper function replaced manual temp folder management with tempfile module removed redundant "inundated" attribute from database removed gc.collect() * Update geometry_manipulation.py Removed boto3 * Removed boto3 from Dockerfile * Rebuilt lock file based on revised environment.yml * Optimize Raster refactor (NOAA-OWP#1137) Overhauls the code in the optimize raster lambda image. 1. Removes the ESRI modules. We weren't really using any of their exclusive features. Instead we call gdal_translate directly. 2. When calling gdal_translate, we read directly from S3 instead of caching the file locally. 3. Use fsspec to upload results to S3. 4. Move image to ARM64. 5. Optimized rasters are now compressed using DEFLATE. In my tests, the compressed files are only 5-10% the size of the old "optimized" rasters. * Updated Dockerfile to micromamba standard. Updated DuckDB Spatail Extension install/load method. Improved error handling. * Added mrf nbm 10day max initialization yml and db postprocess sql file. Enabled in initialize pipeline lambda. * TI v2 2 manual downloads (NOAA-OWP#1130) All changes to load the latest HAND 4.6.1.4 data. Note: All data loads complete. * Added mrf nbm 10 day max mapx and service definition yml. * Update README * New BITS, Bridge Inundation Threat Service, layer inside QC FIM / FIM Pact service layers (NOAA-OWP#1129) This is a new set of layers to be added to all services that have a QC FIM / FIM pact layer. Generally, for all layers that have building data, this layer will belong in the same services. The service will be point based with the colors of gray, orange or red and is dynamically calculated based on the current flow data supporting that QC FIM service layer. The data supplied by the FIM dev teams for the bridge points have to key columns, `threshold_discharge_cfs` and , `threshold_discharge_75_cfs`. These represent a flow value which can help identify if the bridge is under no / low risk of flooding, moderate risk of flooding or likely is inundated. Note: is it easiest to think of the threshold_discharge_cfs as 100, max discharge calculated by FIM and threshold_discharge_75_cfs as 75% of the max discharge before being threatened. The logic scheme for the system is based on using actual flow data provide dynamically at run time and it's relationship to the two threshold discharge values above. - If the actual flow value is above `threshold_discharge_cfs` (100%), then the point color is red and the message says "High Risk of Flooding" in the metadata. - If the actual flow value is below `threshold_discharge_cfs` (100%) and above `threshold_discharge_75_cfs`, then the point color is orange and the meta data says "Potentially Flooded" - If the actual flow is below `threshold_discharge_75_cfs`, then the point color is gray and meta data says "Low Risk of Flooding". **Services it is plugged into are:** - ana_inundation_extent_noaa.mapx - ana_inundation_extent_hi_noaa.mapx - ana_inundation_extent_prvi_noaa.mapx - ana_past_14day_max_inundation_extent_noaa.mapx (layers for both 7 day and 14 day of course) - mrf_nbm_10day_max_inundation_extent_noaa.mapx (layers for 3 day, 5 day and 10 day) - mrf_gfs_10day_max_inundation_extent_noaa.mapx (layers for 3 day, 5 day and 10 day) - rfc_based_5day_max_inundation_extent_noaa.mapx - srf_18hr_max_inundation_extent_noaa.mapx - srf_48hr_max_inundation_extent_hi_noaa.mapx - srf_48hr_max_inundation_extent_prvi_noaa.mapx * Tweak developer depoly and fix test-wrds-db lambda * Removed insatll spatial commands. Updated exception handling. Removed DuckDB redundant check from lambda_function.py. * Add terraform code for zonal-coastal-fim resources * Incorporated Ryan's feedback on revising vecotirze_binary_extent function. Removed hard coded python executable path. Removed unused imports. * Update and renamed ras2fim boundaries service to hecras boundaries service (NOAA-OWP#1141) Upgraded and renamed the original `ras2fim boundary` service to `HECRAS boundary` service which reflects hecras data from both ras2fim and ripple. Note: it has a column for 'is_active' which is supposed to be for huc8 records where we are actually using ras2fim/ripple data in our service. However, while we know all of the huc8s with ras2fim data is included, we do not know which ripple HUCs were actually used in the service. As a result, for now, all HUC8s are "is_active" = True. Once the new Ripple datasets are used, this table will need to be updated. * Change true to false for new deploy_resource * Resolved Shawn's suggestions. Revised vectorize_binary_extent for memory efficiency. * Changed DB engine to be from the Shared Lambdas Classes module * Using newer isolated DB module * Creating DB connection as a global variable to be shared across Lambda executions. * Address comments by @nickchadwick-noaa and @groutr * Add viz_classes.py to terraform deploy zip * One final change * Calling for the creation of the engine only once it's needed. * Deployment fixes * deployment fixes * JIRA 5923: RnR docker compose addition (reuploaded for rebasing issue) (#27) * adding new docker development tools for RnR * patch: error with the .venv file location * patching error with volume mounts and venv paths * fixed container warnings * JIRA 5924: Historical test case / tests for RnR (#28) * initial commit * added notebook to explore geopackage and outputs * updated output text * added more notebooks * updated mock data * updating saved test data * patch: adding missing dirs and a README.md * Pi 6 ngwpc 6402 (#30) * adding test_rnr to test and removing venv from git update * adding test_rnr to test * removed traceback for cleaner github view --------- Co-authored-by: root <root@U-1BVT2XY7DCIBM.nextgenwaterprediction.com> Co-authored-by: taddyb <tkb5476@psu.edu> * JIRA 6400: Documenting Replace and Route (#29) * initial commit: updating wheels * added docs for explaining RnR * updated UV versions to FOSS approved ones, added a flag to run_rnr.sh for debugging (#31) * updated testing documents and README.md for icefabric (#32) * updated existing terraform and docs to have new scripts (#33) * Update Dockerfile.troute to downgrade rocky version * patch: updating YUM dir within the t-route container * Update Dockerfile.troute to include git install * fix: specified uv python version in docker files --------- Co-authored-by: Shawn Crawley <shawn.crawley@noaa.gov> Co-authored-by: Lorne Leonard <lorne.leonard@noaa.gov> Co-authored-by: Drix Tabligan <drix@tabligan.net> Co-authored-by: Nick Chadwick <nick.chadwick@noaa.gov> Co-authored-by: GeorgeRought-NOAA <george.rought@noaa.gov> Co-authored-by: Ryan Grout <groutr@users.noreply.github.com> Co-authored-by: Ryan Grout <ryan.grout@noaa.gov> Co-authored-by: katherine powell <katherine.powell@noaa.gov> Co-authored-by: CatherineFitzpatrick-NOAA <49403615+cfitzpat479@users.noreply.github.com> Co-authored-by: Catherine <catherine.fitzpatrick@noaa.gov> Co-authored-by: George Rought <1061699810113305@mil> Co-authored-by: Rob Hanna - NOAA <90854818+RobHanna-NOAA@users.noreply.github.com> Co-authored-by: Farshid Rahmani <59573719+FRahmani368@users.noreply.github.com> Co-authored-by: root <root@U-1BVT2XY7DCIBM.nextgenwaterprediction.com>
taddyb
added a commit
that referenced
this pull request
Aug 31, 2025
taddyb
added a commit
that referenced
this pull request
Sep 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
a Jupyter file named "test_rnr.ipynb" has been added to /Source/test