Feature 2904 tif fuse case by j-opatz · Pull Request #2987 · dtcenter/METplus

j-opatz · 2025-05-20T23:26:13Z

Pull Request Testing

Describe testing already performed for these changes:
Ran on Seneca, confirmed successful run.
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
check documentation in feature branch for any grammar issues, missing data, etc. -- https://metplus.readthedocs.io/en/feature_2904_tiffusecase/generated/model_applications/short_range/GridStat_fcstGSWR_obsMRMS_GeoTIFF_multiField.html#sphx-glr-generated-model-applications-short-range-gridstat-fcstgswr-obsmrms-geotiff-multifield-py
check GHA tests for any failures NOT related to new data/output
if desired, the use case can be run on Seneca. You will need to activate the rioxarray_env and ensure MET_PYTHON_EXE is set to python3 before it will properly run.
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the test suite? [Yes]

If yes, describe the new output and/or changes to the existing output:
New use case == new truth data
Do these changes introduce new SonarQube findings? [Yes or No]

If yes, please describe:
Please complete this pull request review by [Fill in date].
5/22 (prior to rc1 release!)

Pull Request Checklist

See the METplus Workflow for details.

Add any new Python packages to the METplus Components Python Requirements table.
For any new datasets, an entry to the METplus Verification Datasets Guide.
Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or METplus-Wrappers-X.Y.Z Development project for official releases
After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

… doc rendering

…XE to the version of python in the rioxarray conda env

…n be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided

georgemccabe

I confirmed that the use case ran as expected.

I made a few changes to clean up the docs to fix typos, fix formatting, and fix the path to the thumbnail image.

I also fixed some formatting and thumbnail path issues with another use case that was not being rendered properly.

This PR also contains changes to add a new conda environment (rioxarray) to the automated tests.

I approve. The truth data should be updated after this PR is merged. This could be done after #2985 is also merged.

…empt 2

…empt 3

…empt 4

* Update the instructions for official MET relases with a reminder to create a new version of the MET flowchart. * Feature #2772 Apptainer documentation (#2783) * update version of action to latest * per #2772, added instructions to run a METplus use case using Apptainer * Apply suggestions from code review Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * bugfix dtcenter/MET#2899 prevents differences in output from use case, so removing the 'disabled' item to run use case in GHA * hotfix: remove quotes around input files because logic that obtains python embedding commands as input file arguments handles quotes * fix tests to match new expected commands --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature #2560 Fire GridStat use case (#2769) * added config file and python scripts for fire weather use case * Improved logic if multiple files were found to use the closest file to the valid time. Assuming that the files are in order, it will use the first file if multiple files match the valid hour or the last file if multiple files match 1 hour prior to the valid hour. This does not factor in if files before and after the valid time were found. Modifications are needed to obtain the correct file if, for example, the valid time is 12Z and there are 1258 and 1159 files available -- in this case the 1258 file will be used even though 1159 is closer. * rename use case and update configuration to read data from appropriate directories * per #2560, add new use case to automated tests * turn of use case to prepare for PR * removed comments * added in-progress use case documentation - still needs updates * fix formatting * more formatting * more doc formatting and cleanup * remove notes from sections that were updated * doc cleanup * add thumbnail for new use case * add description of fire use case category * turn off use case * fix SonarQube complaints * update use case doc to match use case template * minor formatting change to match template * update path for kml schema file * use wildcard to avoid having to reference namespace that includes insecure protocol * Update documentation to resemble template --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature #2597 remove manage_externals and its config files (#2785) * Feature #2757 allow missing docs (#2786) * per #2757, add documentation for allow missing input logic * fix incorrect link to glossary item * fix typo * Per #2757, add info and examples for allow missing input logic * Feature 2741 add version (#2793) * adding contents and version * adding contents and version * pbl and precip directory updates * s2s directory updates * short_range directory updates * adding contents and version numbers to last 3 directories * Removing "UPDATE_SECTION_CONTENT" text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update component version script to NOT return develop for an rcN version unless an option is set to override that behavior (in case we need to change how rc is handled). Update automated scripts to get the correct MET Docker images for RC versions (get met-dev/main_vX.Y for RC, get met-dev/develop for betas or dev version, get met/X.Y-latest for official releases). Updated unit tests for component version script to ensure correct behavior occurs * Feature #2789 v6.0.0-rc1 release (#2792) * Per #2779, assigned a number to all use cases that are not run in GHA and added a reason to describe why the use cases have been disabled * adding location information * updating location information * Revert "adding location information" This reverts commit cc6d18506b12d30b931276163f6da52dfaaebb4f. * Revert "updating location information" This reverts commit 5920779fca3a1e5fd1b82538a9b206ef97f76320. * added location of input data that is not found with rest of input data for use case category * added information on how to run generate_release_notes.py script and updated script to clean up some formatting issues * Per #2789, added release notes for rc1 release * update version for rc1 release * fixed broken commands in use case scripts --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use ca…

* update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extens…

* Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) …

* prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) * update instructions to create a new project board to use templates (#3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * develop docs v6.2.0 release timeline (#3083) * Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file. * Feature #3068 DataIngest wrapper (#3084) * clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to sk…

* Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) * update instructions to create a new project board to use templates (#3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * develop docs v6.2.0 release timeline (#3083) * Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file. * Feature #3068 DataIngest wrapper (#3084) * clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to skip replacement of double slash with single slash. The replacement was added to prevent errors with paths that contain double slashes in MET config files, but this causes issues with URLs. * clean up to remove unnecessary variable * Rename DATA_INGEST_<n>_URL/LOCAL_PATH to INPUT/OUTPUT_TEMPLATE to be consistent with other wrappers. Fix incorrect has_option config function name. Improve logging so it is clear that a file is currently being downloaded. Added skip-if-output-exists logic * added basic use case for new wrapper * Handle gz extension as gzip file. Refactor new functions to reduce cognitive complexity * add option to auto decompress files before writing to disk * add use case documentation * add new use case to run in automated test suite * added config options for INPUT/OUTPUT_DIR to be consistent with other wrappers -- added workaround for getdir function to prevent replacement of double slash in URLs * fixed typo * added documentation for config variables for new wrapper * turn off new use case before creating PR * remove test that takes a very long time to download file * removed deprecated config variable * fixed bug in enhancement to config getdir to properly handle skip of double slash handling * improve logging for readability * change lead sequence in example because it is not used * Per #3068, add unit tests for new DataIngest wrapper * fix typo * refactor automated test scripts to reduce duplication in logic to run commands * improve logging so it is more clear that a diff test failed because it is likely that the truth data does not exist yet (e.g. for new use cases) and exit 2 to match return code for a use case that contains actual diffs in the output * Update the METplus-6.1 version of MET from 12.1.0 to 12.1.1 after the first bugfix release. (#3093) * Per #3087, add support for setting message_type_group_map in PB2NC and EnsembleStat wrappers (#3092) * update climo stdev variable name after input data was updated by @j-opatz * change sonar host url to env to view url for quality gate scan results * Fix FSTDEV typo in statistics_list.rst * Update release-notes.rst for the METplus-6.2 Release Schedule * Feature #3028 RMWAnalysis configs (#3094) * Per #3028, added support for many new rmw_analysis MET config options. Refactor logic to read METplus configs and handle MET config env vars to loop over names and handle them consistently. Reordered variables to match new order in RMWAnalysis_default config file * fix missing line break that caused warnings in doc build * change input URLs to obtain files from DTC web server because MADIS data changed since the truth data was created --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #3069 Generate run times using templates (#3089) * removed unused imports * removed unused import * Remove loop_over_times_and_call function because time looping should be handled in RuntimeFreq wrapper. Report an error if a wrapper does not properly inherit from RuntimeFreq or implement its own version of run_all_times * update unit tests because CommandBuilder run_all_times now reports an error * added a fixture to create empty files for testing file I/O logic in the unit tests * cleanup to reduce cognitive complexity, remove unnecessary variable, and clean up whitespace for readability * Per #3069, create get_files_and_time_info function to reuse logic to find files using a directory and template for file window logic and new logic to generate list of times to run based on existing files on disk. Add logic to time_generator to support TIME_GENERATOR_INPUT_DIR/TEMPLATE to generate init or valid times to process. Add unit tests to ensure that correct, sorted list of init/valid times are generated. Add helper function for unit tests to ensure that the expected keys for each time dictionary are present and call helper function from tests that use time_generator to generate run times from beg/end/increment and time list. Added stub to implement similar logic to extract available forecast leads for each init/valid time * Per #3069, use TIME_GENERATOR_INPUT_DIR/TEMPLATE to get list of forecast leads that correspond to the current init or valid time being processed. Updated tests to include this functionality * clean up docstring comments * Per #3069, add tests to ensure that logic to allow multiple time generator dir/templates and get the intersection of the run times works as expected before implementing it * Per #3069, enhance time generator dir/template logic to support multiple templates and get the intersection of the run times * add test for a single directory with multiple templates and add helper functions to reduce duplication * whitespace * add function to intelligently split dir/template values so that the dir will contain the path without any template tags and the template will contain any filename template tags so users can include filename template tags in _INPUT_DIR variables and file window and time generator parsing will still work as expected. Added ample tests to ensure correct behavior * correct comments * update test to include full list of existing wrappers * clean up tests and add test for case that could occur in containers and breaks dir/template split logic * implement fix that properly splits dir/template when a template is used in the root (/) directory * check if either TIME_GENERATOR_INPUT_DIR or _TEMPLATE are set to execute logic to get …

j-opatz and others added 6 commits May 19, 2025 23:48

first pass over doc, basic files. Needs config update and finalize doc

31dae50

add rioxarray conda environment

b1a7777

update out-of-date info

2611982

added image, updated docs, attempt to resolve compile issue

c975847

removing space for error

0076b61

updated remaining files, adjusted docs

1b5fcb0

j-opatz linked an issue May 20, 2025 that may be closed by this pull request

New Use Case: Read and Process GeoTIFF format using python embedding #2904

Closed

25 tasks

j-opatz added this to the METplus-6.1.0 milestone May 20, 2025

github-project-automation bot added this to METplus-6.1 Development May 20, 2025

github-project-automation bot moved this to 🩺 Needs Triage in METplus-6.1 Development May 20, 2025

georgemccabe self-requested a review May 21, 2025 13:00

georgemccabe marked this pull request as ready for review May 21, 2025 13:00

georgemccabe added 5 commits May 21, 2025 07:16

fix incorrect naming for another use case that had errors in use case…

44e7fc6

… doc rendering

fix typos and formatting

7f7a70d

fix path to thumbnail image

b7fa65f

add py_embed keyword to tell automated test suite to set MET_PYTHON_E…

9be8d96

…XE to the version of python in the rioxarray conda env

fix SonarQube complaint "Fix this attribute access on a value that ca…

d3f0e10

…n be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided

georgemccabe moved this from 🩺 Needs Triage to 🔎 In review in METplus-6.1 Development May 21, 2025

georgemccabe approved these changes May 21, 2025

View reviewed changes

j-opatz merged commit 6367067 into develop May 21, 2025
82 of 83 checks passed

github-project-automation bot moved this from 🔎 In review to 🏁 Done in METplus-6.1 Development May 21, 2025

georgemccabe mentioned this pull request May 21, 2025

Bugfix #2890 develop - prevent incorrect error for skipped time #2988

Merged

16 tasks

metplus-bot added a commit that referenced this pull request May 22, 2025

added entry to update truth change log: develop #2985 and #2987

4d5f901

github-actions bot mentioned this pull request May 22, 2025

Update develop-ref after #2985 and #2987 #2990

Merged

georgemccabe pushed a commit that referenced this pull request May 22, 2025

Update develop-ref after #2985 and #2987 (#2990)

05f85db

metplus-bot added a commit that referenced this pull request May 23, 2025

added entry to update truth change log: develop #2985 and #2987 - att…

ca6c32e

…empt 2

github-actions bot mentioned this pull request May 23, 2025

Update develop-ref after #2985 and #2987 - attempt 2 #2992

Merged

georgemccabe pushed a commit that referenced this pull request May 23, 2025

Update develop-ref after #2985 and #2987 - attempt 2 (#2992)

e4253c9

metplus-bot added a commit that referenced this pull request May 23, 2025

added entry to update truth change log: develop #2985 and #2987 - att…

80da069

…empt 3

github-actions bot mentioned this pull request May 23, 2025

Update develop-ref after #2985 and #2987 - attempt 3 #2993

Merged

georgemccabe pushed a commit that referenced this pull request May 23, 2025

Update develop-ref after #2985 and #2987 - attempt 3 (#2993)

780f0f3

metplus-bot added a commit that referenced this pull request May 23, 2025

added entry to update truth change log: develop #2985 and #2987 - att…

9429b0a

…empt 4

github-actions bot mentioned this pull request May 23, 2025

Update develop-ref after #2985 and #2987 - attempt 4 #2994

Merged

georgemccabe pushed a commit that referenced this pull request May 23, 2025

Update develop-ref after #2985 and #2987 - attempt 4 (#2994)

91d3847

georgemccabe deleted the feature_2904_TIFFuseCase branch October 17, 2025 13:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature 2904 tif fuse case#2987

Feature 2904 tif fuse case#2987
j-opatz merged 11 commits intodevelopfrom
feature_2904_TIFFuseCase

j-opatz commented May 20, 2025

Uh oh!

georgemccabe left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

j-opatz commented May 20, 2025

Pull Request Testing

Pull Request Checklist

Uh oh!

georgemccabe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants