Feature #2781 PairStat wrapper by georgemccabe · Pull Request #2872 · dtcenter/METplus

georgemccabe · 2025-01-23T23:22:07Z

Pull Request Testing

Describe testing already performed for these changes:
Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

Review code changes, ensure all tests pass (exceptions described below)

Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]
Do these changes include sufficient testing updates? [Yes]
Will this PR result in changes to the test suite? [Yes]

If yes, describe the new output and/or changes to the existing output:

Most of differences resulting from this pull request are expected. This PR includes refactoring to use the same function to write file list files for the CompareGriddedWrappers (GridStat, PointStat, EnsembleStat, MODE, MTD, and PairStat). Also, the new PairStat basic use case created new output data.

There are differences in one s2s use case that should be investigated.

met_tool_wrapper:0-29,59-64
Rename file list files, e.g.
stage/file_lists/20091231120000_24_ensemble_stat.txt
to
stage/file_lists/ensemble_stat_files_FCST_init_20091231120000_valid_20100101120000_lead_86400.txt

met_tool_wrapper:66
Truth output does not yet exist in DockerHub. We could improve the logging so it is more clear that this is why it fails

air_quality_and_comp:0
Rename file list file
stage/file_lists/20160815000000_12_ensemble_stat.txt
to
stage/file_lists/ensemble_stat_files_FCST_init_20160815000000_valid_20160815120000_lead_43200.txt

preciptation:3-7
Rename file list file
stage/file_lists/20200615170000_1_ensemble_stat.txt
to
stage/file_lists/ensemble_stat_files_FCST_init_20200615170000_valid_20200615180000_lead_3600.txt

s2s_stratosphere:0
Rename file lists files, e.g.
series_analysis_files_fcst_init_ALL_valid_ALL_lead_ALL.txt
and
series_analysis_files_obs_init_ALL_valid_ALL_lead_ALL.txt
to
series_analysis_files_fcst_init_ALL_valid_20180201000000_lead_ALL.txt
and
series_analysis_files_obs_init_ALL_valid_20180201000000_lead_ALL.txt

short_range:0
Rename file list files, e.g.
stage/file_lists/20180709120000_2_ensemble_stat.txt
to
stage/file_lists/ensemble_stat_files_FCST_init_20180709120000_valid_20180709140000_lead_7200.txt

short_range:13
Rename file list files
stage/file_lists/20210201000000_21_mode_fcst.txt
and
stage/file_lists/20210201000000_21_mode_obs.txt
to
stage/file_lists/mode_files_FCST_init_20210201000000_valid_20210201210000_lead_75600.txt
and
stage/file_lists/mode_files_OBS_init_20210201000000_valid_20210201210000_lead_75600.txt

Do these changes introduce new SonarQube findings? [No]

If yes, please describe:
Please complete this pull request review by 1/27/2025.

Pull Request Checklist

See the METplus Workflow for details.

Add any new Python packages to the METplus Components Python Requirements table.
For any new datasets, an entry to the METplus Verification Datasets Guide.
Review the source issue metadata (required labels, projects, and milestone).
Complete the PR definition above.
Ensure the PR title matches the feature or bugfix branch name.
Define the PR metadata, as permissions allow.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or METplus-Wrappers-X.Y.Z Development project for official releases
After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
Close the linked issue and delete your feature or bugfix branch from GitHub.

…ease on mohawk when creating rc1 release instead of after official release

…script

…ng convention. Add command line arguments to pair_stat command

…INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code

…s are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this.

…to test that the fix is correct

…just the first one

…nning

…to multi-variate mode

…d remove unused imports

…documentation to new functions

…te function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers

…ported by pair_stat

JohnHalleyGotway · 2025-01-25T01:45:26Z

@georgemccabe we have differences flagged for many use cases in this testing workflow run. Can you please take a look and advise as to whether or not these diffs are expected for the changes proposed by this pull request?

docs/Users_Guide/glossary.rst

docs/Users_Guide/wrappers.rst

internal/tests/pytests/wrappers/pair_stat/test_pair_stat_wrapper.py

…by only setting custom loop list for the first instance of SeriesAnalysis

…ultiple fields are read from the same file. Removed unused function

…th from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat

georgemccabe · 2025-01-28T14:11:46Z

@JohnHalleyGotway, I made updates to the PairStat wrapper based on your suggestions. I was testing with an older version of the MET tool. The wrapper now handles the output file name base properly. Other wrappers that write files this way, like TCGen, use the *_OUTPUT_DIR and *_OUTPUT_TEMPLATE naming convention. I removed support for setting output_prefix.

I also fixed a bug in SeriesAnalysis that was preventing the correct field information to be set in regards to time, e.g. when data is in a single input file and the field information dictates what to read from the file.

The differences in output are expected and are described in the PR body.

JohnHalleyGotway

I approve of these changes. Thanks for all your work on this @georgemccabe!

I downloaded the diff artifacts and confirmed that they're limited to ASCII file lists, which is fine and expected.
I downloaded the new met_tool_wrapper:66 output and confirmed that it does contain Pair-Stat output.
I stepped through the changes in 51 files and note the following:
- met_tool_wrapper:66 is a new use case group for which output does not yet exist. This is for Pair-Stat tests:66::PairStat:: met_tool_wrapper/PairStat/PairStat.conf
Functionality changes:
- MATCH_MONTH is fully deprecated throughout, in the code and the docs.
Doc changes:
- glossary.rst adds PAIR_STAT entries. Reviewed contents for accuracy and made a couple suggestions.
Config files removed (not entirely clear why but all the test run so that's fine)
- GridStatConfig_prob_precip
- STATAnalysisConfig
- ascii2nc_mask_grid.conf, ascii2nc_mask_poly.conf, ascii2nc_mask_sid.conf, ascii2nc_multi_file.conf, ascii2nc_single_file.conf, grid_gefs_prob.conf
Updates to pytests related to removing MATCH_MONTH.
Consolidation and standardization of time looping logic.

* add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1cac57c431a047ee250e9dae9b0a813a78ba. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666a73685e654e0146d4fed56f2382f1bfc7. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c6e7031c5dafb1177d4b4ca3f32a999dac. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b46db353b4b4709183be2fe7e5ce64ff5ff. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e11096aace3ccc2c79cd631533fc6426900. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b03a45bb871c71e770ff9e602739d6b63d5. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6b0b351ab1b4bca5b563c1f5beef7737a9. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4f0d7851bea720a50fac6011cd381017dc. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809dc3bd73ace969b046062967796035f4d86. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> (cherry picked from commit 44335f33ab152a0b254041961a41895dde614ae0) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe8226c58e9a028a7979664bbf224296fdd6ea. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updated language for tar file Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * Bugfix #2705 develop - SeriesAnalysis fix time substitution into field info (#2709) * fix SonarQube complaints * Per #2705, set correct time information in field info by using the time info used to find the input files (self.c_dict['ALL_FILES']) instead of relying on parsing the time info from the other input files (fcst vs. obs) * clean up unused code after bugfix for #2705 * Feature 2647 use case mv mode for rrfs (#2706) * establish documentation, need to fill out with provided info * updated documentation, needs to have img path and output path fixed * updated output strings, still need to update pic link * removing pound sign to render correctly * Revert "removing pound sign to render correctly" This reverts commit ffe607a96fd33dc5ba15a479f76778dcba7e9656. * targeted removal of pound sign with bullet points section. * Per #2647, attempt to fix syntax error * Per #2647, adding back contents with formatting changes to test if it works * Per #2647, adding back contents in a different location to try to resolve syntax error * Per #2647, testing pipe for horizontal scroll bar * Per #2647, removed the pipe symbol and are attempting to fix syntax error again * Per #2647, reverting to original failed attempt of a table of contents * Per #2647, trying new formatting * Per #2647, fixing new formatting * Per #2647, attempt formatting changes * Per #2647, reverting back to original state * added to automated testing, updated docs, config file * added docs back in * added image, new quick search term, disabled auto tests * testing use_case_groups.json to see if/how this fails * Updated to disable use_case_groups.json run * added use case to list of exceeding disk space --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Bugfix 2675 move to pyproject.toml (#2699) * 2675: move to pyproject.toml * 2675: update toml, remove setup.py ref * bugfix 2675: reinstate VERSION file * bugfix 2675: hardcode version in pyproject.toml * use relative versions * slightly modify import to make it easier to move produtil into metplus * read version from version file * disable install of run_metplus script since it is broken * only install metplus and sub-packages, add version files to package data (parm cannot be added because it is non-package data) * add newline back * install produtil package so metplus can be imported in python * remove unused import * move produtil under metplus to differentiate it from the full NCEP produtil package and isolate it to metplus when installing metplus as a package -- previously produtil would be installed as its own package when metplus is installed * fixed more imports * Create links for parm directory and scripts/run_metplus.py under metplus so the parm files and run script will be installed as part of the metplus python package. Modified config_metplus to find the parm directory from under metplus to be consistent between running via python package and via scripts * Exclude parm from sonar sources because it will be found under metplus. Add metplus/parm/** to list of coverage exclusions * fixed more imports * add parm back to list of sonar sources --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Removing stated location of the version selector menu * Feature 2489 update docs (#2717) * adding new Literal Include section * under bold section adding "Environment" to Variables * Adding Substitution References section * removing blank space * fixing code typos * another typo * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2540 Point2Grid config file (#2580) * per #2540, add support for setting values in a config file for Point2Grid wrapper -- removed -qc flag argument in favor of setting obs_quality_inc in the config file * Added support for setting time_offset_warning now that the changes for other wrappers was merged into develop. Also added support for MET_CONFIG_OVERRIDES for Point2Grid * per #2540, set -goes_qc command line argument from POINT2GRID_GOES_QC_FLAGS and continue to support legacy POINT2GRID_QC_FLAGS. Other fixes to resolve incorrectly resolved conflicts from previous commit merging develop into branch * per #2540, remove quality_mark_thresh from config file because it has been removed from MET point2grid * Per discussion in PR #2580, remove time_offset_warning support from tools that don't actually use the value -- to add support for a new wrapper, update metplus/util/constants.py TIME_OFFSET_WARNING_WRAPPERS to include the wrapper's name * removed env vars for time_offset_warning for wrappers that no longer set it * install ImageMagick convert because it appears to no longer be available in the image that runs the unit tests (ubuntu 24.04?) * Bugfix metdbload config (#2728) * Update METdbLoadConfig.xml A schema is now used to validate all XML specification files and elements must adhere to a particular order. The <group> element follows the headers and load elements and preceeds the folder tmpl element. * Update use_case_groups.json temporarily run tests for the short range use cases * Update use_case_groups.json Return all the short_range use case test "run" values back to false * install convert because some tests fail if it is not available * Feature #1882 docs: new use case that uses METplus Analysis (#2726) * Fixing mistake * Fixing reference * Fixing link * Updating sections * clean up new content --------- Co-authored-by: Christina Kalb <kalb@ucar.edu> * Feature 918 add use case template (#2690) * updated template, 1st part * added more to template, updated internal references. Needs final two cats * finished updating template * updated syntax, length of lines * moved example to separate file, testing indents for discussion * added new collapse menu for MET configs, documentation on adding keywords, testing 2 separate options to stop paragraph jumping during bullet lists * updated spacing used for 2nd line indentation * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Minor changes, updates hyperlink to template to be directly to the file and not to the directoy in GitHub. * Update add_use_case.rst * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update add_use_case.rst * Missing space in hyperlink. * Changes to use a keyword to assist users in checking off each section they should review and also PR reviewers in ensuring each section has been visited and reviewed by the contributor. * updated Python section to User Scripting --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2727 point_weight_flag and obtype_as_group_val_flag (#2730) * Per #2727, add support for setting point_weight_flag * Per #2727, add support for obtype_as_group_val_flag * per #2714, deprecate master_metplus.py * Update VERSION * Feature #2710 - clean up use cases (#2711) * per #2710, remove line that has no effect because next line overwrites value * per #2710, fix bug in if/else block where both were doing the same thing. Removed commented code and create variable for constant to satisfy SonarQube * updated config file commands * updated level of winds to capture change in script --------- Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature #2682 Component version lookup (#2733) * per #2714, deprecate master_metplus.py * exclude metplus/scripts from SonarQube scan to prevent incorrect duplicate code error since script lives in ush * added Dockerfile to build image with MET/METplus and METplus Analysis Python packages * start of script to get versions of METplus components * per #2682, created METplus component version lookup table and functions to get formatted version of requested components * add descriptions for tests * add update information and made script callable * made script executable * add workflow to trigger on release and create dtcenter/metplus and dtcenter/metplus-analysis Docker images * add scripts used by new workflow * removed build hook scripts since we will be building images via GHA * change logic to get MET version using component version script instead of build hook file that just adds 6 to major version * add missing shebang * make script callable directly and make output component a required input * pass LATEST_TAG to push script * ensure only 1st line of version file is read to avoid newline * turn on use case to test * turn off use case * Feature #2735 v6.0.0 beta6 (#2736) * change version for beta6 release * added script to help generate release notes for development releases * add default title for issues and added documentation issue template * sort issues by number, improve formatting for release notes title, add logging to alert users what is happening when timely github queries are running * added release notes for beta6 release * Add option to component version script to return 'develop' if the input version is a beta or rc version to preserve previous behavior of GHA scripts. Update GHA scripts to use new option * applied suggestions from feedback in PR #2736 * bold some release notes * update version for development towards rc1 release * fix logic to determine offical X.Y.Z release * Feature #2739 labels (#2740) * Per #2739, add NRL reporting label and add METplus-Internal to the list of repositories to be updated. * Per #2739, add info about labels to the Contributor's Guide. * Per #2739, update the list of METplus repos in process_labels.sh but remove them from the other shell scripts to avoid keeping the same list updated in multiple spots. * Per #2739, move label details from the README to the Contributor's Guide * Updated release notes adding this issue: Develop an RST template for use cases * Feature 2716 existing builds (#2748) * adding the first 4 paragraphs * fixing web addresses * Highest level of dropdown menus added * NCAR machines dropdown menus * CASPER dropdown menu * web addresses can't be bolded. Fixing dropdown problems * blank line problem * italics date adding Derechio parts * DERECHO dropdown * fixing problems * fixing mor problems * fixing bolding * trying to fix web address * adding NCAR RAL MACHINES and trying to fix website issue * NCAR/RAL Internal Development dropdown * adding remaining dropdown structure * WCOSS2 dropdown * HERA dropdown * fixing last updated to be consistent * HERCULES dropdown * fixing pip * orion dropdown * JET dropdown * GAEA dropdown * fixing typo * Community machines, Docker Hub & AWS * fixing typo * Changing to "Users should create a file like" * to set a personalized input... * adding last updated and Docker Hub web links * Updating Jet information after recent install and changing some formatting for consistency * Removing bolding for consistency * creating a separate file for existing builds to add to 2 areas make it section 2.1 where METplus Components Release Note Links and Release Notes * adding existing_builds as an include * Rename existing_builds_rst to existing_builds.rst I had an underline instead of a period * changing to a section, not a chapter * trying a different section * changing existing builds to an include file * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing existing builds completely from the Release Guide --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update release-notes.rst * add custom string to time info dictionary for substituting values into the field info string * Feature #2742 develop fix StatAnalysis to set fcst_lev/obs_lev (#2760) * Feature 2763 existing builds (#2765) * Per #2763, updated build for Casper * Per #2763, updating Orion * Bugfix #2762 PCPCombine derive field info (#2764) * per #2762, change derive mode to add field info for each file instead of adding the first field info only * Per #2762, improve how the accumulation amounts are handling using relativedelta to ensure that the correct number of seconds are used for each search window when using inconsistent time intervals like months or years * Bugfix #2762 part 2 -- fix PCPCombine leap year issue (#2766) * remove check if script is master_metplus.py because it has been deprecated * per #2762, fix bug that caused failure depending on valid time relating to leap years * Feature #2754 rc1_main_vX.Y (#2755) * Per #2754, update instructions to create the main_vX.Y and main_vX.Y-ref branches (when applicable) for the first release candidate rather than waiting until the official release. Note that I deleted the push_release_branch.rst instructions since it's not needed. The preceeding instructions execute the push and these are redundant - even for METplus which has custom instructions. * Per #2754, add a note to make the detail stand out better * Per #2754, make the GitHub release instructions more explicit to match the names of the actual fields on GitHub. * Per #2754, not changing comment, just using more consisent formatting options. * Per #2754, change example from 'betaN' to 'rc1' since this step is only run for the rc1 release. * Per #2754, use consistent single quotes instead of italics for formatting. * Updates to METplus instructions to properly handle version numbers for rc1 release. Update METplus Official release instructions to use main branch and update version number for official release since main branch will contain -rcN version numbers * update instructions to checkout correct branch * clarify next deveopment version can be X+1.0 OR X.Y+1 * update version table for other components -- this could likely be refactored to make the version file path a variable and include the same update_version_on_develop.rst file for METplus, METcalcpy, METdataio, METplotpy, and METviewer --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Resolving use case template formatting problems and adding dropdowns * Resolving use case template formatting problem * Fixing formatting * Updating to add language if no Python embedding or User Scripting is used. * Feature #2562 version lookup documentation and automated tests (#2773) * set default input version to the latest official release, which is the 2nd highest coordinated release version number in the version lookup table. This assumes that the highest version is what is being developed towards * per #2562, update Release Guide to include instructions for updating the component versions lookup table in the METplus repository * Per #2562 and #2597, replace automated test logic that used manage_externals to use component version lookup to get versions of METplotpy/METcalcpy/METdataio * clean up SonarQube complaints * turn on some use cases to test changes * fix clone * do not get python dependencies for METplotpy/METcalcpy/METdataio installs * use full path to package to import * clone METplus Analysis repos on the same level as METplus to match previous location using manage_externals, remove manage_externals config files that are no longer used by the automated tests * clean up linter complaints * turn off use cases after confirming that they run successfully * clean up usage statement formatting * remove info about updating manage_externals config files used in automated tests that have been removed * added description of component versions script to Contributor's Guide and added stubs for other utility descriptions * change format of example results and add text to describe what is being demonstrated from each example * fix formatting * fixed bad variable replacement * Update docs/Contributors_Guide/utilities.rst --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature 2737 release instruct (#2775) * Per #2737 updating version instructions * Per #2737, update version instructions for METcalcpy * Per #2737, renaming development file * Per #2737, update version instructions for METplotpy * Per #2737, updating version instructions for METdataio * Per #2737, updating version instructions for METviewer * Per #2737, adding pyproject.toml to the list of files to update * Per #2751, this work was actually completed already, adding text to make the directions more clear. (#2776) * Made minor modifications for consistency * Feature 2339 dependencies (#2777) * Per #2339, update documention to include info about dependencies and coordinated release announcement * Per #2339, renaming file and header * Per #2339, correcting typo in formatting * Per #2339, updated formatting * Per #2339, updating additional dependencies * Per #2300, add Python package disclaimer for the METplus components (#2782) * Feature #2666 sonarqube (#2784) * Unrelated to #2666 but found while working on it, update the expected output file names for the TCDiag basic met_tool use case. * Per #2666, add details about SonarQube. Still need to proof-read and wordsmith * Per #2666, more SonarQube details * Per #2666, fix up anchor links, switch to *'s for bullets like we use everywhere else, and massage the wording * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update the instructions for official MET relases with a reminder to create a new version of the MET flowchart. * Feature #2772 Apptainer documentation (#2783) * update version of action to latest * per #2772, added instructions to run a METplus use case using Apptainer * Apply suggestions from code review Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * bugfix dtcenter/MET#2899 prevents differences in output from use case, so removing the 'disabled' item to run use case in GHA * hotfix: remove quotes around input files because logic that obtains python embedding commands as input file arguments handles quotes * fix tests to match new expected commands --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature #2560 Fire GridStat use case (#2769) * added config file and python scripts for fire weather use case * Improved logic if multiple files were found to use the closest file to the valid time. Assuming that the files are in order, it will use the first file if multiple files match the valid hour or the last file if multiple files match 1 hour prior to the valid hour. This does not factor in if files before and after the valid time were found. Modifications are needed to obtain the correct file if, for example, the valid time is 12Z and there are 1258 and 1159 files available -- in this case the 1258 file will be used even though 1159 is closer. * rename use case and update configuration to read data from appropriate directories * per #2560, add new use case to automated tests * turn of use case to prepare for PR * removed comments * added in-progress use case documentation - still needs updates * fix formatting * more formatting * more doc formatting and cleanup * remove notes from sections that were updated * doc cleanup * add thumbnail for new use case * add description of fire use case category * turn off use case * fix SonarQube complaints * update use case doc to match use case template * minor formatting change to match template * update path for kml schema file * use wildcard to avoid having to reference namespace that includes insecure protocol * Update documentation to resemble template --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature #2597 remove manage_externals and its config files (#2785) * Feature #2757 allow missing docs (#2786) * per #2757, add documentation for allow missing input logic * fix incorrect link to glossary item * fix typo * Per #2757, add info and examples for allow missing input logic * Feature 2741 add version (#2793) * adding contents and version * adding contents and version * pbl and precip directory updates * s2s directory updates * short_range directory updates * adding contents and version numbers to last 3 directories * Removing "UPDATE_SECTION_CONTENT" text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update component version script to NOT return develop for an rcN version unless an option is set to override that behavior (in case we need to change how rc is handled). Update automated scripts to get the correct MET Docker images for RC versions (get met-dev/main_vX.Y for RC, get met-dev/develop for betas or dev version, get met/X.Y-latest for official releases). Updated unit tests for component version script to ensure correct behavior occurs * Feature #2789 v6.0.0-rc1 release (#2792) * Per #2779, assigned a number to all use cases that are not run in GHA and added a reason to describe why the use cases have been disabled * adding location information * updating location information * Revert "adding location information" This reverts commit cc6d18506b12d30b931276163f6da52dfaaebb4f. * Revert "updating location information" This reverts commit 5920779fca3a1e5fd1b82538a9b206ef97f76320. * added location of input data that is not found with rest of input data for use case category * added information on how to run generate_release_notes.py script and updated script to clean up some formatting issues * Per #2789, added release notes for rc1 release * update version for rc1 release * fixed broken commands in use case scripts --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line …

* Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> (cherry picked from commit 44335f33ab152a0b254041961a41895dde614ae0) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe8226c58e9a028a7979664bbf224296fdd6ea. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updated language for tar file Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * Bugfix #2705 develop - SeriesAnalysis fix time substitution into field info (#2709) * fix SonarQube complaints * Per #2705, set correct time information in field info by using the time info used to find the input files (self.c_dict['ALL_FILES']) instead of relying on parsing the time info from the other input files (fcst vs. obs) * clean up unused code after bugfix for #2705 * Feature 2647 use case mv mode for rrfs (#2706) * establish documentation, need to fill out with provided info * updated documentation, needs to have img path and output path fixed * updated output strings, still need to update pic link * removing pound sign to render correctly * Revert "removing pound sign to render correctly" This reverts commit ffe607a96fd33dc5ba15a479f76778dcba7e9656. * targeted removal of pound sign with bullet points section. * Per #2647, attempt to fix syntax error * Per #2647, adding back contents with formatting changes to test if it works * Per #2647, adding back contents in a different location to try to resolve syntax error * Per #2647, testing pipe for horizontal scroll bar * Per #2647, removed the pipe symbol and are attempting to fix syntax error again * Per #2647, reverting to original failed attempt of a table of contents * Per #2647, trying new formatting * Per #2647, fixing new formatting * Per #2647, attempt formatting changes * Per #2647, reverting back to original state * added to automated testing, updated docs, config file * added docs back in * added image, new quick search term, disabled auto tests * testing use_case_groups.json to see if/how this fails * Updated to disable use_case_groups.json run * added use case to list of exceeding disk space --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Bugfix 2675 move to pyproject.toml (#2699) * 2675: move to pyproject.toml * 2675: update toml, remove setup.py ref * bugfix 2675: reinstate VERSION file * bugfix 2675: hardcode version in pyproject.toml * use relative versions * slightly modify import to make it easier to move produtil into metplus * read version from version file * disable install of run_metplus script since it is broken * only install metplus and sub-packages, add version files to package data (parm cannot be added because it is non-package data) * add newline back * install produtil package so metplus can be imported in python * remove unused import * move produtil under metplus to differentiate it from the full NCEP produtil package and isolate it to metplus when installing metplus as a package -- previously produtil would be installed as its own package when metplus is installed * fixed more imports * Create links for parm directory and scripts/run_metplus.py under metplus so the parm files and run script will be installed as part of the metplus python package. Modified config_metplus to find the parm directory from under metplus to be consistent between running via python package and via scripts * Exclude parm from sonar sources because it will be found under metplus. Add metplus/parm/** to list of coverage exclusions * fixed more imports * add parm back to list of sonar sources --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Removing stated location of the version selector menu * Feature 2489 update docs (#2717) * adding new Literal Include section * under bold section adding "Environment" to Variables * Adding Substitution References section * removing blank space * fixing code typos * another typo * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2540 Point2Grid config file (#2580) * per #2540, add support for setting values in a config file for Point2Grid wrapper -- removed -qc flag argument in favor of setting obs_quality_inc in the config file * Added support for setting time_offset_warning now that the changes for other wrappers was merged into develop. Also added support for MET_CONFIG_OVERRIDES for Point2Grid * per #2540, set -goes_qc command line argument from POINT2GRID_GOES_QC_FLAGS and continue to support legacy POINT2GRID_QC_FLAGS. Other fixes to resolve incorrectly resolved conflicts from previous commit merging develop into branch * per #2540, remove quality_mark_thresh from config file because it has been removed from MET point2grid * Per discussion in PR #2580, remove time_offset_warning support from tools that don't actually use the value -- to add support for a new wrapper, update metplus/util/constants.py TIME_OFFSET_WARNING_WRAPPERS to include the wrapper's name * removed env vars for time_offset_warning for wrappers that no longer set it * install ImageMagick convert because it appears to no longer be available in the image that runs the unit tests (ubuntu 24.04?) * Bugfix metdbload config (#2728) * Update METdbLoadConfig.xml A schema is now used to validate all XML specification files and elements must adhere to a particular order. The <group> element follows the headers and load elements and preceeds the folder tmpl element. * Update use_case_groups.json temporarily run tests for the short range use cases * Update use_case_groups.json Return all the short_range use case test "run" values back to false * install convert because some tests fail if it is not available * Feature #1882 docs: new use case that uses METplus Analysis (#2726) * Fixing mistake * Fixing reference * Fixing link * Updating sections * clean up new content --------- Co-authored-by: Christina Kalb <kalb@ucar.edu> * Feature 918 add use case template (#2690) * updated template, 1st part * added more to template, updated internal references. Needs final two cats * finished updating template * updated syntax, length of lines * moved example to separate file, testing indents for discussion * added new collapse menu for MET configs, documentation on adding keywords, testing 2 separate options to stop paragraph jumping during bullet lists * updated spacing used for 2nd line indentation * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Minor changes, updates hyperlink to template to be directly to the file and not to the directoy in GitHub. * Update add_use_case.rst * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update add_use_case.rst * Missing space in hyperlink. * Changes to use a keyword to assist users in checking off each section they should review and also PR reviewers in ensuring each section has been visited and reviewed by the contributor. * updated Python section to User Scripting --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2727 point_weight_flag and obtype_as_group_val_flag (#2730) * Per #2727, add support for setting point_weight_flag * Per #2727, add support for obtype_as_group_val_flag * per #2714, deprecate master_metplus.py * Update VERSION * Feature #2710 - clean up use cases (#2711) * per #2710, remove line that has no effect because next line overwrites value * per #2710, fix bug in if/else block where both were doing the same thing. Removed commented code and create variable for constant to satisfy SonarQube * updated config file commands * updated level of winds to capture change in script --------- Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature #2682 Component version lookup (#2733) * per #2714, deprecate master_metplus.py * exclude metplus/scripts from SonarQube scan to prevent incorrect duplicate code error since script lives in ush * added Dockerfile to build image with MET/METplus and METplus Analysis Python packages * start of script to get versions of METplus components * per #2682, created METplus component version lookup table and functions to get formatted version of requested components * add descriptions for tests * add update information and made script callable * made script executable * add workflow to trigger on release and create dtcenter/metplus and dtcenter/metplus-analysis Docker images * add scripts used by new workflow * removed build hook scripts since we will be building images via GHA * change logic to get MET version using component version script instead of build hook file that just adds 6 to major version * add missing shebang * make script callable directly and make output component a required input * pass LATEST_TAG to push script * ensure only 1st line of version file is read to avoid newline * turn on use case to test * turn off use case * Feature #2735 v6.0.0 beta6 (#2736) * change version for beta6 release * added script to help generate release notes for development releases * add default title for issues and added documentation issue template * sort issues by number, improve formatting for release notes title, add logging to alert users what is happening when timely github queries are running * added release notes for beta6 release * Add option to component version script to return 'develop' if the input version is a beta or rc version to preserve previous behavior of GHA scripts. Update GHA scripts to use new option * applied suggestions from feedback in PR #2736 * bold some release notes * update version for development towards rc1 release * fix logic to determine offical X.Y.Z release * Feature #2739 labels (#2740) * Per #2739, add NRL reporting label and add METplus-Internal to the list of repositories to be updated. * Per #2739, add info about labels to the Contributor's Guide. * Per #2739, update the list of METplus repos in process_labels.sh but remove them from the other shell scripts to avoid keeping the same list updated in multiple spots. * Per #2739, move label details from the README to the Contributor's Guide * Updated release notes adding this issue: Develop an RST template for use cases * Feature 2716 existing builds (#2748) * adding the first 4 paragraphs * fixing web addresses * Highest level of dropdown menus added * NCAR machines dropdown menus * CASPER dropdown menu * web addresses can't be bolded. Fixing dropdown problems * blank line problem * italics date adding Derechio parts * DERECHO dropdown * fixing problems * fixing mor problems * fixing bolding * trying to fix web address * adding NCAR RAL MACHINES and trying to fix website issue * NCAR/RAL Internal Development dropdown * adding remaining dropdown structure * WCOSS2 dropdown * HERA dropdown * fixing last updated to be consistent * HERCULES dropdown * fixing pip * orion dropdown * JET dropdown * GAEA dropdown * fixing typo * Community machines, Docker Hub & AWS * fixing typo * Changing to "Users should create a file like" * to set a personalized input... * adding last updated and Docker Hub web links * Updating Jet information after recent install and changing some formatting for consistency * Removing bolding for consistency * creating a separate file for existing builds to add to 2 areas make it section 2.1 where METplus Components Release Note Links and Release Notes * adding existing_builds as an include * Rename existing_builds_rst to existing_builds.rst I had an underline instead of a period * changing to a section, not a chapter * trying a different section * changing existing builds to an include file * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing existing builds completely from the Release Guide --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update release-notes.rst * add custom string to time info dictionary for substituting values into the field info string * Feature #2742 develop fix StatAnalysis to set fcst_lev/obs_lev (#2760) * Feature 2763 existing builds (#2765) * Per #2763, updated build for Casper * Per #2763, updating Orion * Bugfix #2762 PCPCombine derive field info (#2764) * per #2762, change derive mode to add field info for each file instead of adding the first field info only * Per #2762, improve how the accumulation amounts are handling using relativedelta to ensure that the correct number of seconds are used for each search window when using inconsistent time intervals like months or years * Bugfix #2762 part 2 -- fix PCPCombine leap year issue (#2766) * remove check if script is master_metplus.py because it has been deprecated * per #2762, fix bug that caused failure depending on valid time relating to leap years * Feature #2754 rc1_main_vX.Y (#2755) * Per #2754, update instructions to create the main_vX.Y and main_vX.Y-ref branches (when applicable) for the first release candidate rather than waiting until the official release. Note that I deleted the push_release_branch.rst instructions since it's not needed. The preceeding instructions execute the push and these are redundant - even for METplus which has custom instructions. * Per #2754, add a note to make the detail stand out better * Per #2754, make the GitHub release instructions more explicit to match the names of the actual fields on GitHub. * Per #2754, not changing comment, just using more consisent formatting options. * Per #2754, change example from 'betaN' to 'rc1' since this step is only run for the rc1 release. * Per #2754, use consistent single quotes instead of italics for formatting. * Updates to METplus instructions to properly handle version numbers for rc1 release. Update METplus Official release instructions to use main branch and update version number for official release since main branch will contain -rcN version numbers * update instructions to checkout correct branch * clarify next deveopment version can be X+1.0 OR X.Y+1 * update version table for other components -- this could likely be refactored to make the version file path a variable and include the same update_version_on_develop.rst file for METplus, METcalcpy, METdataio, METplotpy, and METviewer --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Resolving use case template formatting problems and adding dropdowns * Resolving use case template formatting problem * Fixing formatting * Updating to add language if no Python embedding or User Scripting is used. * Feature #2562 version lookup documentation and automated tests (#2773) * set default input version to the latest official release, which is the 2nd highest coordinated release version number in the version lookup table. This assumes that the highest version is what is being developed towards * per #2562, update Release Guide to include instructions for updating the component versions lookup table in the METplus repository * Per #2562 and #2597, replace automated test logic that used manage_externals to use component version lookup to get versions of METplotpy/METcalcpy/METdataio * clean up SonarQube complaints * turn on some use cases to test changes * fix clone * do not get python dependencies for METplotpy/METcalcpy/METdataio installs * use full path to package to import * clone METplus Analysis repos on the same level as METplus to match previous location using manage_externals, remove manage_externals config files that are no longer used by the automated tests * clean up linter complaints * turn off use cases after confirming that they run successfully * clean up usage statement formatting * remove info about updating manage_externals config files used in automated tests that have been removed * added description of component versions script to Contributor's Guide and added stubs for other utility descriptions * change format of example results and add text to describe what is being demonstrated from each example * fix formatting * fixed bad variable replacement * Update docs/Contributors_Guide/utilities.rst --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature 2737 release instruct (#2775) * Per #2737 updating version instructions * Per #2737, update version instructions for METcalcpy * Per #2737, renaming development file * Per #2737, update version instructions for METplotpy * Per #2737, updating version instructions for METdataio * Per #2737, updating version instructions for METviewer * Per #2737, adding pyproject.toml to the list of files to update * Per #2751, this work was actually completed already, adding text to make the directions more clear. (#2776) * Made minor modifications for consistency * Feature 2339 dependencies (#2777) * Per #2339, update documention to include info about dependencies and coordinated release announcement * Per #2339, renaming file and header * Per #2339, correcting typo in formatting * Per #2339, updated formatting * Per #2339, updating additional dependencies * Per #2300, add Python package disclaimer for the METplus components (#2782) * Feature #2666 sonarqube (#2784) * Unrelated to #2666 but found while working on it, update the expected output file names for the TCDiag basic met_tool use case. * Per #2666, add details about SonarQube. Still need to proof-read and wordsmith * Per #2666, more SonarQube details * Per #2666, fix up anchor links, switch to *'s for bullets like we use everywhere else, and massage the wording * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update the instructions for official MET relases with a reminder to create a new version of the MET flowchart. * Feature #2772 Apptainer documentation (#2783) * update version of action to latest * per #2772, added instructions to run a METplus use case using Apptainer * Apply suggestions from code review Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * bugfix dtcenter/MET#2899 prevents differences in output from use case, so removing the 'disabled' item to run use case in GHA * hotfix: remove quotes around input files because logic that obtains python embedding commands as input file arguments handles quotes * fix tests to match new expected commands --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature #2560 Fire GridStat use case (#2769) * added config file and python scripts for fire weather use case * Improved logic if multiple files were found to use the closest file to the valid time. Assuming that the files are in order, it will use the first file if multiple files match the valid hour or the last file if multiple files match 1 hour prior to the valid hour. This does not factor in if files before and after the valid time were found. Modifications are needed to obtain the correct file if, for example, the valid time is 12Z and there are 1258 and 1159 files available -- in this case the 1258 file will be used even though 1159 is closer. * rename use case and update configuration to read data from appropriate directories * per #2560, add new use case to automated tests * turn of use case to prepare for PR * removed comments * added in-progress use case documentation - still needs updates * fix formatting * more formatting * more doc formatting and cleanup * remove notes from sections that were updated * doc cleanup * add thumbnail for new use case * add description of fire use case category * turn off use case * fix SonarQube complaints * update use case doc to match use case template * minor formatting change to match template * update path for kml schema file * use wildcard to avoid having to reference namespace that includes insecure protocol * Update documentation to resemble template --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature #2597 remove manage_externals and its config files (#2785) * Feature #2757 allow missing docs (#2786) * per #2757, add documentation for allow missing input logic * fix incorrect link to glossary item * fix typo * Per #2757, add info and examples for allow missing input logic * Feature 2741 add version (#2793) * adding contents and version * adding contents and version * pbl and precip directory updates * s2s directory updates * short_range directory updates * adding contents and version numbers to last 3 directories * Removing "UPDATE_SECTION_CONTENT" text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update component version script to NOT return develop for an rcN version unless an option is set to override that behavior (in case we need to change how rc is handled). Update automated scripts to get the correct MET Docker images for RC versions (get met-dev/main_vX.Y for RC, get met-dev/develop for betas or dev version, get met/X.Y-latest for official releases). Updated unit tests for component version script to ensure correct behavior occurs * Feature #2789 v6.0.0-rc1 release (#2792) * Per #2779, assigned a number to all use cases that are not run in GHA and added a reason to describe why the use cases have been disabled * adding location information * updating location information * Revert "adding location information" This reverts commit cc6d18506b12d30b931276163f6da52dfaaebb4f. * Revert "updating location information" This reverts commit 5920779fca3a1e5fd1b82538a9b206ef97f76320. * added location of input data that is not found with rest of input data for use case category * added information on how to run generate_release_notes.py script and updated script to clean up some formatting issues * Per #2789, added release notes for rc1 release * update version for rc1 release * fixed broken commands in use case scripts --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development …

* update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <johnhg@ucar.edu> (cherry picked from commit 44335f33ab152a0b254041961a41895dde614ae0) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe8226c58e9a028a7979664bbf224296fdd6ea. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updated language for tar file Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * Bugfix #2705 develop - SeriesAnalysis fix time substitution into field info (#2709) * fix SonarQube complaints * Per #2705, set correct time information in field info by using the time info used to find the input files (self.c_dict['ALL_FILES']) instead of relying on parsing the time info from the other input files (fcst vs. obs) * clean up unused code after bugfix for #2705 * Feature 2647 use case mv mode for rrfs (#2706) * establish documentation, need to fill out with provided info * updated documentation, needs to have img path and output path fixed * updated output strings, still need to update pic link * removing pound sign to render correctly * Revert "removing pound sign to render correctly" This reverts commit ffe607a96fd33dc5ba15a479f76778dcba7e9656. * targeted removal of pound sign with bullet points section. * Per #2647, attempt to fix syntax error * Per #2647, adding back contents with formatting changes to test if it works * Per #2647, adding back contents in a different location to try to resolve syntax error * Per #2647, testing pipe for horizontal scroll bar * Per #2647, removed the pipe symbol and are attempting to fix syntax error again * Per #2647, reverting to original failed attempt of a table of contents * Per #2647, trying new formatting * Per #2647, fixing new formatting * Per #2647, attempt formatting changes * Per #2647, reverting back to original state * added to automated testing, updated docs, config file * added docs back in * added image, new quick search term, disabled auto tests * testing use_case_groups.json to see if/how this fails * Updated to disable use_case_groups.json run * added use case to list of exceeding disk space --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Bugfix 2675 move to pyproject.toml (#2699) * 2675: move to pyproject.toml * 2675: update toml, remove setup.py ref * bugfix 2675: reinstate VERSION file * bugfix 2675: hardcode version in pyproject.toml * use relative versions * slightly modify import to make it easier to move produtil into metplus * read version from version file * disable install of run_metplus script since it is broken * only install metplus and sub-packages, add version files to package data (parm cannot be added because it is non-package data) * add newline back * install produtil package so metplus can be imported in python * remove unused import * move produtil under metplus to differentiate it from the full NCEP produtil package and isolate it to metplus when installing metplus as a package -- previously produtil would be installed as its own package when metplus is installed * fixed more imports * Create links for parm directory and scripts/run_metplus.py under metplus so the parm files and run script will be installed as part of the metplus python package. Modified config_metplus to find the parm directory from under metplus to be consistent between running via python package and via scripts * Exclude parm from sonar sources because it will be found under metplus. Add metplus/parm/** to list of coverage exclusions * fixed more imports * add parm back to list of sonar sources --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Removing stated location of the version selector menu * Feature 2489 update docs (#2717) * adding new Literal Include section * under bold section adding "Environment" to Variables * Adding Substitution References section * removing blank space * fixing code typos * another typo * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2540 Point2Grid config file (#2580) * per #2540, add support for setting values in a config file for Point2Grid wrapper -- removed -qc flag argument in favor of setting obs_quality_inc in the config file * Added support for setting time_offset_warning now that the changes for other wrappers was merged into develop. Also added support for MET_CONFIG_OVERRIDES for Point2Grid * per #2540, set -goes_qc command line argument from POINT2GRID_GOES_QC_FLAGS and continue to support legacy POINT2GRID_QC_FLAGS. Other fixes to resolve incorrectly resolved conflicts from previous commit merging develop into branch * per #2540, remove quality_mark_thresh from config file because it has been removed from MET point2grid * Per discussion in PR #2580, remove time_offset_warning support from tools that don't actually use the value -- to add support for a new wrapper, update metplus/util/constants.py TIME_OFFSET_WARNING_WRAPPERS to include the wrapper's name * removed env vars for time_offset_warning for wrappers that no longer set it * install ImageMagick convert because it appears to no longer be available in the image that runs the unit tests (ubuntu 24.04?) * Bugfix metdbload config (#2728) * Update METdbLoadConfig.xml A schema is now used to validate all XML specification files and elements must adhere to a particular order. The <group> element follows the headers and load elements and preceeds the folder tmpl element. * Update use_case_groups.json temporarily run tests for the short range use cases * Update use_case_groups.json Return all the short_range use case test "run" values back to false * install convert because some tests fail if it is not available * Feature #1882 docs: new use case that uses METplus Analysis (#2726) * Fixing mistake * Fixing reference * Fixing link * Updating sections * clean up new content --------- Co-authored-by: Christina Kalb <kalb@ucar.edu> * Feature 918 add use case template (#2690) * updated template, 1st part * added more to template, updated internal references. Needs final two cats * finished updating template * updated syntax, length of lines * moved example to separate file, testing indents for discussion * added new collapse menu for MET configs, documentation on adding keywords, testing 2 separate options to stop paragraph jumping during bullet lists * updated spacing used for 2nd line indentation * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Minor changes, updates hyperlink to template to be directly to the file and not to the directoy in GitHub. * Update add_use_case.rst * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update add_use_case.rst * Missing space in hyperlink. * Changes to use a keyword to assist users in checking off each section they should review and also PR reviewers in ensuring each section has been visited and reviewed by the contributor. * updated Python section to User Scripting --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2727 point_weight_flag and obtype_as_group_val_flag (#2730) * Per #2727, add support for setting point_weight_flag * Per #2727, add support for obtype_as_group_val_flag * per #2714, deprecate master_metplus.py * Update VERSION * Feature #2710 - clean up use cases (#2711) * per #2710, remove line that has no effect because next line overwrites value * per #2710, fix bug in if/else block where both were doing the same thing. Removed commented code and create variable for constant to satisfy SonarQube * updated config file commands * updated level of winds to capture change in script --------- Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature #2682 Component version lookup (#2733) * per #2714, deprecate master_metplus.py * exclude metplus/scripts from SonarQube scan to prevent incorrect duplicate code error since script lives in ush * added Dockerfile to build image with MET/METplus and METplus Analysis Python packages * start of script to get versions of METplus components * per #2682, created METplus component version lookup table and functions to get formatted version of requested components * add descriptions for tests * add update information and made script callable * made script executable * add workflow to trigger on release and create dtcenter/metplus and dtcenter/metplus-analysis Docker images * add scripts used by new workflow * removed build hook scripts since we will be building images via GHA * change logic to get MET version using component version script instead of build hook file that just adds 6 to major version * add missing shebang * make script callable directly and make output component a required input * pass LATEST_TAG to push script * ensure only 1st line of version file is read to avoid newline * turn on use case to test * turn off use case * Feature #2735 v6.0.0 beta6 (#2736) * change version for beta6 release * added script to help generate release notes for development releases * add default title for issues and added documentation issue template * sort issues by number, improve formatting for release notes title, add logging to alert users what is happening when timely github queries are running * added release notes for beta6 release * Add option to component version script to return 'develop' if the input version is a beta or rc version to preserve previous behavior of GHA scripts. Update GHA scripts to use new option * applied suggestions from feedback in PR #2736 * bold some release notes * update version for development towards rc1 release * fix logic to determine offical X.Y.Z release * Feature #2739 labels (#2740) * Per #2739, add NRL reporting label and add METplus-Internal to the list of repositories to be updated. * Per #2739, add info about labels to the Contributor's Guide. * Per #2739, update the list of METplus repos in process_labels.sh but remove them from the other shell scripts to avoid keeping the same list updated in multiple spots. * Per #2739, move label details from the README to the Contributor's Guide * Updated release notes adding this issue: Develop an RST template for use cases * Feature 2716 existing builds (#2748) * adding the first 4 paragraphs * fixing web addresses * Highest level of dropdown menus added * NCAR machines dropdown menus * CASPER dropdown menu * web addresses can't be bolded. Fixing dropdown problems * blank line problem * italics date adding Derechio parts * DERECHO dropdown * fixing problems * fixing mor problems * fixing bolding * trying to fix web address * adding NCAR RAL MACHINES and trying to fix website issue * NCAR/RAL Internal Development dropdown * adding remaining dropdown structure * WCOSS2 dropdown * HERA dropdown * fixing last updated to be consistent * HERCULES dropdown * fixing pip * orion dropdown * JET dropdown * GAEA dropdown * fixing typo * Community machines, Docker Hub & AWS * fixing typo * Changing to "Users should create a file like" * to set a personalized input... * adding last updated and Docker Hub web links * Updating Jet information after recent install and changing some formatting for consistency * Removing bolding for consistency * creating a separate file for existing builds to add to 2 areas make it section 2.1 where METplus Components Release Note Links and Release Notes * adding existing_builds as an include * Rename existing_builds_rst to existing_builds.rst I had an underline instead of a period * changing to a section, not a chapter * trying a different section * changing existing builds to an include file * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing existing builds completely from the Release Guide --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update release-notes.rst * add custom string to time info dictionary for substituting values into the field info string * Feature #2742 develop fix StatAnalysis to set fcst_lev/obs_lev (#2760) * Feature 2763 existing builds (#2765) * Per #2763, updated build for Casper * Per #2763, updating Orion * Bugfix #2762 PCPCombine derive field info (#2764) * per #2762, change derive mode to add field info for each file instead of adding the first field info only * Per #2762, improve how the accumulation amounts are handling using relativedelta to ensure that the correct number of seconds are used for each search window when using inconsistent time intervals like months or years * Bugfix #2762 part 2 -- fix PCPCombine leap year issue (#2766) * remove check if script is master_metplus.py because it has been deprecated * per #2762, fix bug that caused failure depending on valid time relating to leap years * Feature #2754 rc1_main_vX.Y (#2755) * Per #2754, update instructions to create the main_vX.Y and main_vX.Y-ref branches (when applicable) for the first release candidate rather than waiting until the official release. Note that I deleted the push_release_branch.rst instructions since it's not needed. The preceeding instructions execute the push and these are redundant - even for METplus which has custom instructions. * Per #2754, add a note to make the detail stand out better * Per #2754, make the GitHub release instructions more explicit to match the names of the actual fields on GitHub. * Per #2754, not changing comment, just using more consisent formatting options. * Per #2754, change example from 'betaN' to 'rc1' since this step is only run for the rc1 release. * Per #2754, use consistent single quotes instead of italics for formatting. * Updates to METplus instructions to properly handle version numbers for rc1 release. Update METplus Official release instructions to use main branch and update version number for official release since main branch will contain -rcN version numbers * update instructions to checkout correct branch * clarify next deveopment version can be X+1.0 OR X.Y+1 * update version table for other components -- this could likely be refactored to make the version file path a variable and include the same update_version_on_develop.rst file for METplus, METcalcpy, METdataio, METplotpy, and METviewer --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Resolving use case template formatting problems and adding dropdowns * Resolving use case template formatting problem * Fixing formatting * Updating to add language if no Python embedding or User Scripting is used. * Feature #2562 version lookup documentation and automated tests (#2773) * set default input version to the latest official release, which is the 2nd highest coordinated release version number in the version lookup table. This assumes that the highest version is what is being developed towards * per #2562, update Release Guide to include instructions for updating the component versions lookup table in the METplus repository * Per #2562 and #2597, replace automated test logic that used manage_externals to use component version lookup to get versions of METplotpy/METcalcpy/METdataio * clean up SonarQube complaints * turn on some use cases to test changes * fix clone * do not get python dependencies for METplotpy/METcalcpy/METdataio installs * use full path to package to import * clone METplus Analysis repos on the same level as METplus to match previous location using manage_externals, remove manage_externals config files that are no longer used by the automated tests * clean up linter complaints * turn off use cases after confirming that they run successfully * clean up usage statement formatting * remove info about updating manage_externals config files used in automated tests that have been removed * added description of component versions script to Contributor's Guide and added stubs for other utility descriptions * change format of example results and add text to describe what is being demonstrated from each example * fix formatting * fixed bad variable replacement * Update docs/Contributors_Guide/utilities.rst --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature 2737 release instruct (#2775) * Per #2737 updating version instructions * Per #2737, update version instructions for METcalcpy * Per #2737, renaming development file * Per #2737, update version instructions for METplotpy * Per #2737, updating version instructions for METdataio * Per #2737, updating version instructions for METviewer * Per #2737, adding pyproject.toml to the list of files to update * Per #2751, this work was actually completed already, adding text to make the directions more clear. (#2776) * Made minor modifications for consistency * Feature 2339 dependencies (#2777) * Per #2339, update documention to include info about dependencies and coordinated release announcement * Per #2339, renaming file and header * Per #2339, correcting typo in formatting * Per #2339, updated formatting * Per #2339, updating additional dependencies * Per #2300, add Python package disclaimer for the METplus components (#2782) * Feature #2666 sonarqube (#2784) * Unrelated to #2666 but found while working on it, update the expected output file names for the TCDiag basic met_tool use case. * Per #2666, add details about SonarQube. Still need to proof-read and wordsmith * Per #2666, more SonarQube details * Per #2666, fix up anchor links, switch to *'s for bullets like we use everywhere else, and massage the wording * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update the instructions for official MET relases with a reminder to create a new version of the MET flowchart. * Feature #2772 Apptainer documentation (#2783) * update version of action to latest * per #2772, added instructions to run a METplus use case using Apptainer * Apply suggestions from code review Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * bugfix dtcenter/MET#2899 prevents differences in output from use case, so removing the 'disabled' item to run use case in GHA * hotfix: remove quotes around input files because logic that obtains python embedding commands as input file arguments handles quotes * fix tests to match new expected commands --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature #2560 Fire GridStat use case (#2769) * added config file and python scripts for fire weather use case * Improved logic if multiple files were found to use the closest file to the valid time. Assuming that the files are in order, it will use the first file if multiple files match the valid hour or the last file if multiple files match 1 hour prior to the valid hour. This does not factor in if files before and after the valid time were found. Modifications are needed to obtain the correct file if, for example, the valid time is 12Z and there are 1258 and 1159 files available -- in this case the 1258 file will be used even though 1159 is closer. * rename use case and update configuration to read data from appropriate directories * per #2560, add new use case to automated tests * turn of use case to prepare for PR * removed comments * added in-progress use case documentation - still needs updates * fix formatting * more formatting * more doc formatting and cleanup * remove notes from sections that were updated * doc cleanup * add thumbnail for new use case * add description of fire use case category * turn off use case * fix SonarQube complaints * update use case doc to match use case template * minor formatting change to match template * update path for kml schema file * use wildcard to avoid having to reference namespace that includes insecure protocol * Update documentation to resemble template --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature #2597 remove manage_externals and its config files (#2785) * Feature #2757 allow missing docs (#2786) * per #2757, add documentation for allow missing input logic * fix incorrect link to glossary item * fix typo * Per #2757, add info and examples for allow missing input logic * Feature 2741 add version (#2793) * adding contents and version * adding contents and version * pbl and precip directory updates * s2s directory updates * short_range directory updates * adding contents and version numbers to last 3 directories * Removing "UPDATE_SECTION_CONTENT" text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update component version script to NOT return develop for an rcN version unless an option is set to override that behavior (in case we need to change how rc is handled). Update automated scripts to get the correct MET Docker images for RC versions (get met-dev/main_vX.Y for RC, get met-dev/develop for betas or dev version, get met/X.Y-latest for official releases). Updated unit tests for component version script to ensure correct behavior occurs * Feature #2789 v6.0.0-rc1 release (#2792) * Per #2779, assigned a number to all use cases that are not run in GHA and added a reason to describe why the use cases have been disabled * adding location information * updating location information * Revert "adding location information" This reverts commit cc6d18506b12d30b931276163f6da52dfaaebb4f. * Revert "updating location information" This reverts commit 5920779fca3a1e5fd1b82538a9b206ef97f76320. * added location of input data that is not found with rest of input data for use case category * added information on how to run generate_release_notes.py script and updated script to clean up some formatting issues * Per #2789, added release notes for rc1 release * update version for rc1 release * fixed broken commands in use case scripts --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes…

* Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updated language for tar file Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * Bugfix #2705 develop - SeriesAnalysis fix time substitution into field info (#2709) * fix SonarQube complaints * Per #2705, set correct time information in field info by using the time info used to find the input files (self.c_dict['ALL_FILES']) instead of relying on parsing the time info from the other input files (fcst vs. obs) * clean up unused code after bugfix for #2705 * Feature 2647 use case mv mode for rrfs (#2706) * establish documentation, need to fill out with provided info * updated documentation, needs to have img path and output path fixed * updated output strings, still need to update pic link * removing pound sign to render correctly * Revert "removing pound sign to render correctly" This reverts commit ffe607a96fd33dc5ba15a479f76778dcba7e9656. * targeted removal of pound sign with bullet points section. * Per #2647, attempt to fix syntax error * Per #2647, adding back contents with formatting changes to test if it works * Per #2647, adding back contents in a different location to try to resolve syntax error * Per #2647, testing pipe for horizontal scroll bar * Per #2647, removed the pipe symbol and are attempting to fix syntax error again * Per #2647, reverting to original failed attempt of a table of contents * Per #2647, trying new formatting * Per #2647, fixing new formatting * Per #2647, attempt formatting changes * Per #2647, reverting back to original state * added to automated testing, updated docs, config file * added docs back in * added image, new quick search term, disabled auto tests * testing use_case_groups.json to see if/how this fails * Updated to disable use_case_groups.json run * added use case to list of exceeding disk space --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Bugfix 2675 move to pyproject.toml (#2699) * 2675: move to pyproject.toml * 2675: update toml, remove setup.py ref * bugfix 2675: reinstate VERSION file * bugfix 2675: hardcode version in pyproject.toml * use relative versions * slightly modify import to make it easier to move produtil into metplus * read version from version file * disable install of run_metplus script since it is broken * only install metplus and sub-packages, add version files to package data (parm cannot be added because it is non-package data) * add newline back * install produtil package so metplus can be imported in python * remove unused import * move produtil under metplus to differentiate it from the full NCEP produtil package and isolate it to metplus when installing metplus as a package -- previously produtil would be installed as its own package when metplus is installed * fixed more imports * Create links for parm directory and scripts/run_metplus.py under metplus so the parm files and run script will be installed as part of the metplus python package. Modified config_metplus to find the parm directory from under metplus to be consistent between running via python package and via scripts * Exclude parm from sonar sources because it will be found under metplus. Add metplus/parm/** to list of coverage exclusions * fixed more imports * add parm back to list of sonar sources --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Removing stated location of the version selector menu * Feature 2489 update docs (#2717) * adding new Literal Include section * under bold section adding "Environment" to Variables * Adding Substitution References section * removing blank space * fixing code typos * another typo * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2540 Point2Grid config file (#2580) * per #2540, add support for setting values in a config file for Point2Grid wrapper -- removed -qc flag argument in favor of setting obs_quality_inc in the config file * Added support for setting time_offset_warning now that the changes for other wrappers was merged into develop. Also added support for MET_CONFIG_OVERRIDES for Point2Grid * per #2540, set -goes_qc command line argument from POINT2GRID_GOES_QC_FLAGS and continue to support legacy POINT2GRID_QC_FLAGS. Other fixes to resolve incorrectly resolved conflicts from previous commit merging develop into branch * per #2540, remove quality_mark_thresh from config file because it has been removed from MET point2grid * Per discussion in PR #2580, remove time_offset_warning support from tools that don't actually use the value -- to add support for a new wrapper, update metplus/util/constants.py TIME_OFFSET_WARNING_WRAPPERS to include the wrapper's name * removed env vars for time_offset_warning for wrappers that no longer set it * install ImageMagick convert because it appears to no longer be available in the image that runs the unit tests (ubuntu 24.04?) * Bugfix metdbload config (#2728) * Update METdbLoadConfig.xml A schema is now used to validate all XML specification files and elements must adhere to a particular order. The <group> element follows the headers and load elements and preceeds the folder tmpl element. * Update use_case_groups.json temporarily run tests for the short range use cases * Update use_case_groups.json Return all the short_range use case test "run" values back to false * install convert because some tests fail if it is not available * Feature #1882 docs: new use case that uses METplus Analysis (#2726) * Fixing mistake * Fixing reference * Fixing link * Updating sections * clean up new content --------- Co-authored-by: Christina Kalb <kalb@ucar.edu> * Feature 918 add use case template (#2690) * updated template, 1st part * added more to template, updated internal references. Needs final two cats * finished updating template * updated syntax, length of lines * moved example to separate file, testing indents for discussion * added new collapse menu for MET configs, documentation on adding keywords, testing 2 separate options to stop paragraph jumping during bullet lists * updated spacing used for 2nd line indentation * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * Minor changes, updates hyperlink to template to be directly to the file and not to the directoy in GitHub. * Update add_use_case.rst * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Update add_use_case.rst * Missing space in hyperlink. * Changes to use a keyword to assist users in checking off each section they should review and also PR reviewers in ensuring each section has been visited and reviewed by the contributor. * updated Python section to User Scripting --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2727 point_weight_flag and obtype_as_group_val_flag (#2730) * Per #2727, add support for setting point_weight_flag * Per #2727, add support for obtype_as_group_val_flag * per #2714, deprecate master_metplus.py * Update VERSION * Feature #2710 - clean up use cases (#2711) * per #2710, remove line that has no effect because next line overwrites value * per #2710, fix bug in if/else block where both were doing the same thing. Removed commented code and create variable for constant to satisfy SonarQube * updated config file commands * updated level of winds to capture change in script --------- Co-authored-by: j-opatz <jopatz@ucar.edu> * Feature #2682 Component version lookup (#2733) * per #2714, deprecate master_metplus.py * exclude metplus/scripts from SonarQube scan to prevent incorrect duplicate code error since script lives in ush * added Dockerfile to build image with MET/METplus and METplus Analysis Python packages * start of script to get versions of METplus components * per #2682, created METplus component version lookup table and functions to get formatted version of requested components * add descriptions for tests * add update information and made script callable * made script executable * add workflow to trigger on release and create dtcenter/metplus and dtcenter/metplus-analysis Docker images * add scripts used by new workflow * removed build hook scripts since we will be building images via GHA * change logic to get MET version using component version script instead of build hook file that just adds 6 to major version * add missing shebang * make script callable directly and make output component a required input * pass LATEST_TAG to push script * ensure only 1st line of version file is read to avoid newline * turn on use case to test * turn off use case * Feature #2735 v6.0.0 beta6 (#2736) * change version for beta6 release * added script to help generate release notes for development releases * add default title for issues and added documentation issue template * sort issues by number, improve formatting for release notes title, add logging to alert users what is happening when timely github queries are running * added release notes for beta6 release * Add option to component version script to return 'develop' if the input version is a beta or rc version to preserve previous behavior of GHA scripts. Update GHA scripts to use new option * applied suggestions from feedback in PR #2736 * bold some release notes * update version for development towards rc1 release * fix logic to determine offical X.Y.Z release * Feature #2739 labels (#2740) * Per #2739, add NRL reporting label and add METplus-Internal to the list of repositories to be updated. * Per #2739, add info about labels to the Contributor's Guide. * Per #2739, update the list of METplus repos in process_labels.sh but remove them from the other shell scripts to avoid keeping the same list updated in multiple spots. * Per #2739, move label details from the README to the Contributor's Guide * Updated release notes adding this issue: Develop an RST template for use cases * Feature 2716 existing builds (#2748) * adding the first 4 paragraphs * fixing web addresses * Highest level of dropdown menus added * NCAR machines dropdown menus * CASPER dropdown menu * web addresses can't be bolded. Fixing dropdown problems * blank line problem * italics date adding Derechio parts * DERECHO dropdown * fixing problems * fixing mor problems * fixing bolding * trying to fix web address * adding NCAR RAL MACHINES and trying to fix website issue * NCAR/RAL Internal Development dropdown * adding remaining dropdown structure * WCOSS2 dropdown * HERA dropdown * fixing last updated to be consistent * HERCULES dropdown * fixing pip * orion dropdown * JET dropdown * GAEA dropdown * fixing typo * Community machines, Docker Hub & AWS * fixing typo * Changing to "Users should create a file like" * to set a personalized input... * adding last updated and Docker Hub web links * Updating Jet information after recent install and changing some formatting for consistency * Removing bolding for consistency * creating a separate file for existing builds to add to 2 areas make it section 2.1 where METplus Components Release Note Links and Release Notes * adding existing_builds as an include * Rename existing_builds_rst to existing_builds.rst I had an underline instead of a period * changing to a section, not a chapter * trying a different section * changing existing builds to an include file * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing existing builds completely from the Release Guide --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update release-notes.rst * add custom string to time info dictionary for substituting values into the field info string * Feature #2742 develop fix StatAnalysis to set fcst_lev/obs_lev (#2760) * Feature 2763 existing builds (#2765) * Per #2763, updated build for Casper * Per #2763, updating Orion * Bugfix #2762 PCPCombine derive field info (#2764) * per #2762, change derive mode to add field info for each file instead of adding the first field info only * Per #2762, improve how the accumulation amounts are handling using relativedelta to ensure that the correct number of seconds are used for each search window when using inconsistent time intervals like months or years * Bugfix #2762 part 2 -- fix PCPCombine leap year issue (#2766) * remove check if script is master_metplus.py because it has been deprecated * per #2762, fix bug that caused failure depending on valid time relating to leap years * Feature #2754 rc1_main_vX.Y (#2755) * Per #2754, update instructions to create the main_vX.Y and main_vX.Y-ref branches (when applicable) for the first release candidate rather than waiting until the official release. Note that I deleted the push_release_branch.rst instructions since it's not needed. The preceeding instructions execute the push and these are redundant - even for METplus which has custom instructions. * Per #2754, add a note to make the detail stand out better * Per #2754, make the GitHub release instructions more explicit to match the names of the actual fields on GitHub. * Per #2754, not changing comment, just using more consisent formatting options. * Per #2754, change example from 'betaN' to 'rc1' since this step is only run for the rc1 release. * Per #2754, use consistent single quotes instead of italics for formatting. * Updates to METplus instructions to properly handle version numbers for rc1 release. Update METplus Official release instructions to use main branch and update version number for official release since main branch will contain -rcN version numbers * update instructions to checkout correct branch * clarify next deveopment version can be X+1.0 OR X.Y+1 * update version table for other components -- this could likely be refactored to make the version file path a variable and include the same update_version_on_develop.rst file for METplus, METcalcpy, METdataio, METplotpy, and METviewer --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Resolving use case template formatting problems and adding dropdowns * Resolving use case template formatting problem * Fixing formatting * Updating to add language if no Python embedding or User Scripting is used. * Feature #2562 version lookup documentation and automated tests (#2773) * set default input version to the latest official release, which is the 2nd highest coordinated release version number in the version lookup table. This assumes that the highest version is what is being developed towards * per #2562, update Release Guide to include instructions for updating the component versions lookup table in the METplus repository * Per #2562 and #2597, replace automated test logic that used manage_externals to use component version lookup to get versions of METplotpy/METcalcpy/METdataio * clean up SonarQube complaints * turn on some use cases to test changes * fix clone * do not get python dependencies for METplotpy/METcalcpy/METdataio installs * use full path to package to import * clone METplus Analysis repos on the same level as METplus to match previous location using manage_externals, remove manage_externals config files that are no longer used by the automated tests * clean up linter complaints * turn off use cases after confirming that they run successfully * clean up usage statement formatting * remove info about updating manage_externals config files used in automated tests that have been removed * added description of component versions script to Contributor's Guide and added stubs for other utility descriptions * change format of example results and add text to describe what is being demonstrated from each example * fix formatting * fixed bad variable replacement * Update docs/Contributors_Guide/utilities.rst --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature 2737 release instruct (#2775) * Per #2737 updating version instructions * Per #2737, update version instructions for METcalcpy * Per #2737, renaming development file * Per #2737, update version instructions for METplotpy * Per #2737, updating version instructions for METdataio * Per #2737, updating version instructions for METviewer * Per #2737, adding pyproject.toml to the list of files to update * Per #2751, this work was actually completed already, adding text to make the directions more clear. (#2776) * Made minor modifications for consistency * Feature 2339 dependencies (#2777) * Per #2339, update documention to include info about dependencies and coordinated release announcement * Per #2339, renaming file and header * Per #2339, correcting typo in formatting * Per #2339, updated formatting * Per #2339, updating additional dependencies * Per #2300, add Python package disclaimer for the METplus components (#2782) * Feature #2666 sonarqube (#2784) * Unrelated to #2666 but found while working on it, update the expected output file names for the TCDiag basic met_tool use case. * Per #2666, add details about SonarQube. Still need to proof-read and wordsmith * Per #2666, more SonarQube details * Per #2666, fix up anchor links, switch to *'s for bullets like we use everywhere else, and massage the wording * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update the instructions for official MET relases with a reminder to create a new version of the MET flowchart. * Feature #2772 Apptainer documentation (#2783) * update version of action to latest * per #2772, added instructions to run a METplus use case using Apptainer * Apply suggestions from code review Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * bugfix dtcenter/MET#2899 prevents differences in output from use case, so removing the 'disabled' item to run use case in GHA * hotfix: remove quotes around input files because logic that obtains python embedding commands as input file arguments handles quotes * fix tests to match new expected commands --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature #2560 Fire GridStat use case (#2769) * added config file and python scripts for fire weather use case * Improved logic if multiple files were found to use the closest file to the valid time. Assuming that the files are in order, it will use the first file if multiple files match the valid hour or the last file if multiple files match 1 hour prior to the valid hour. This does not factor in if files before and after the valid time were found. Modifications are needed to obtain the correct file if, for example, the valid time is 12Z and there are 1258 and 1159 files available -- in this case the 1258 file will be used even though 1159 is closer. * rename use case and update configuration to read data from appropriate directories * per #2560, add new use case to automated tests * turn of use case to prepare for PR * removed comments * added in-progress use case documentation - still needs updates * fix formatting * more formatting * more doc formatting and cleanup * remove notes from sections that were updated * doc cleanup * add thumbnail for new use case * add description of fire use case category * turn off use case * fix SonarQube complaints * update use case doc to match use case template * minor formatting change to match template * update path for kml schema file * use wildcard to avoid having to reference namespace that includes insecure protocol * Update documentation to resemble template --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature #2597 remove manage_externals and its config files (#2785) * Feature #2757 allow missing docs (#2786) * per #2757, add documentation for allow missing input logic * fix incorrect link to glossary item * fix typo * Per #2757, add info and examples for allow missing input logic * Feature 2741 add version (#2793) * adding contents and version * adding contents and version * pbl and precip directory updates * s2s directory updates * short_range directory updates * adding contents and version numbers to last 3 directories * Removing "UPDATE_SECTION_CONTENT" text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update component version script to NOT return develop for an rcN version unless an option is set to override that behavior (in case we need to change how rc is handled). Update automated scripts to get the correct MET Docker images for RC versions (get met-dev/main_vX.Y for RC, get met-dev/develop for betas or dev version, get met/X.Y-latest for official releases). Updated unit tests for component version script to ensure correct behavior occurs * Feature #2789 v6.0.0-rc1 release (#2792) * Per #2779, assigned a number to all use cases that are not run in GHA and added a reason to describe why the use cases have been disabled * adding location information * updating location information * Revert "adding location information" This reverts commit cc6d18506b12d30b931276163f6da52dfaaebb4f. * Revert "updating location information" This reverts commit 5920779fca3a1e5fd1b82538a9b206ef97f76320. * added location of input data that is not found with rest of input data for use case category * added information on how to run generate_release_notes.py script and updated script to clean up some formatting issues * Per #2789, added release notes for rc1 release * update version for rc1 release * fixed broken commands in use case scripts --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and r…

* Update the instructions for official MET relases with a reminder to create a new version of the MET flowchart. * Feature #2772 Apptainer documentation (#2783) * update version of action to latest * per #2772, added instructions to run a METplus use case using Apptainer * Apply suggestions from code review Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * bugfix dtcenter/MET#2899 prevents differences in output from use case, so removing the 'disabled' item to run use case in GHA * hotfix: remove quotes around input files because logic that obtains python embedding commands as input file arguments handles quotes * fix tests to match new expected commands --------- Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Feature #2560 Fire GridStat use case (#2769) * added config file and python scripts for fire weather use case * Improved logic if multiple files were found to use the closest file to the valid time. Assuming that the files are in order, it will use the first file if multiple files match the valid hour or the last file if multiple files match 1 hour prior to the valid hour. This does not factor in if files before and after the valid time were found. Modifications are needed to obtain the correct file if, for example, the valid time is 12Z and there are 1258 and 1159 files available -- in this case the 1258 file will be used even though 1159 is closer. * rename use case and update configuration to read data from appropriate directories * per #2560, add new use case to automated tests * turn of use case to prepare for PR * removed comments * added in-progress use case documentation - still needs updates * fix formatting * more formatting * more doc formatting and cleanup * remove notes from sections that were updated * doc cleanup * add thumbnail for new use case * add description of fire use case category * turn off use case * fix SonarQube complaints * update use case doc to match use case template * minor formatting change to match template * update path for kml schema file * use wildcard to avoid having to reference namespace that includes insecure protocol * Update documentation to resemble template --------- Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * Feature #2597 remove manage_externals and its config files (#2785) * Feature #2757 allow missing docs (#2786) * per #2757, add documentation for allow missing input logic * fix incorrect link to glossary item * fix typo * Per #2757, add info and examples for allow missing input logic * Feature 2741 add version (#2793) * adding contents and version * adding contents and version * pbl and precip directory updates * s2s directory updates * short_range directory updates * adding contents and version numbers to last 3 directories * Removing "UPDATE_SECTION_CONTENT" text --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update component version script to NOT return develop for an rcN version unless an option is set to override that behavior (in case we need to change how rc is handled). Update automated scripts to get the correct MET Docker images for RC versions (get met-dev/main_vX.Y for RC, get met-dev/develop for betas or dev version, get met/X.Y-latest for official releases). Updated unit tests for component version script to ensure correct behavior occurs * Feature #2789 v6.0.0-rc1 release (#2792) * Per #2779, assigned a number to all use cases that are not run in GHA and added a reason to describe why the use cases have been disabled * adding location information * updating location information * Revert "adding location information" This reverts commit cc6d18506b12d30b931276163f6da52dfaaebb4f. * Revert "updating location information" This reverts commit 5920779fca3a1e5fd1b82538a9b206ef97f76320. * added location of input data that is not found with rest of input data for use case category * added information on how to run generate_release_notes.py script and updated script to clean up some formatting issues * Per #2789, added release notes for rc1 release * update version for rc1 release * fixed broken commands in use case scripts --------- Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> * update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use ca…

* update version for development toward next release first beta * Updating RC1 release instructions for MET to create the vX.Y input test data directory prior to creating the main_vX.Y-ref branch. * prevent error when input version requested does not exist in lookup table * handle if input version is develop * Feature 2741 update use cases (#2795) * updating template first pass * removing backslash * removing extra # symbol * removing spaces. Keep as is * removing spaces for consistency * adding changes to match template * fixing spacing * fixing spacing take 2 * Per #2741, correcting typo in OUPUT_BASE in template file * Per #2741, correcting typo in OUPUT_BASE * Per #2741, updating sections based on template and template description * Updated language in several sections, needs additional work on output and config file * updating for consistency * Updating file based on suggestions from George * Updating with suggestions from George * Updating file based on suggestions from George * reviewing with Julie * Per #2741 attempting to resolve errors * Per #2741, updating template to fix rendering problems * Per #2741, fixing formatting * Per #2741 attempting to resolve warnings * Per #2741, resolving error * Resolving formatting problems with template * Fixing formatting problems * updating for consistency * fixing formatting * removing extra spaces * Updating formatting * updating for consistency * adding METplus Workflow times * modifications * Update GridStat_fcstCESM_obsGFS_ConusTemp.py Added some information to the use case * Update MODE_fcstCESM_obsGPCP_AsianMonsoonPrecip.py Updates * Added an extra blank line * updating for consistency * fixing spacing * adding space * removing Update content * updating for consistency * adding a period * updating for consistency * updating for consistency * updating for consistency * updating for consistency * adding return * replacing underlines * replacing return? * fixing blank line * more attempts to fix warnings * updating for consistency * adding double colons and stuff * trying stuff * trying extra spacing * Changing Datasets Location info to match template * Per #2741, updating formatting and wording * Per #2741, updating after reviewing * Per #2741, attempting to resolve errors * updating for consistency * fixing brackets and section breaks * adding return * Per #2741, updating after reviewing * updating for consistency * fixing spacing * adding a pound * Updating wording for User Scripting * Updating the text for User Scripting * Updating User Script section * Updating text for User Scripting * Updating text * adding location information * adding location information * updated first cloud file, will use for remaining cloud use cases * updated language in numerous use case doc files * updated final marine use case --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix typo * update timeline to include actual date of rc1 release * added missing script to create mp_analysis conda environment for use case tests * add mp_analysis conda env to workflow to create conda environments for DockerHub * Feature 2741 running metplus (#2801) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * Updating Datasets section * Added needed return character before "#Pass" * Added /path/to/user_system.conf to end of run_metplus.py line * Added a comma after "command line" * Updating file path * Added fields for empty Satasets section --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Updated HERA information * Resolving error * Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extens…

* Updated jet and hera * prevent error in the command to pull the changelog file when it does not exist yet * Feature 2751 develop release updates (#2798) * Adding period to end of sentence. * Release Guide (develop) - updating existing builds (#2806) * update existing builds for upcoming coordinated 6.1 (beta1) release * added template for existing builds for next dev cycle and added it to ignore list * Update release guide instructions to set the new location of the Existing Builds page on the Downloads page. Update instructions for creating a new existing builds page to instead copy the template and replace values in the develop branch * added a step to update the development timeline info for an official release * Updating existing_builds files * changed Docker Hub to DockerHub * added step to update the DockerHub info on the Existing Builds page * change version to official release because RAL-IT doesn't install development releases on RAL machines --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * prevent error when merge commit messages are too long * prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) …

* prevent error when change summary text contains backticks * Update METplus-6.0.0 release date from 12/11/24 to 12/18/24. * Feature 2741 dropdowns (#2808) * marine_and_cryosphere directories * fixing underlining * trying to fix warning msg. * trying to fix warning msg. again * Removing space before i.e. * updating medium_range and pbl directories * deleting empty file * removing lots of old text * precipitation files * removing junk file * removing extra # * add return * updating 3 dirs s2s, space_weather and unstructured_grids * updating short_range files * adding tc_and_extra_tc files * removing defunct -c option * added dropdown menus for marine_and_cryosphere * fixing problems and dropdowns for medium_range * fixing new problems * dropdowns for pbl and precip * adding double colons * Attempting to resolve errors * dropdowns for s2s and s2s_mjo * adding dropdowns for short_range files * final dropdown directories * Update docs/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py removing note Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Apply suggestions from code review adding all of the suggestions Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Update GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.py ensuring literal include is in dropdown menu * Update GridStat_fcstRTOFS_obsOSTIA_iceCover.py ensuring literal include is in dropdown * ensuring literal include is in dropdown menu * marine_and_cryosphere MET Configuration dropdown fixes * remaining dropdown directories for MET Configuration * Julie changing dropdowns Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * HEAD needed to be removed * There are not MET tools used in this use case * There are no MET tools used in this use case * Apply suggestions from code review Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Per #2741, few small changes * updates from pull request discussion * removing space Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * fixing dropdown Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * removing bash Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update from Julie Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Fixing error * lots of typos Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Adding period * Adding colon * Added period * Added period * Added period * typos and more Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #511 lists in command line single config overrides (#2815) * exclude metplus/scripts dir from PyCharm project to prevent incorrect duplicate code warnings with files that are sym linked * to expand on #2772, updating instructions to include note to set tmp directory for apptainer to prevent issues pulling large images that require a lot of temp space * per #511, add unit test for expected behavior to support comma-separated lists in a command line single config override that should fail until fix is made * add a test to ensure that the -c argument is properly ignored since it is can be used in old use cases that were created when the argument was required * per #511, add support for command line single config overrides to include values that are lists. Simplify logic to parse arguments to strip out -c/--config/-config arguments and skip check/error if argument is invalid because it is already handled in the metplus_config setup step that parses the arguments * added unit test to ensure that an invalid command line argument causes the appropriate failure from run_metplus.py * Feature metplus 2780 dedication (#2819) * adding In Memoriam section * Changing to "This Coordinated Release is dedicated to" removing version * adding links * fixing Randy's link * Per #2780, change case of words and add comma after year --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * Feature #2816 enhance workflow dispatch testing (#2818) * per #2816, update title of workflow displayed in Actions tab to display the repo that triggered the workflow on workflow_dispatch events. Preserve the behavior of pull_request and push events by displaying the PR title or head commit message respectively * fix logic for run-name * Update event info to note that the workflow was triggered manually if the repo/title was not provided. Update description of workflow_dispatch argument to describe what it can be used for when triggering manually. Only run step to build URL to commit that triggered workflow if it was triggered from an external repo * add optional argument for workflow_dispatch to specify the title of the workflow run * revert change to repository argument description since title argument was added to handle custom workflow run titles * Feature #2814 develop - doc single config overrides (#2825) * per #2814, add documentation to describe how to set a single config variable on the command line * fixed typos and formatting issues * move new content to top of chapter and reword intro sentence * change label and add link to user env var section * per #2814, added info and examples for single config overrides with a list of values * Sorts the list of FLUXNET csv files returned from glob. (#2836) * Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) * update instructions to create a new project board to use templates (#3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * develop docs v6.2.0 release timeline (#3083) * Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file. * Feature #3068 DataIngest wrapper (#3084) * clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to sk…

* Bugfix #2830 develop fix missing log output (#2841) * Per #2830, skip closing of log handlers for METplusConfig objects that are created for copying values for process list instances so they are not closed before the end of the run. * remove some output directories after tests are run * Updated METexpress versions in component_versions.py (#2842) * added script to regroup release notes for official release by parsing dev release notes. improve naming of drop downs for generating dev release notes to be consistent * improve script to handle different formatting for categories with no issues * per suggestion from @bikegeek, output 'None' if there were no items under a category * reorder instructions to follow more natural progression * update instructions for updating ReadTheDocs based on the changes to the RTD web interface * add next coordinated release to version lookup table * move instructions to create directory for data for new dev cycle on DTC web server to be done for the rc1 release instead of after the official release * update versions of packages that have vulnerabilities * Feature #2586 GenVxMask improvements (#2833) * resolve some SonarQube complaints * per #2586, added function with tests to properly parse list of command line arguments that can now contain comma-separated lists that should not be split up into separate items * add support for {app}_{data_type}_FILE_WINDOW_BEGIN/END, e.g. GEN_VX_MASK_OBS_FILE_WINDOW_BEGIN. This just adds support for an additional variation of the config variable names * add support for an empty label for input templates * update wrapper to be consistent with other wrappers wrt finding input files, progress towards #2492. Allow file window range to be specified separately for mask and input files. Other cleanup to move towards consistent wrappers with fewer wrapper-specific overrides of functions like get_command * update unit tests to align with changes for #2492 * add documentation for config variables that are newly supported to allow file window range to be specified separately for mask and input files * renamed GEN_VX_MASK_OBS variables to be GEN_VX_MASK_INPUT as suggested by @JohnHalleyGotway in PR review * fix logic to properly read input files by handling inputs that support multiple inputs with labels (used by GridDiag and UserScript wrappers) and typical inputs (all other wrappers). Prior to this change only input templates that have the FCST or OBS identifier were read properly via get_input_templates * Feature 2628 documentation updates (#2817) * Feature #2844 release_guide_dropdowns (#2847) * Work in progress adding instructions dropdowns to the release guide * Per #2844, more dropdowns * Per #2844, add instruction dropdowns for all MET instructions * Per #2844, more dropdowns. * Per #2844, use instruction dropdowns throughout * Per #2844, remove METexpress AWS instruction since its no longer relevant. * Remove accidentally committed .DS_Store file. * Update the MET Release Guide instructions about the dependent library tar files. * Minor formatting change * update URL for posting sample data * Doc-only change to rename the 'Code support' section as 'User support' based on 1/9/2025 decision at the METplus-Analysis Tools project meeting * fixed typo * Add 2 new 'alert' labels to flag changes that modify config options and/or output formats. * Update the common labels by adding the 'pull request:' prefix, one new reporting label and update scripts to run /bin/bash instead of /bin/sh, which now points to dash on seneca. * Feature #2827 SeriesAnalysis gradient (#2835) * Add 'requestor: NOAA/NWS' label for use in METplus Discussions. * Feature #2758 SonarQube (1) (#2866) * test that this file is not needed anymore because RTD controls the version selector * various changes to appease SonarQube complaints * remove deprecated docker command * resolve more SonarQube code smells * update URL for posting sample data * Add note to the Release and Contributor's Guide pointing readers to the develop branch content. * Updating common_labels.txt in the METplus develop branch * Update the details of the 'pull request' labels. * Feature #2781 PairStat wrapper (#2872) * update release guide for METplus to create data location for next release on mohawk when creating rc1 release instead of after official release * cleanup * per #2781, start first implementation of PairStat wrapper * appease SonarQube by increasing code coverage for component versions script * add new wrapper to lookup dictionary * per #2781 add wrapped MET config file * use runtime freq logic to find input files * start unit tests (broken) * reorder assert to align expected and actual values with PyCharm notation * update unit tests * rename field info variable 'field' to 'pairs' to match Pair-Stat naming convention. Add command line arguments to pair_stat command * reorder assert to align actual and expected in PyCharm * refactor function to get start/end times to prevent error when using INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code * change many wrappers to be consistent with finding input files. Errors are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this. * remove line that is not needed * fix bug introduced with recent changes and run use cases that failed to test that the fix is correct * fix multi-variate MODE run to pass all fields to the call instead of just the first one * update SeriesAnalysis test to use logic that is actually used when running * turn off use cases that now succeed and turn on use case to test fix to multi-variate mode * turn off use case after confirming that it now runs successfully * refactor logic to satisfy SonarQube to reduce cognitive complexity and remove unused imports * remove functions that are not used * handle -pairs flag like other command line args for input files. Add documentation to new functions * updates to new wrapper, add documentation, basic use case stubs. create function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers * remove unused import * remove config variable that is no longer used in MET * remove match_month support from climo_mean/stdev * remove obs_valid_beg/end from command line args because it is not supported by pair_stat * remove test config files that are not being used * configure basic use case to run command that is being used to test/develop the app * error if invalid tool name is provided * add missing tests * remove support for config variables that did not make it into the final implementation of the MET pair_stat tool * fix formatting on a few config values * correct input file info * turn on use case to test * fix command line argument for output directory that changed since the last test * refactor and clean up to reduce SonarQube issues * update use case to prevent redundant runs of SeriesAnalysis(run_two) by only setting custom loop list for the first instance of SeriesAnalysis * Fix bug that prevents correct field information from being set when multiple fields are read from the same file. Removed unused function * per feedback in PR #2872, remove support for setting output_prefix * add PairStat to list of wrappers that use the MODEL config variable * per feedback in PR #2872, modify wrapper behavior to change output path from a directory to a filename base that will be used to write output files with various extensions, e.g. _mpr.txt or .stat * remove output_prefix from tables * regroup met_tool_wrapper use cases * Feature #2877 v6.1.0-beta1 release (#2878) * update version for beta1 release * add release notes for beta1 * update script to help generate release notes to add a set of issues to be marked as pass because no extenral testing required, removed SonarQube complaints about duplicate string literals * added missing issue * rename 'Internal' category to 'Build, repository, and test' to be consistent with MET * update version for development towards beta2 * fix list * Feature #2880 SeriesAnalysis allow time templates in field name (#2881) * per #2880, allow filename template tags to be substituted in the fcst/obs name -- previously it was only supported in the level value * remove support for running SeriesAnalysis wrapper with RUNTIME_FREQ=RUN_ONCE_FOR_EACH because this will not produce any useful output. The default setting of RUN_ONCE_PER_INIT_OR_VALID will be used instead * Update version v12.0.1 MET (#2884) * add support for overriding METplotpy, METcalcpy, and METdataio branch for testing workflow * Update version 12.0.2 MET (#2896) * use develop branch version of the component versions script to determine version numbers for creating docker images for releases * updated configuration (#2901) * fixed typo * Feature 2891 develop existing builds modulefiles (#2909) * Per #2891, updated for 6.1.0 release with coming soon messages added * Per #2891, updating all modulefiles for 6.1.0 * Per #2891, fixing formatting * Feature 2741 datsets and workflow (#2839) * new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * add run_metplus.py to bin directory upon install * Revert "add run_metplus.py to bin directory upon install" This reverts commit d762570a803e4c8ed87c4ead5630a2a6c4967638. * Feature #2697 Python 3.12.0 - using Python 3.10 still (#2918) * Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see https://github.com/SciTools/cartopy/pull/2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube * Updating METplus-6.1.0 Development Timeline * Feature #2923 SonarQube for 6.1.0-beta2 (#2924) * fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method * prevent future warning * Feature 2744 nrl gfs goes (#2926) * Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review. * Feature #2697 python 3.12 round 2 (#2930) * fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0) * Feature 2745 nrl gfs asos (#2922) * Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> * fix use case script to work with cfgrib 0.9.11.0 and 0.9.15.0 using changes from @DanielAdriaansen * update instructions to download cartopy static files * Feature #2863 initial conda recipe (#2900) * initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description * Feature 2743 nrl gfs cloud analysis (#2850) * Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updated examples to reflect new paths on seneca, cleaned up instructions to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions * Feature 2771 subprojects (#2942) * Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest * Feature 2937 update data physics tendency (#2938) * Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature #2923 SonarQube 6.1.0-beta2 round 2 (#2946) * sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup * Per #2944, add support for fcst/obs_valid/init_beg/end/inc/exc/hour and remove obs_window for PairStat wrapper (#2945) * update apt to prevent imagemagick install failure and don't cancel other python version unit test if one fails * fix imagemagick install by updating apt packages * Feature #2941 v6.1.0-beta2 (#2950) * add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * update version for dev towards rc1 * Add requestor and reporting labels for NOAA/EPIC, as requested by @michelleharrold. * Feature 2953 installation files (#2958) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Feature #2560 Update Fire use case (#2956) * add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * added suggestions from conda-forge code review * force using python 3.10 locally for now * fix skip of any python except 3.10 and change env var name to prevent conflict with reserved env var used by METplus to set parm base * remove dependencies that don't need to be listed because they are made available by other dependencies * add MET license file * remove GS Fonts due to licensing issues and it is optional * prevent warning for numpy because it is actually used * add back libopenblas for linux only * Feature 2746 sfs use cases (#2954) * Adding Soil Moisture averaging case * Adding a new use case to compare soil moisture of SFS-GSL vs ERA5. The use case uses a python embedding file to read the SFS-GSL files. * Adding documentation to the soil moisture use case to compare SFS-GSL against ERA5 * Adding a thumbnail for the use case * Adding updated soil moisture use case * Updating title line * Adding Use Case image * Updating docs * Trying to fix use case image * Updating documentation * Updating documentation * Fixing documenation * Updating documentation * Updating directory * Adding use case data * Adding the s2s_soil_moisture directory * Updating documentation location * Adding Documentation README * Adding config override for actions * Fixing typo * Testing Config overrides * Testing new timeframe * Changing CI overrides * Renamed files and moved them to s2s_soil_moisture directory * Updated file to change ERA to ERA5Land * Moved file to the correct directory and updated * Changed file name * Fixed directory name * Fixed directory name * Fixed directory name * Added test for the soil moisture use case * Adding another use case * Setting tests to false for pull requests * Fixed image file name * Testing sonar cube change * Turning a test back on to check a change * Update sfs_gsl_model_wrapper.py * Testing a sonar cube update * Updated documentation as per new template * Updated documentation as per new template * Gallery path update * Update use_case_groups.json * Turning tests off for PR * Removed boilerplate language * Changing log level --------- Co-authored-by: Mrinal Biswas <biswas@ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> * add example of using the allow_empty extra arg keyword * change minimum supported version of python to 3.10.4 because the wrappers use logic that requires that version and we are unable to run the unit tests for earlier versions in GitHub Actions * fix tests that check python versions * update function that checks if a value is a number to properly handle numpy masked values (MaskedConstant) * fix incorrect label * Feature 2904 tif fuse case (#2987) * first pass over doc, basic files. Needs config update and finalize doc * add rioxarray conda environment * update out-of-date info * added image, updated docs, attempt to resolve compile issue * removing space for error * updated remaining files, adjusted docs * fix incorrect naming for another use case that had errors in use case doc rendering * fix typos and formatting * fix path to thumbnail image * add py_embed keyword to tell automated test suite to set MET_PYTHON_EXE to the version of python in the rioxarray conda env * fix SonarQube complaint "Fix this attribute access on a value that can be 'None'." despite the function cannot return None because it will exit the script if an invalid input is provided --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Hotfix to Release Guide instructions for MET to fix a minor formatting issue. * change version of sphinx-rtd-theme to a version that is available on conda-forge * Feature #2962 RMWAnalysis wrapper (#2985) * per #2962, begin implementation of RMWAnalysis wrapper * start basic use case conf * read BOTH_VAR<n> variables into data field * change pycharm python interpreter * ignore file generated by building docs locally * update use case to read input data * Per #2962, add documentation, unit tests, and basic use case for RMWAnalysis wrapper * per #2962, added thumbnail for use case documentation * fix incorrect formatting * per feedback from @JohnHalleyGotway, change 5 config variables to support lists of values instead of a single string * Bugfix #2890 develop - prevent incorrect error for skipped time (#2988) * per #2890, refactor Example wrapper to support all EXAMPLE_RUNTIME_FREQ settings to expand its use and reproduce the bug to be addressed * add docs for setting runtime freq for Example wrapper * prevent crash checking an init or valid time that is a wildcard * clean up function docs * Per #2890, add more checks for skipping times so wrappers don't incorrectly report an error if an init/valid time is skipped. Create function to handle skip checks, logging run time, and adding instance/custom to templates to reduce duplicate code. Change skip logging from DEBUG to INFO so it is more clear what is being skipped. Rename run_at_time function to be more clear of its purpose * update metplus_dev environment to include the documentation requirements * Per #2890, added unit tests for example wrapper to increase code coverage on new code. Improved error reporting if incorrect time configs are set * Feature #2984 v6.1.0-rc1 (#2991) * add release notes for rc1 * update DockerHub rc1 releases * fix indentation * Per #2984, updating title of issue 2863 --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * change deprecated maintainer to label * prevent GHA 'System.IO.IOException: No space left on device' error by removing data from original location as well * remove docker image after pushing to DockerHub to free up disk space * add new workflow to only run and update the truth data for use case groups that had differences in the last non-ref branch run. runs all use case groups if there is no previous non-ref run to handle first main_vX.Y-ref run before main_vX.Y has been created. also supports workflow dispatch to force update of all truth data as needed * remove problematic run-name logic because PR merge that will trigger the run has the information we need * remove logic to update truth data from testing workflow since it is now handled in update_truth_data.yml * rename job so workflow looks more similar to testing * rename workflow so it is clear that it updates the *-ref branch instead of actually updating the truth data on DockerHub * add release note for missing bugfix issue * push fix when no latest run for main_v6.1 branch exists * do not run anything in testing workflow for -ref branches * update version for development towards 6.2.0-rc1 * Feature 2932 use case amdar pbl (#2989) * Enhanced AMDAR PBLH METplus use case * Fixes title underline length error. * Adds missing .csv for airport filename in the conf file. * Reverts addition of .csv suffix since the Python embedding script adds it, and adjusts Python embedding script locations to use PARM_BASE instead of INPUT_BASE. * Changes the NumPy where statement to use the string nan instead of the NumPy nan constant to match the data type of the tailNumber column at this point. The string nan is used also on the subsequent line for valid_tails. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2998 develop - updating truth data instructions (#3001) * per #2998, update instructions for updating truth data to note that the latest develop branch run needs to finish running. also cleaned up some out-of-date info * add note to delete the temporary branch generated for updating develop-ref * enhance pytest fixture so it will work with wrappers that don't have class variable WRAPPER_ENV_VAR_KEYS * clean up tests * Feature 2953 installation files (#3004) * Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore * Per #2953, updating existing builds page. * Bugfix #2980 develop - PCPCombine 3 year accum bug (#3005) * per #2980, add test to recreate bug when building a 3 year accumulation using 1 year accums * Per #2980, fix bug in determining accumulation seconds used to find last valid time to check for an accumulation -- The search time minus the output accumulation is used as the reference time to compute the number of seconds of the input accumulation. The input accumulation is subtracted from this time to determine the number of seconds, which depending on the time can cross over a leap year when the actual accumulation calculations do not. This fix adds the input accumulation to the 'first' time to ensure that it is correctly calculated * Feature #3008 Support lead months/years in template substitution (#3009) * clean up code formatting * Per #3008, add support for months and years in {lead} filename template tags. Added tests to ensure correct behavior occurs. Rename format_hms function to be more clear of what it actually does now * reduce SonarQube findings by reducing cognitive complexity. Clean up documentation * Feature 2857 support (#3012) * Per #2857, updated User Support section and also modified case of other headers * Per #2857, resolved duplicate explicit target name warning. * Per 2857, added information about the team drive where data is uploaded to and labelling for organization or institution. * add swpc_metpy env to GHA workflow to create conda envs * pull changes from metplus-feedstock * reduce required python version to match metplus/PYTHON_VERSION_MIN * add met_tool_wrapper use case to existing group * Feature 2999 improve diffs for met tests (#3019) * added some temp code to check for .log files being diffed * removed checks for .log files / reverted * added some more detail to output, including number files * changed calc_transport.log to calc_transport.out and updated ref in docs * skip .log files * changes to diff_text_lines * added temp check for files without extensions * added temp check for files without extensions / fixed * don't compare files without extensions, e.g. file_list in MET tests * forgot to hit save after deleting a couple lines before the last commit... * improved handling of nc files * enhanced numeric diffs for nc files * implemented logic for setting tiny values to zero as pper MET feature 3134 * implemented diff checking on sig figs * slight tweak to order of equality checks * added debug line * more debugging * fixed _round_sig_figs * fixed bug in _set_zero * another fix * reinstated rounding precision for diffs after trying only sig fig rouding * bumped SIG_FIG for diffs to 7 * fix to _round_sig_figs * simplified _is_number() * some tidying * refactor/cleanup to remove code smells reported by SonarQube * more cleanup * added unit tests for skip file extensions logic and new equal comparison functions * add comments to describe each list of file extensions --------- Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * add config variables for setting file_type in the climo_mean/stdev dictionaries (#3027) * Feature 2921 develop sphinx macports (#3025) * Per #2921, adding documentation for setting up a sphinx environment using MacPorts * Per #2921, adding context to explain why a developer may need to run these commands. * Feature #2575 TCI use case - generic model (#2768) * Generalizes METplus config file entries relevant to the FCST dataset used to compute TCI. * per #2575, rename variables, scripts, etc. to avoid using model-specific wording to encourage users to modify this use case to use other models * cleanup * merged (old) develop and resolved conflicts * fixed incorrect merge * update use case doc file to more closely match use case documentation template * replace bad quotation mark character and fix indentation * fix formatting * more formatting changes * fix formatting to match template for METplus Workflow section and moved use case specific config variable descriptions inside dropdown for python scripts * Removes errant include of Python embedding script, tidies up glossary RST, and adds special note under the dataset section that if a user has GRIB or other data without a time dimension, they need to somehow create a data object or file with a time dimension in order to use the fcst_tci Python embedding script since it assumes there is a time dimension on the forecast data. --------- Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> * Feature #2980 refactor PCPCombine file search (#3017) * per #2980, refactor PCPCombine wrapper to find input files like other RuntimeFreq wrappers to be consistent. Clean up logic to remove duplicate code. Update pytests to match changes to wrapper * fix typo in name of temporary var list * clean up logic for SonarQube * handle if temp var (field) list is None or an empty list * fixed bug where USER_DEFINED method was reporting an error when it shouldn't be because it doesn't gather any files * refactored logic into function to reduce duplicate code * move call to set_environment_variables right before call to app so environment variables are not printed if the command will not be run * update ExtractTiles error count if regrid_data_plane has any errors so entire run fails from error * add unit test to capture bug where ExtractTiles is failing * fix bug from previous commit by settings output template/dir from FCST/OBS * update test to run just FCST and just OBS instead of always both * change input template config to legacy supported OBS_REGRID_DATA_PLANE_TEMPLATE (instead of OBS_REGRID_DATA_PLANE_INPUT_TEMPLATE) to reproduce bug in RegridDataPlane_multi_field_one_file use case * clean up formatting complaints * reduce cognitive complexity of function. use utility function instead of calling regex logic explicitly. make returns consistent * fix bug in RegridDataPlane -- this wrapper supports setting a generic FCST/OBS_REGRID_DATA_PLANE_TEMPLATE that can be used as both _INPUT_TEMPLATE and _OUTPUT_TEMPLATE. This is used in one of the met_tool_wrapper examples but was not supported by RuntimeFreq's get_input_templates, so I modified the function to look for generic TEMPLATE if INPUT_TEMPLATE was not set * reset level to be used in the output filename template substitution back to the output accumulation aka lookback seconds * Only require input template to be set for run methods other than USER_DEFINED and SUM. * add unit test for USER_DEFINED run method * refactor to reduce duplicate code * add pytest fixtures to reduce duplicate code in unit tests * improve/expand SUPPRESS_WARNINGS logic * add argument to skip logging when input files cannot be found and just return True/False * clean up formatting * refactor RegridDataPlane wrapper to use RuntimeFreq input file finding logic and remove functions that are no longer needed * update unit tests to match changes to wrapper * fix ExtractTiles to not report errors when files are not found (previously reported warnings) * reduce cognitive complexity * split copyable environment variable commands onto their own lines so it is easier to read and share with others for debugging purposes * refactor ExtractTiles wrapper to find files using RuntimeFreq logic since it calls RegridDataPlane to create the tiles. Update unit tests to reflect changes and ensure that failures are captured properly * clean up formatting * clean up formatting * Feature 3014 sfs updates (#3020) * Adding graphic * Updating python embedding script location * Updating environment * Updating environment * Removing bad path * Adding CI override * Updating documentation * Updating documenttion again * Take 3 * Another documentation update * Updating use case * Updating the order of processes * Updating documentation * Cleaning up unused variables * Updating documentation * Adding climatology. Testing will fail b/c haven't uploaded data * Adding Series Analysis climatology and updating documentation * Another doc update * Fixing valid_beg * fixing time * Fixing documentation typo * Fixed another typo * Fixing log verbosity and block size * Updating some code * Fixing typo * Testing removal of file_type * Removing file type * Setting tests to false for pull requests * minor grammar, spelling updates * Fixing increment * Fixing Log level * Update GridStat_fcstSFSGSL_obsERA5Land_SoilMoisture.conf * editing plot title --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> * Feature 2723 use case sep climates (#3030) * added config and docs. Added use case to testing files. Still needs img, testing * updated config with file_type, use case img * Small grammar change --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> * Feature 2390 ctp hum index (#2981) * Initial Python embedding script for computing CTP from GDAS prepbufr radiosondes (obs). * Cleanup of extra debugging and development blocks. * Initial Python embedding script for computing HI from GDAS prepbufr radiosondes (obs). * Minor changes to print statements. * Major cleanup. * Removes check for highest pressure since that could vary. Check was moved to function in METcalcpy. * Conf file for this use case. * Python embedding script for FCST for CTP. * Changes for UserScript and GenVxMask. * Changes for function signature to take a station_id argument. * Python embedding script to compute Humidity Index from UFS/HR1 forecast data. * Script to create a lat/lon text file for GenVxMask to read and create a gridded mask file to use, based on radiosonde station locations in a GDAS prepBUFR file. * Updates to account for new station_id argument by calc_ctp function in METcalcpy. * Major changes to workflow using multiprocessing for computing CTP from UFS/HR1 forecast data. * Removes erroneous code from script that this script was copied from. * Removes unused code from development and print statements from development. * Removes print statements used during development. * Removes unused code from development and print statements from development. * Updated use case configuration file including config sections for each metric, GenVxMask options, and UserScript options. * Turns down verbosity for PointStat. * Adds print statement that the Python embedding script is starting for log purposes. * Adds print statement to log when it is starting to each Python embedding script, and also removes the interp argument from calc_humidity_index for obs to use the default value in METcalcpy. * Adds debug message for printing the station being processed, with the default turned off. * Major updates to documentation file for use case. * Updates to the UserScript to allow a user to control which sites are included in the mask file for GenVxMask. * Cleanup of use case config gile to remove unused items. * Switches to F strings, wraps some print statements in DEBUG, and switches to NumPy max/min. * Removes unused functions from forecast Python embedding scripts. * Switches to F strings and wraps some print statements in DEBUG flag. * Switches to F strings and adds script name to print statements. * Adds space around print statements to aid log file inspection. * Adds DEBUG level as a command line argument and wraps last remaining print statement in DEBUG flag. This can now be controlled via the METplus config file. * Adds controlling DEBUG level for observation Python embedding scripts. * Updates documentation URL in config file. * Changes subsequent processing steps to read from OUTPUT_DIR, adjusts for subset of global grid to demonstrate functionality, and adjusts input data paths to be relative to INPUT_BASE and the use case category structure for GHA. * Updates for sample data and running the use case in GHA. * Typo in the JSON file. * Changes the MET_PYTHON_DIR to the installed location rather than the source code location. * Adds user script to create graphic from PointStat MPR output. * Adds use case gallery image and updates observation dataset info in documentation. * Fixes path to config file include in documentation. * Adds new user script for plotting to the documentation file. * Updates list of expected output files to include UserScript outpout and PB2NC output. * Changes to eliminate code smells. * Finalizes forecast dataset description and scientific objective. * Adds info on additional Python packages required for Python embedding and UserScripts. * Removes extra #. * Removes uneccesary imports in user script. * Adds more clarity that METcalcpy is a required Python package for Python embedding. * Moves the setting of MET_PYTHON_EXE into the configuration file, as the user should ensure the version of Python in their environment at the time of running the use case meets the requirements for both Python embedding and Python UserScripts. * Removes MET_PYTHON_EXE since it will not be needed if a user has compiled MET against a version of Python that has the requirements for Python embedding. MET_PYTHON_EXE should be set in the users' local conf file, and as described in the documentation, simply be set to python3. * Adds updated note for users in the Running METplus section that provides guidance on how to ensure the correct Python is used for the various components (Python embedding, Python UserScripts). * prevent exception when trying to compare a masked (nan) value to a number * turn off use case * added workflow dispatch to workflow to allow rebuild of Docker images periodically to fix any security vulnerabilities in upstream images * added catch for log10(0) (#3043) Co-authored-by: Natalie Babij <nbabij@seneca.rap.ucar.edu> * Feature #2577 develop - WRF Hurricane Matthew use case (#3036) * Feature #2577 WRF Hurricane Matthew use case (#3015) * per #2577, add use case files * turn off use case that shouldn't be on * fix path to data and scripts * fix path to plotting scripts * added scripts to create wrf_plot conda env that contains the METplus Analysis requirements and the wrf package. change use case to use that env and obtain METdataio and METplotpy * fix name of wrf python package * override input and output directories in WRF plotting script * clean up script re: SonarQube * renaming variables for SonarQube * turn off 100-m wind speed plot because zlev files are not included in the input data * added missing closing single quote * add version of package that was obtained for reproducibility * refactor WRF plotting script to split logic into functions to reduce cognitive complexity * try creating output directory before running reformatter * METdataio reformatter does not accept environment variables in its config file, so attempting to change directory to output directory and find input data relative to that directory * change approach to reference plot directory * add WRF and MADIS file types to quick search keywords and start doc for new use case * fix formatting for output file lists and correct info about output directories (use case path are added for automated tests, but are relative to {OUTPUT_BASE} if run by a user) * write log file to logs directory and rename it to include .log extension * add more information for use case doc * turn off use case for PR * refactor map functions script to reduce cognitive complexity for SonarQube * update scientific objective with input from @jaredalee * update use case to reference environment variables in reformatted config file instead of using relative paths and changing directories since changes needed to support env vars in the reformatted are available in develop now * test that changes to use case did not break anything * Feature 2533 ismn use case (#3051) * Initial config file for this use case. * Adds ismn as an acceptable format for ASCI2NC_INPUT_FORMAT. * Latest param file changes. * Documentation file for this use case. * Updates documentation URL in config file. * Updated conf file that produces matched pairs for soil moisture from PointStat. * Updates to documentation for ISMN use case. * Adds UserScripts for plotting. * Latest conf file settings for use case. * Use case gallery image. * Adds summary using NEAREST for point_stat matched pairs. * Adds path to gallery image for use case. * Changes use case template to have the sphinx gallery line separate from the previous section to avoid having the actual sphinx_gallery_path appear in the rendered HTML documentation. Still need to fix use cases that don't have this change. * Per discussion with @georgemccabe and experience using the StatAnalysis Wrapper for the ISMN use case, moving two stat-analysis wrapper config items that were listed as required to the optional section since it worked without setting them. Presumably they may be required based on how a user is running stat-analysis or the stat-analysis wrapper, but they are not always required. * Per discussion with @georgemccabe, adds additional details for adding a new use case instructing the user to return the run setting to false in the use_case_groups.json file prior to merging the PR, which is the desired default setting for all use cases. * Minor change to use case documentation. * Initial work for adding ISMN to Verification Datasets guide. * Adds documentation for the ISMN dataset in the Verification Dataset Guide. * Adds VxDataISMN to use case documentation to link the use case to the verification datasets guide. * Adds new use case to testing suite after adding sample data. * Removes mention of RAL FTP site for sending sample data for a new use case, since that is not allowed anymore and generically refers to the GitHub Discussion describing the new process using Google Drive with a link to the Discussion. * Forgot a space. * Update to MET 12.0.3 bugfix version (#3059) * Update to MET 12.0.3 bugfix version. * Switch from using "secrets.DOCKER_USERNAME/secrets.DOCKER_PASSWORD" to using "dtcenter/secrets.DOCKER_TOKEN". * Feature #3054 develop scan_for_CVEs (#3055) * Per #3054, enhance release-docker-images.yml to support CVE scanning, scheduled weekly runs for one or more versions, and update the logic for pushing X.Y-latest tags to Docker Hub. * Per #3054, update the Release Guide with instructions about updating the GHA release workflows. * Per #3054, fix workflow typo. * Per #3054, update release guide formatting. * Per #3054, add CMD_LOGFILE logic to the time_command function. * Per #3054, refine release guide instructions. * Per #3054, refine release guide instructions. * Per #3054, update the versions in the release-docker-images.yml workflow for version 6.1.0. * Per #3054, update to plotly 6.1.1 and kaleido 1.0.0 versions. * Per #3054, revert changes to python package versions since those do not actually impact the GHA testing environment. * Per #3054, updating plotly version to 6.1.1 and python-kaledio version to 1.0.0 in the documentation and internal scripts. Hoping that will enable the 4 failing METplus use cases to succeed. * Replace metplotpy 3.0 with 3.0.1 under the METplus 6.0 release (#3060) * Replace metviewer 6.0.0 with 6.0.1 (#3065) * updates that were made to main_v6.1 to handle chrome dependency for METplotpy use cases * rotate authorship as was done in main_v6.1 branch * fix typo * update version of METexpress for 6.1 Coordinated Release * add look up for upcoming 6.2 coordinated release * Update the release guide with by adding a |downloadURL| variable since MET/METviewer/METexpress store downloads in a separate website location than the other components. (#3074) * update instructions to create a new project board to use templates (#3075) * update instructions to create a new project board to use templates * improve instructions for updating Zenodo * Apply suggestions from code review Co-authored-by: John Halley Gotway <johnhg@ucar.edu> --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * develop docs v6.2.0 release timeline (#3083) * Update the development timeline for METplus version 6.2.0 and fix a minor error in the Release Guide insturctions for MET. * Removing accidental new file. * Feature #3068 DataIngest wrapper (#3084) * clean up * per #3068, add utility function to download files and added unit tests to confirm it works as expected * return from function is uncompress extension is not present * remove unnecessary close() calls because with handles closing of file handles * Log error and return from init function if c_dict was not initialized properly. This allows create_c_dict function to return None in the child class wrappers (not parent classes) * per #3068, create basic components of DataIngest wrapper * Add argument to getraw function to skip replacement of double slash with single slash. The replacement was added to prevent errors with paths that contain double slashes in MET config files, but this causes issues with URLs. * clean up to remove unnecessary variable * Rename DATA_INGEST_<n>_URL/LOCAL_PATH to INPUT/OUTPUT_TEMPLATE to be consistent with other wrappers. Fix incorrect has_option config function name. Improve logging so it is clear that a file is currently being downloaded. Added skip-if-output-exists logic * added basic use case for new wrapper * Handle gz extension as gzip file. Refactor new functions to reduce cognitive complexity * add option to auto decompress files before writing to disk * add use case documentation * add new use case to run in automated test suite * added config options for INPUT/OUTPUT_DIR to be consistent with other wrappers -- added workaround for getdir function to prevent replacement of double slash in URLs * fixed typo * added documentation for config variables for new wrapper * turn off new use case before creating PR * remove test that takes a very long time to download file * removed deprecated config variable * fixed bug in enhancement to config getdir to properly handle skip of double slash handling * improve logging for readability * change lead sequence in example because it is not used * Per #3068, add unit tests for new DataIngest wrapper * fix typo * refactor automated test scripts to reduce duplication in logic to run commands * improve logging so it is more clear that a diff test failed because it is likely that the truth data does not exist yet (e.g. for new use cases) and exit 2 to match return code for a use case that contains actual diffs in the output * Update the METplus-6.1 version of MET from 12.1.0 to 12.1.1 after the first bugfix release. (#3093) * Per #3087, add support for setting message_type_group_map in PB2NC and EnsembleStat wrappers (#3092) * update climo stdev variable name after input data was updated by @j-opatz * change sonar host url to env to view url for quality gate scan results * Fix FSTDEV typo in statistics_list.rst * Update release-notes.rst for the METplus-6.2 Release Schedule * Feature #3028 RMWAnalysis configs (#3094) * Per #3028, added support for many new rmw_analysis MET config options. Refactor logic to read METplus configs and handle MET config env vars to loop over names and handle them consistently. Reordered variables to match new order in RMWAnalysis_default config file * fix missing line break that caused warnings in doc build * change input URLs to obtain files from DTC web server because MADIS data changed since the truth data was created --------- Co-authored-by: MET Tools Test Account <met_test@seneca.rap.ucar.edu> * Feature #3069 Generate run times using templates (#3089) * removed unused imports * removed unused import * Remove loop_over_times_and_call function because time looping should be handled in RuntimeFreq wrapper. Report an error if a wrapper does not properly inherit from RuntimeFreq or implement its own version of run_all_times * update unit tests because CommandBuilder run_all_times now reports an error * added a fixture to create empty files for testing file I/O logic in the unit tests * cleanup to reduce cognitive complexity, remove unnecessary variable, and clean up whitespace for readability * Per #3069, create get_files_and_time_info function to reuse logic to find files using a directory and template for file window logic and new logic to generate list of times to run based on existing files on disk. Add logic to time_generator to support TIME_GENERATOR_INPUT_DIR/TEMPLATE to generate init or valid times to process. Add unit tests to ensure that correct, sorted list of init/valid times are generated. Add helper function for unit tests to ensure that the expected keys for each time dictionary are present and call helper function from tests that use time_generator to generate run times from beg/end/increment and time list. Added stub to implement similar logic to extract available forecast leads for each init/valid time * Per #3069, use TIME_GENERATOR_INPUT_DIR/TEMPLATE to get list of forecast leads that correspond to the current init or valid time being processed. Updated tests to include this functionality * clean up docstring comments * Per #3069, add tests to ensure that logic to allow multiple time generator dir/templates and get the intersection of the run times works as expected before implementing it * Per #3069, enhance time generator dir/template logic to support multiple templates and get the intersection of the run times * add test for a single directory with multiple templates and add helper functions to reduce duplication * whitespace * add function to intelligently split dir/template values so that the dir will contain the path without any template tags and the template will contain any filename template tags so users can include filename template tags in _INPUT_DIR variables and file window and time generator parsing will still work as expected. Added ample tests to ensure correct behavior * correct comments * update test to include full list of existing wrappers * clean up tests and add test for case that could occur in containers and breaks dir/template split logic * implement fix that properly splits dir/template when a template is used in the root (/) directory * check if either TIME_GENERATOR_INPUT_DIR or _TEMPLATE are set to execute logic to get …

georgemccabe added 30 commits December 16, 2024 14:58

update release guide for METplus to create data location for next rel…

fa1e5d4

…ease on mohawk when creating rc1 release instead of after official release

cleanup

29e1065

per #2781, start first implementation of PairStat wrapper

98ff97e

merge develop and resolve conflicts

0d9f991

appease SonarQube by increasing code coverage for component versions …

39e55a9

…script

add new wrapper to lookup dictionary

1ec2375

per #2781 add wrapped MET config file

4e2905a

use runtime freq logic to find input files

b70b2eb

start unit tests (broken)

1c29d7a

Merge branch 'develop' into feature_2781_pair_stat

4267972

reorder assert to align expected and actual values with PyCharm notation

71dc444

update unit tests

34a9df8

rename field info variable 'field' to 'pairs' to match Pair-Stat nami…

d8e352f

…ng convention. Add command line arguments to pair_stat command

reorder assert to align actual and expected in PyCharm

a1b5356

refactor function to get start/end times to prevent error when using …

b488c30

…INIT/VALID_LIST and use time_generator to be more consistent and reduce duplicate code

change many wrappers to be consistent with finding input files. Error…

828bf4d

…s are now thrown when any input file is not found, checking other input types even when another input was not found. This increased errors reported in unit tests, so updated tests to reflect this.

remove line that is not needed

a6dfac7

fix bug introduced with recent changes and run use cases that failed …

766a5a0

…to test that the fix is correct

fix multi-variate MODE run to pass all fields to the call instead of …

6662d22

…just the first one

update SeriesAnalysis test to use logic that is actually used when ru…

01fb68f

…nning

turn off use cases that now succeed and turn on use case to test fix …

2ff3bdf

…to multi-variate mode

turn off use case after confirming that it now runs successfully

4cac443

refactor logic to satisfy SonarQube to reduce cognitive complexity an…

c30b2d1

…d remove unused imports

remove functions that are not used

083b4ed

handle -pairs flag like other command line args for input files. Add …

f227f84

…documentation to new functions

updates to new wrapper, add documentation, basic use case stubs. crea…

259e765

…te function for duplicate code to handle land_mask and topo_mask in PointStat and PairStat wrappers

remove unused import

0c9ba22

remove config variable that is no longer used in MET

fd51e3a

remove match_month support from climo_mean/stdev

d5ac433

remove obs_valid_beg/end from command line args because it is not sup…

045cbfb

…ported by pair_stat