Update develop-ref after #2341, #2337 (#2343)

Co-authored-by: Lisa Goodrich <lisag@ucar.edu> Co-authored-by: Julie Prestopnik <jpresto@seneca.rap.ucar.edu> Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com> Co-authored-by: Hank Fisher <fisherh@ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> Co-authored-by: johnhg <johnhg@ucar.edu> Co-authored-by: John Halley Gotway <johnhg@ucar.edu> Co-authored-by: jprestop <jpresto@ucar.edu> Co-authored-by: Tracy Hertneky <hertneky@seneca.rap.ucar.edu> Co-authored-by: Giovanni Rosa <giovanni.rosa@unimol.it> Co-authored-by: j-opatz <59586397+j-opatz@users.noreply.github.com> Co-authored-by: Mrinal Biswas <biswas@seneca.rap.ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu> Co-authored-by: Jonathan Vigh <jvigh@ucar.edu> Co-authored-by: root <root@localhost> Co-authored-by: bikegeek <3753118+bikegeek@users.noreply.github.com> Co-authored-by: Will Mayfield <59745143+willmayfield@users.noreply.github.com> Co-authored-by: lisagoodrich <33230218+lisagoodrich@users.noreply.github.com> Co-authored-by: metplus-bot <97135045+metplus-bot@users.noreply.github.com> Co-authored-by: Tracy Hertneky <39317287+hertneky@users.noreply.github.com> Co-authored-by: Giovanni Rosa <g.rosa1@studenti.unimol.it> Co-authored-by: mrinalbiswas <biswas@ucar.edu> Co-authored-by: Christina Kalb <kalb@ucar.edu> Co-authored-by: jason-english <73247785+jason-english@users.noreply.github.com> Co-authored-by: John Sharples <41682323+John-Sharples@users.noreply.github.com> fix GitHub Actions warnings (#1864) fix #1884 develop PCPCombine {custom} in subtract method (#1887) fix #1939 develop - failure reading obs when zipped file also exists (#1941) Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313)
dtcenter · Sep 13, 2023 · 2ea18bf · 2ea18bf
1 parent 9a3864f
commit 2ea18bf
Show file tree

Hide file tree

Showing 20 changed files with 975 additions and 317 deletions.
diff --git a/.github/ISSUE_TEMPLATE/update_truth.md b/.github/ISSUE_TEMPLATE/update_truth.md
@@ -0,0 +1,60 @@
+---
+name: Update Truth
+about: Review use case differences that are caused by changes in an external repository and update truth dataset if necessary.
+title: 'Update Truth: '
+labels: 'alert: NEED ACCOUNT KEY, alert: NEED MORE DEFINITION, alert: NEED CYCLE ASSIGNMENT, type: update truth, priority: blocker, component: CI/CD, requestor: METplus Team'
+assignees: ''
+
+---
+
+## Describe Expected Changes ##
+
+*Write a short summary of the differences that will likely be found in the GitHub Actions testing workflow that was triggered by the pull request that warranted this issue*
+
+## Define the Metadata ##
+
+### Title ###
+- [ ] Add a link to the pull request that warranted this issue to the issue title using format dtcenter/{REPO}#{PR_NUMBER}.
+
+**Example:** Update Truth: For dtcenter/MET#2655
+
+### Assignee ###
+
+*Assign this issue to the author of the pull request that warranted this issue. Optionally assign anyone else who should review the differences in the output.*
+
+- [ ] Select **engineer(s)** or **no engineer** required
+- [ ] Select **scientist(s)** or **no scientist** required
+
+### Projects and Milestone ###
+- [ ] Select **Repository** and/or **Organization** level **Project(s)** or add **alert: NEED CYCLE ASSIGNMENT** label
+- [ ] Select **Milestone** as the next official version or **Future Versions**
+
+## Update Truth Checklist ###
+- [ ] Review the GitHub Actions workflow that was triggered by the PR merge
+  - If no differences were found, note this in a comment.
+  - If all of the differences are expected, note this in a comment.
+    Include any details of how the review was performed.
+  - If unexpected differences are found, the following instructions can
+    help uncover potential explanations. If none of these apply and the
+    source of the differences cannot be determined, contact the
+    METplus wrappers lead engineer (@georgemccabe) for assistance.
+    - Search for other open issues that have the label `type: update truth`
+      applied by clicking on the label on this issue. Coordinate with the
+      author of these issues to ensure all diffs are properly reviewed.
+    - Check if any additional GitHub Actions testing workflows have been
+      triggered since the workflow that corresponds to this issue was run.
+      Review the latest run to ensure that there are no diffs that are
+      unrelated to this issue.
+    - If the incorrect differences are caused by the changes from the
+      issue that warranted this issue, consider reverting the PR and
+      re-opening the issue.
+  - Iterate until one of the above conditions apply.
+- [ ] Approve the update of the truth data
+  - Contact the METplus wrappers lead engineer (@georgemccabe) or
+    backup lead (@jprestop) to let them know that the truth data can
+    be updated.
+- [ ] Update the truth data.
+      This should be handled by a METplus wrappers engineer.
+      See the (instructions to update the truth data)[https://metplus.readthedocs.io/en/develop/Contributors_Guide/continuous_integration.html#update-truth-data-update-truth-data-yml]
+      for more info.
+- [ ] Close this issue.
diff --git a/.github/actions/run_tests/Dockerfile.run_gempak b/.github/actions/run_tests/Dockerfile.run_gempak
@@ -6,8 +6,9 @@ FROM dtcenter/metplus-envs:${METPLUS_ENV_TAG} as env
 ARG METPLUS_IMG_TAG=develop
 FROM dtcenter/metplus-dev:${METPLUS_IMG_TAG}
 
-COPY --from=env /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64 /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/
+COPY --from=env /usr/lib/jvm /usr/lib/jvm/
+COPY --from=env /etc/java-17-openjdk /etc/java-17-openjdk/
 COPY --from=env /data/input/GempakToCF.jar /data/input/GempakToCF.jar
 
 # add java executable to path
-RUN echo "export PATH=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin:\$PATH;" >> /root/.bashrc
+RUN echo "export PATH=/usr/lib/jvm/java-17-openjdk-amd64/bin:\$PATH;" >> /root/.bashrc
diff --git a/.github/jobs/build_conda_image.sh b/.github/jobs/build_conda_image.sh
@@ -0,0 +1,9 @@
+DOCKER_ENV_DIR=${GITHUB_WORKSPACE}/internal/scripts/docker_env
+cmd="docker build \
+  -t dtcenter/metplus-envs:${ENV_NAME}.${METPLUS_ENV_VERSION} \
+  --build-arg METPLUS_ENV_VERSION \
+  --build-arg BASE_ENV \
+  --build-arg ENV_NAME \
+  -f ${DOCKER_ENV_DIR}/${DOCKERFILE} ${DOCKER_ENV_DIR}"
+echo $cmd
+$cmd
diff --git a/.github/jobs/docker_utils.py b/.github/jobs/docker_utils.py
@@ -16,7 +16,7 @@
 DOCKERHUB_METPLUS_DATA_DEV = 'dtcenter/metplus-data-dev'
 
 # extension to add to conda environments
-VERSION_EXT = '.v5.1'
+VERSION_EXT = '.v6.0'
 
 
 def get_data_repo(branch_name):

diff --git a/.github/jobs/push_conda_image.sh b/.github/jobs/push_conda_image.sh
@@ -0,0 +1,4 @@
+echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
+cmd="docker push dtcenter/metplus-envs:${ENV_NAME}.${METPLUS_ENV_VERSION}"
+echo $cmd
+$cmd
diff --git a/.github/workflows/create_conda_envs.yml b/.github/workflows/create_conda_envs.yml
@@ -0,0 +1,298 @@
+name: Create Conda Environments in Docker
+
+on:
+  workflow_dispatch:
+    inputs:
+      subset:
+        description: 'Comma-separated list of environments to build. Leave blank to build all.'
+        required: false
+        default: ''
+
+env:
+  METPLUS_ENV_VERSION: v6.0
+  DOCKERFILE: Dockerfile
+  BASE_ENV: metplus_base
+  DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+  DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+
+jobs:
+  check:
+    name: Check inputs to determine what to run
+    runs-on: ubuntu-latest
+    steps:
+      - id: parse_list
+        run: |
+          run_str="${{ github.event.inputs.subset }}"
+          if [ "$run_str" == "" ]; then echo "no_skip=true" >> $GITHUB_OUTPUT; exit 0; fi
+          echo "no_skip=false" >> $GITHUB_OUTPUT
+          IFS=', ' read -r -a arr <<< "$run_str"
+          JSON="["
+          for item in ${arr[@]}; do JSON="$JSON\"$item\","; done
+          if [[ $JSON == *, ]]; then JSON="${JSON%?}"; fi
+          JSON="$JSON]"
+          echo "run_list=$( echo "$JSON" )" >> $GITHUB_OUTPUT
+          echo $JSON
+    outputs:
+      no_skip: ${{ steps.parse_list.outputs.no_skip }}
+      run_list: ${{ steps.parse_list.outputs.run_list }}
+
+  conda:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'conda'))
+    runs-on: ubuntu-latest
+    needs: [check]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.conda
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  metplus_base:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'metplus_base'))
+    runs-on: ubuntu-latest
+    needs: [check,conda]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.metplus_base
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  py_embed_base:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'py_embed_base'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.py_embed_base
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  h5py:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'h5py'))
+    runs-on: ubuntu-latest
+    needs: [check,py_embed_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      BASE_ENV: py_embed_base
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  metdataio:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'metdataio'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  pygrib:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'pygrib'))
+    runs-on: ubuntu-latest
+    needs: [check,py_embed_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      BASE_ENV: py_embed_base
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  cfgrib:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'cfgrib'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  netcdf4:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'netcdf4'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  xesmf:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'xesmf'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  spacetime:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'spacetime'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  metplotpy:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'metplotpy'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.cartopy
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  weatherregime:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'weatherregime'))
+    runs-on: ubuntu-latest
+    needs: [check,metplotpy]
+    env:
+      ENV_NAME: ${{ github.job }}
+      BASE_ENV: metplotpy
+      DOCKERFILE: Dockerfile.cartopy
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  cycloneplotter:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'cycloneplotter'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.cartopy
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  icecover:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'icecover'))
+    runs-on: ubuntu-latest
+    needs: [check,py_embed_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      BASE_ENV: py_embed_base
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  gempak:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'gempak'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.gempak_env
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  gfdl-tracker:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'gfdl-tracker'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+      DOCKERFILE: Dockerfile.gfdl-tracker
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  geovista:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'geovista'))
+    runs-on: ubuntu-latest
+    needs: [check,metplus_base]
+    env:
+      ENV_NAME: ${{ github.job }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  pandac:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'pandac'))
+    runs-on: ubuntu-latest
+    needs: [check,metplotpy]
+    env:
+      ENV_NAME: ${{ github.job }}
+      BASE_ENV: metplotpy
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh
+
+  diff:
+    if: |
+      always() && (needs.check.outputs.no_skip == 'true' ||
+      contains(fromJSON(needs.check.outputs.run_list), 'diff'))
+    runs-on: ubuntu-latest
+    needs: [check,netcdf4]
+    env:
+      ENV_NAME: ${{ github.job }}
+      BASE_ENV: netcdf4
+    steps:
+      - uses: actions/checkout@v4
+      - run: .github/jobs/build_conda_image.sh
+      - run: .github/jobs/push_conda_image.sh