Replies: 4 comments 14 replies
2026-07-31 zppy testDisclaimers:
Below I put the results into the format explained on the manual testing docs Step 1: Determine what the current expected results arels -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Last updated on May 20The expected results correspond to the 5/19-20 test Step 2: Review changes since expected results were updated
The automated test scriptThe test script developed in #774 handles these steps:
Running the scriptcd ~/ez/zppy
git status
# On branch copilot/update-default-www-and-simboard-type
# nothing to commit, working tree clean
git fetch upstream automate-main-branch-testing
git checkout automate-main-branch-testing
git log --oneline | head -n 33
# Good, already matches https://github.com/E3SM-Project/zppy/pull/774/commits
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y # <4 min
conda env create -f conda/dev.yml -n zppy-automate-main-branch-testing-20260731
conda activate zppy-automate-main-branch-testing-20260731
pre-commit run --all-files
python -m pip install .
# Use scratch copies of the repos to avoid interference:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_to_cmip && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_diags && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/MPAS-Analysis && git status
# nothing to commit, working tree clean
# Note: we won't actually be using the MPAS-Analysis repo for this run.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy-interfaces && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# nothing to commit, working tree clean
squeue -u $USER # No jobs queued
cd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
mkdir -p ~/ez/zppy_main_branch_tests/test_20260731_run1
cd ~/ez/zppy_main_branch_tests/test_20260731_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Edit configuration parameters
emacs zppy_test.cfg
# ZPPY_BASE_BRANCH="automate-main-branch-testing"
# MPAS_ENV_TYPE="unified"
# EZ_DIR="/lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs"
screen
ulimit -s unlimited
cd ~/ez/zppy_main_branch_tests/test_20260731_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# Ctrl-A D to detach from screen
screen -ls
# 2128940.pts-41.chrlogin1 (Detached)
tail -f integration_test_run1.logApproximately 2.5 hours after launching the script it completes. Note the runtime will depend heavily on node availability. Test results: # test_last_year.py
# 1 passed in 2.38s
# tests/integration/test_bash_generation.py
# 1 failed in 3.66s -- failures expected based on code changes
# test_campaign.py
# 6 failed in 3.37s -- failures expected based on code changes
# test_defaults.py
# 1 failed in 1.47s -- failures expected based on code changes
# test_bundles.py
# 2 passed in 0.84s
# [2026-07-31 19:12:02] ⚠ test_images.py requires a compute node and must be run manually.
# [2026-07-31 19:12:02] To run it on chrysalis:
# [2026-07-31 19:12:02] salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
# [2026-07-31 19:12:02] source /home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh
# [2026-07-31 19:12:02] conda activate test-zppy-automate-main-branch-testing-20260731_run1
# [2026-07-31 19:12:02] cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
# [2026-07-31 19:12:02] pytest tests/integration/test_images.py
# [2026-07-31 19:12:02] cat test_images_summary.md
# [2026-07-31 19:12:02] ✓ Phase 3 automated tests complete!
# [2026-07-31 19:12:02] ✓ Remember to run test_images.py manually from a compute node.
# [2026-07-31 19:12:02] ✓ Integration test automation complete!Finishing Step 8: Run Python tests# CTRL C # Exit tail
screen -R
# real 153m43.625s
# user 7m16.449s
# sys 1m46.787s
exit # Exit screen
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
source /home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh
conda activate test-zppy-automate-main-branch-testing-20260731_run1
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_images.py
# 1 failed in 1839.50s (0:30:39)OutputComplete summary table
Summary table – only failing image-check tests, sorted by task
Results analysis
Notes for future testing
|
9 replies
|
@mkstratos thanks for confirming, it does look like the diff was caused by this units conversion, could you elaborate how this variable and e3sm_diags is used in your workflow? I'm asking because we can decide how best to fix this issue, also where is the satellite dataset is served, should it be included in e3sm_diags? |
|
@chengzhuzhang, the workflow in question belongs to @czender, perhaps he can chime in, but the MODIS data are in the private diags directory on Chrysalis, ready to be synced. Is there something else I can do to support this? |
|
thanks @mkstratos! To address the image difference identified in Ryan’s test, I plan to update the hard-coded contour levels for FSNO from [0, 1] to [0, 100] to reflect the unit conversion from a fraction to a percentage. When @czender has a chance to weigh in, we can discuss the future workflow and whether to include the MODIS comparison as a default e3sm_diags plot. |
|
Hi @chengzhuzhang, As @mkstratos says, the units are in percent in the raw MODIS data so we scale FSNO by 100 for comparison. Most people think in percent not fraction for snow cover, and keeping the units in percent rather than as decimal fractions allows us to store the results as unsigned bytes, which reduces storage requirement by 4x relative to floating point. I vote for making this output a default. We would probably have to run that decision by the land group to ensure that FSNO is always output in the future, but FSNO is in the v3 ELM defaults. Any other points to discuss? |
|
@czender thank you for chiming in. I should have clarified that I asked this question because the unit change and MODIS addition is made in e3sm_diags (E3SM-Project/e3sm_diags#1065), but I'm not sure how e3sm_diags is used in this workflow as the MODIS data is not included in e3sm_diags. |
2026-08-04 zppy test (partial run)Disclaimers:
Below I put the results into the format explained on the manual testing docs Step 1: Determine what the current expected results arels -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Last updated on May 20The expected results correspond to the 5/19-20 test. That is, we haven't updated the expected results since the 7/31 test Step 2: Review changes since expected results were updated
The automated test scriptThe test script developed in #774 handles these steps:
Running the scriptcd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
git log --oneline | head -n 33
# Good, already matches https://github.com/E3SM-Project/zppy/pull/774/commits
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y # <4 min
conda env create -f conda/dev.yml -n zppy-automate-main-branch-testing-20260804
conda activate zppy-automate-main-branch-testing-20260804
pre-commit run --all-files
python -m pip install .
# Use scratch copies of the repos to avoid interference:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_to_cmip && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_diags && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/MPAS-Analysis && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy-interfaces && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Uncommitted changes exist!
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
squeue -u $USER # No jobs queued
cd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
mkdir -p ~/ez/zppy_main_branch_tests/test_20260804_run1
cd ~/ez/zppy_main_branch_tests/test_20260804_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Edit configuration parameters -- use mpas_analysis dev environment, only run mpas_analysis & global_time_series
emacs zppy_test.cfg
# ZPPY_BASE_BRANCH="automate-main-branch-testing"
# DIAGS_ENV_TYPE="unified"
# E3SM_TO_CMIP_ENV_TYPE="unified"
# TASKS_TO_RUN="mpas_analysis,global_time_series"
# EZ_DIR="/lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs"
screen
ulimit -s unlimited
cd ~/ez/zppy_main_branch_tests/test_20260804_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# Ctrl-A D to detach from screen
screen -ls
# 591465.pts-28.chrlogin1 (Detached)
tail -f integration_test_run1.log
# ~20 min in: Jobs remaining: 60 (elapsed: 0s / max: 14400s)Approximately 1h20m later: Output directory results: Test results: # test_last_year.py
# 1 passed in 1.92s
# tests/integration/test_bash_generation.py
# 1 failed in 2.62s -- failures expected based on code changes
# test_campaign.py
# 6 failed in 3.02s -- failures expected based on code changes
# test_defaults.py
# 1 failed in 0.60s -- failures expected based on code changes
# test_bundles.py
# 2 failed in 0.20s -- we simply didn't run all the necessary jobs to do this test.Finishing Step 8: Run Python tests# CTRL C # Exit tail
screen -R
# real 78m46.827s
# user 5m49.741s
# sys 1m8.840s
exit # Exit screen
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
source /home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh
conda activate test-zppy-automate-main-branch-testing-20260804_run1
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_images.pyOutputComplete summary table
Summary table – only failing image-check tests, sorted by taskMPAS-Analysis
Global Time Series
Results analysis
|
1 reply
2026-08-05 zppy test (partial run)Disclaimers:
Below I put the results into the format explained on the manual testing docs Step 1: Determine what the current expected results arels -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Last updated on May 20The expected results correspond to the 5/19-20 test. That is, we haven't updated the expected results since the 7/31 test Step 2: Review changes since expected results were updated
The automated test scriptThe test script developed in #774 handles these steps:
Running the scriptcd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
git log --oneline | head -n 36 # 35 commits based off #843
# Good, already matches https://github.com/E3SM-Project/zppy/pull/774/commits
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y # <4 min # 08:47
conda env create -f conda/dev.yml -n zppy-automate-main-branch-testing-20260805
conda activate zppy-automate-main-branch-testing-20260805
pre-commit run --all-files
python -m pip install .
# Use scratch copies of the repos to avoid interference:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_to_cmip && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_diags && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/MPAS-Analysis && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy-interfaces && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Uncommitted changes exist!
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
squeue -u $USER # No jobs queued
cd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
mkdir -p ~/ez/zppy_main_branch_tests/test_20260805_run1
cd ~/ez/zppy_main_branch_tests/test_20260805_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Edit configuration parameters -- use mpas_analysis dev environment, only run mpas_analysis & global_time_series
emacs zppy_test.cfg
# ZPPY_BASE_BRANCH="automate-main-branch-testing"
# DIAGS_ENV_TYPE="unified"
# E3SM_TO_CMIP_ENV_TYPE="unified"
# MPAS_ENV_TYPE="unified"
# ZI_ENV_TYPE="unified"
# TASKS_TO_RUN="mpas_analysis,global_time_series,pcmdi_diags"
# EZ_DIR="/lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs"
screen
ulimit -s unlimited
cd ~/ez/zppy_main_branch_tests/test_20260805_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# Ctrl-A D to detach from screen
screen -ls
# 2812035.pts-41.chrlogin1 (Detached)
tail -f integration_test_run1.log
# ~10 min in: Jobs remaining: 107 (elapsed: 0s / max: 14400s)Approximately 1h later: Unit test results: Output directory results: Test results: # test_last_year.py
# 1 passed in 1.92s
# tests/integration/test_bash_generation.py
# 1 failed in 3.60s -- failures expected based on code changes
# test_campaign.py
# 6 failed in 3.73s -- failures expected based on code changes
# test_defaults.py
# 1 failed in 0.82s -- failures expected based on code changes
# test_bundles.py
# 2 failed in 0.40s -- we simply didn't run all the necessary jobs to do this test.Finishing Step 8: Run Python tests# CTRL C # Exit tail
screen -R
# real 59m30.612s
# user 2m42.693s
# sys 0m30.895s
exit # Exit screen
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
source /home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh
conda activate test-zppy-automate-main-branch-testing-20260805_run1
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_images.pyOutputComplete summary table
Summary table – only failing image-check tests, sorted by taskN/A -- no image-check failures. Results analysis
|
0 replies
2026-08-05 zppy test (partial run)Disclaimers:
Below I put the results into the format explained on the manual testing docs Step 1: Determine what the current expected results arels -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Last updated on May 20The expected results correspond to the 5/19-20 test. That is, we haven't updated the expected results since the 7/31 test Step 2: Review changes since expected results were updated
The automated test scriptThe test script developed in #774 handles these steps:
Running the script# This section is just to make sure the zppy repo is in good shape to copy testing files from.
# 1. Contains the testing files we want to copy.
# 2. Passes pre-commit checks
cd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
git log --oneline | head -n 36 # 35 commits based off #843
# Good, already matches https://github.com/E3SM-Project/zppy/pull/774/commits
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y # <4 min
conda env create -f conda/dev.yml -n zppy-automate-main-branch-testing-20260805-run2
conda activate zppy-automate-main-branch-testing-20260805-run2
pre-commit run --all-files
python -m pip install .
# Use scratch copies of the repos to avoid interference:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_to_cmip && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/e3sm_diags && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/MPAS-Analysis && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy-interfaces && git status
# nothing to commit, working tree clean
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Uncommitted changes exist!
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
squeue -u $USER # No jobs queued
cd ~/ez/zppy
git status
# On branch automate-main-branch-testing
# nothing to commit, working tree clean
mkdir -p ~/ez/zppy_main_branch_tests/test_20260805_run2
cd ~/ez/zppy_main_branch_tests/test_20260805_run2
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Edit configuration parameters -- use mpas_analysis dev environment, only run mpas_analysis & global_time_series
emacs zppy_test.cfg
# RUN_NUMBER=2
# ZPPY_BASE_BRANCH="automate-main-branch-testing"
# DIAGS_ENV_TYPE="unified"
# E3SM_TO_CMIP_ENV_TYPE="unified"
# MPAS_ENV_TYPE="unified"
# TASKS_TO_RUN="mpas_analysis,global_time_series,pcmdi_diags"
# EZ_DIR="/lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs"
screen
ulimit -s unlimited
cd ~/ez/zppy_main_branch_tests/test_20260805_run2
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run2.log
# Ctrl-A D to detach from screen
screen -ls
# 3123240.pts-41.chrlogin1 (Detached)
tail -f integration_test_run2.log # CTRL C # Exit tail
screen -R
# real 70m1.157s
# user 4m18.390s
# sys 0m52.527s
exit # Exit screen
cd ~/ez/zppy_main_branch_tests/test_20260805_run2
grep -n "unit tests passed" integration_test_run2.log
grep -n "Checking all status files..." integration_test_run2.log
grep -n "All status files clean!" integration_test_run2.log
grep -n "passed in" integration_test_run2.log
grep -n "failed in" integration_test_run2.logUnit test results: Output directory results: Test results: # test_last_year.py
# 1 passed in 2.04s
# tests/integration/test_bash_generation.py
# 1 failed in 1.54s -- failures expected based on code changes
# test_campaign.py
# 6 failed in 3.09s -- failures expected based on code changes
# test_defaults.py
# 1 failed in 0.56s -- failures expected based on code changes
# test_bundles.py
# 2 failed in 0.32s -- we simply didn't run all the necessary jobs to do this test.Finishing Step 8: Run Python testssalloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
source /home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh
conda activate test-zppy-automate-main-branch-testing-20260805_run2
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_images.pyOutputComplete summary table
Summary table – only failing image-check tests, sorted by taskMPAS-AnalysisN/A -- all image check tests pass (as expected -- we're comparing results from E3SM-Unified to results from E3SM-Unified) Global Time Series
PCMDI Diags
Results analysis
|
4 replies
|
@zhangshixuan1987 I'm seeing some pretty substantial diffs in the Claude suggests the difference in Python version (and possibly numpy, scipy) could be responsible for the large diffs seen for
Table of dependency versions that Claude found:
|
|
Do you think the version differences in those packages could actually cause the diffs we're seeing in |
|
@forsyth2: if you look at the ta 200: actual vs expected, you will find that the min/max range are exactly the same, while the color bar is changed. I think that this could because some of update in pmcdi_metrics from 4.0.4 to 4.1.0. Also, for the synthetic metrics of Model performance of annual climatology: actual vs expected. I also suspect that this is due to some updates in PCMDI from 4.0.4 to 4.1.0. I recall that there are some changes in the parallel coordinate plot parameters in newer version of the pcmdi. We should confirm with Jiwoo whether these differences are expected consequences of the PCMDI Metrics version update. |
Ah, thanks @zhangshixuan1987! Yes of course this must be the reason. I somehow missed that line of the diff. Note to self: really there's three layers here, not two.
@lee1043 Can you confirm those diffs would be causes by PCMDI 4.0.4 to 4.1.0? The actual is concerning, since it chops off the y-axis labels. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
The original weekly testing log was getting overly long, so I think it will be better to start a new testing log for each development period. With that in mind, here's the testing log for development work for the next Unified release -- E3SM Unified 1.14.0 (Fall 2026)
All reactions