Repository navigation
Replies: 6 comments 26 replies
2026-10-06 E3SM Unified 1.14.0rc2 Chrysalis testI'm going to run the tests using the currently unmerged test improvements of #871 (The Set up and run the testSet up the zppy branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
git checkout issue-869-improve-test-automation
# 36 commits from https://github.com/E3SM-Project/zppy/pull/871/commits
git log --oneline | head -n 37
# 8bfc1cbd Add UNIFIED_LOAD_SCRIPT option to test a specific E3SM-Unified
# 79eb98c3 Allow testing zppy from E3SM-Unified in automated test script
# ...
# a1d8e268 Automate image checker, add per-task env descriptions, and auto-generate test report
# b505ce7a Update pre-commit dependencies (#880)Set up the test script# Now, copy the test script and cfg from the zppy repo into the directory
# that you'll be running the test script from.
mkdir -p ~/ez/zppy_main_branch_tests/test_20261006_run1
cd ~/ez/zppy_main_branch_tests/test_20261006_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Now, edit the test cfg as needed
emacs zppy_test.cfgSet up the test cfgls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 30 17:22 mpas_analysis
# Sep 30 17:22 ilamb
# Sep 30 17:21 livvkit
# Sep 30 17:21 pcmdi_diags
# Sep 30 17:21 global_time_series
# Sep 30 17:19 e3sm_diagsshows the date results were promoted to be official expected results. We can look at the testing log to see what the run before that date was.
Run the test script# NOTE: This part is not listed in the docs, as the script should check this.
# To be on the safer side though, it's a good idea to check that these directories
# don't have uncommitted changes.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
cd ~/ez/zppy_main_branch_tests/test_20261006_run1
screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
cd ~/ez/zppy_main_branch_tests/test_20261006_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# [2026-10-06 12:41:22] Starting zppy integration test automation
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_run1.logD. Review the output# CTRL C # Exit tail
screen -R
# [2026-10-06 15:40:42] ✓ Markdown report: /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261006_run1/test_report_20261006_run1.md
# real 179m20.186s
# user 1m40.167s
# sys 3m10.620s
exit # Exit screen
ReportDetails20261006 zppy testSee automated testing docs page for info on setup. Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for Note: this run tested the released zppy from E3SM-Unified, not the tip of the zppy baseline branch, so the
Environment descriptionsAn
Each task's
Automated test script results
Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by task
Results analysisConfirm Unified rc2 was being used for all taskscd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20261006_run1/v3.LR.historical_0051/post/scripts
grep -l "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc2_chrysalis.sh" *.bash
# Good, even the non-plotting tasks (climo, ts, tc_analysis) match this.
grep "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" *.bash
# Good, no matchesThe non-image-checker integration test failures
cd ~/ez/zppy_main_branch_tests/test_20261006_run1
grep -n "test_bash_generation.py" integration_test_run1.log
# 1672:FAILED tests/integration/test_bash_generation.py::test_bash_generationThis failure is because #878 and #879 were merged after the expected results were generated (on 9/28). grep -n "test_campaign.py" integration_test_run1.log
# 2100:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere - assert ...
# 2101:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere_override
# 2102:FAILED tests/integration/test_campaign.py::test_campaign_high_res_v1 - assert...
# 2103:FAILED tests/integration/test_campaign.py::test_campaign_none - assert 256 == 0
# 2104:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle - assert...
# 2105:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle_overrideThis failure is because of an issue with the expected results updater script, as noted here. grep -n "test_defaults.py" integration_test_run1.log
# 2243:FAILED tests/integration/test_defaults.py::test_defaults - assert 256 == 0This failure is because #878 was merged after the expected results were generated (on 9/28). grep -n "test_bundles.py" integration_test_run1.log
# 2291:FAILED tests/integration/test_bundles.py::test_bundles_bash_file_contentThis is failing simply because we changed the Unified env command (to use the release candidate), so this can be ignored. Image check failurese3sm_diagsSome of the diffs look substantial. We should confirm they are to be expected based on the latest code changes. # Compare dependencies
diff /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261006_run1/env_descriptions_20261006_run1/e3sm_diags.txt /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20261006_run1/v3.LR.historical_0051/e3sm_diags/env_description.txt
# Good, no diff; these are identical env descriptions for this run.
# Now, let's compare this env to the env from the 9/28 test
diff /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20260928_run6/env_descriptions_20260928_run6/e3sm_diags.txt /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261006_run1/env_descriptions_20261006_run1/e3sm_diags.txt > /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/zppy_main_branch_test_20261006_run1/v3.LR.historical_0051/e3sm_diags_env_from_0928_test.txtThe env diff is now available here. ilambAs far as I can tell after looking at a few of the diff grids, the diffs look pretty minor and likely to be expected since we had previously been using the E3SM-Unified 1.13.0 version of ILAMB (there's no mpas_analysisAll of these diffs appear inconsequential. |
The 9/28 test. Its results were promoted to baseline on 9/30, across the board (for all tasks). In the "Set up and run the test" expandable section above I noted the following, indicating that the baseline was updated on 9/30. ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 30 17:22 mpas_analysis
# Sep 30 17:22 ilamb
# Sep 30 17:21 livvkit
# Sep 30 17:21 pcmdi_diags
# Sep 30 17:21 global_time_series
# Sep 30 17:19 e3sm_diagsMeanwhile, the Report > "Details" expandable section above includes full documentation on commits made since that 9/28 test run. (That's where I got the e3sm_diags PRs to check: E3SM-Project/e3sm_diags#1096, E3SM-Project/e3sm_diags#1094, E3SM-Project/e3sm_diags#1089, E3SM-Project/e3sm_diags#1088) |
|
It's also important to note the 9/28 test was of course done with dev environments. Those can be quite different from the Unified environment. I.e., many dependencies having different versions may end up producing different results. |
Yes, the differences seems to just be roundoff in the last printed digits of the stats for a few plots. Nothing concerning. |
|
@forsyth2 I tracked down the missing What's missing (v2
The v3 Why:
I reported it upstream: xCDAT/xcdat#850, tagging @tomvothecoder. |
|
xCDAT v0.11.5 is now available on conda-forge and e3sm_diags v3.3.0rc2 will be available within 1-2 hours. Should be ready to re-test tomorrow. |
2026-10-09 E3SM Unified 1.14.0rc3 Chrysalis testI'm going to run the tests using the currently unmerged test improvements of #871 (The Set up and run the testSet up the zppy branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
git checkout issue-869-improve-test-automation
# 36 commits from https://github.com/E3SM-Project/zppy/pull/871/commits
git log --oneline | head -n 37
# 8bfc1cbd Add UNIFIED_LOAD_SCRIPT option to test a specific E3SM-Unified
# 79eb98c3 Allow testing zppy from E3SM-Unified in automated test script
# ...
# a1d8e268 Automate image checker, add per-task env descriptions, and auto-generate test report
# b505ce7a Update pre-commit dependencies (#880)Set up the test script# Now, copy the test script and cfg from the zppy repo into the directory
# that you'll be running the test script from.
mkdir -p ~/ez/zppy_main_branch_tests/test_20261009_run1
cd ~/ez/zppy_main_branch_tests/test_20261009_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Now, edit the test cfg as needed
emacs zppy_test.cfgSet up the test cfgls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 30 17:22 mpas_analysis
# Sep 30 17:22 ilamb
# Sep 30 17:21 livvkit
# Sep 30 17:21 pcmdi_diags
# Sep 30 17:21 global_time_series
# Sep 30 17:19 e3sm_diagsshows the date results were promoted to be official expected results. We can look at the testing log to see what the run before that date was.
Run the test script# NOTE: This part is not listed in the docs, as the script should check this.
# To be on the safer side though, it's a good idea to check that these directories
# don't have uncommitted changes.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
cd ~/ez/zppy_main_branch_tests/test_20261009_run1
screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
cd ~/ez/zppy_main_branch_tests/test_20261009_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# [2026-10-09 11:54:53] Starting zppy integration test automation
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_run1.log
# Started 09:55
# Jobs launched by 10:06
# Check in hourly, should be done about 12:55D. Review the output# CTRL C # Exit tail
screen -R
# [2026-10-09 14:34:39] ✓ Markdown report: /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261009_run1/test_report_20261009_run1.md
# real 159m46.127s
# user 1m39.478s
# sys 3m0.590s
exit # Exit screen
ReportDetails20261009 zppy testSee automated testing docs page for info on setup. Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for Note: this run tested the released zppy from E3SM-Unified, not the tip of the zppy baseline branch, so the
Environment descriptionsAn
Each task's
Automated test script results
Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by task
Results analysisConfirm Unified rc3 was being used for all taskscd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20261009_run1/v3.LR.historical_0051/post/scripts
grep -l "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc3_chrysalis.sh" *.bash
# Good, even the non-plotting tasks (climo, ts, tc_analysis) match this.
grep "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" *.bash
# Good, no matchesThe non-image-checker integration test failuresEssentially the same as the rc2 test
cd ~/ez/zppy_main_branch_tests/test_20261009_run1
grep -n "test_bash_generation.py" integration_test_run1.log
# 1669:FAILED tests/integration/test_bash_generation.py::test_bash_generationMatches error from rc2 test: This failure is because #878 and #879 were merged after the expected results were generated (on 9/28). grep -n "test_campaign.py" integration_test_run1.log
# 2097:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere - assert ...
# 2098:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere_override
# 2099:FAILED tests/integration/test_campaign.py::test_campaign_high_res_v1 - assert...
# 2100:FAILED tests/integration/test_campaign.py::test_campaign_none - assert 256 == 0
# 2101:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle - assert...
# 2102:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle_overrideMatches error from rc2 test: This failure is because of an issue with the expected results updater script, as noted here. grep -n "test_defaults.py" integration_test_run1.log
# 2240:FAILED tests/integration/test_defaults.py::test_defaults - assert 256 == 0Matches error from rc2 test: This failure is because #878 was merged after the expected results were generated (on 9/28). grep -n "test_bundles.py" integration_test_run1.log
# 2288:FAILED tests/integration/test_bundles.py::test_bundles_bash_file_contentMatches error from rc2 test: This is failing simply because we changed the Unified env command (to use the release candidate), so this can be ignored. Image check failurese3sm_diagsLet's compare the results with the rc2 test's results. Helper script to compare two Markdown tablesGenerated by Claude: #!/usr/bin/env python3
"""Compare two zppy image-checker summary tables (markdown) and print a
table of `old -> new (net_gain)` values per test.
Usage:
compare_image_checks.py combined.md # first table = old, second = new
compare_image_checks.py old.md new.md # first table in each file
"""
import re
import sys
COLUMNS = [
"Total images", "Correct images", "Identical", "Cosmetic only",
"Missing images", "Needs review (Structural)", "Needs review (Major)",
"Needs review (Moderate)", "Needs review (Minor)",
]
# Severity-column labels -> output columns
SEVERITY_MAP = {
"structural": "Needs review (Structural)",
"major": "Needs review (Major)",
"moderate": "Needs review (Moderate)",
"minor": "Needs review (Minor)",
}
def split_row(line):
# Cells contain markdown links but never "|", so a plain split is safe.
return [c.strip() for c in line.strip().strip("|").split("|")]
def lead_int(cell):
m = re.match(r"\s*(\d+)", cell)
return int(m.group(1)) if m else 0
def parse_tables(text):
"""Return a list of {test_name: {column: int}} dicts, one per table."""
tables, current, header = [], None, None
for line in text.splitlines():
if not line.strip().startswith("|"):
current, header = None, None
continue
cells = split_row(line)
if cells and cells[0] == "Test name":
header, current = cells, {}
tables.append(current)
continue
if header is None or set("".join(cells)) <= set("-: "):
continue # separator row or stray line
row = dict(zip(header, cells))
if not re.match(r"\s*\d", row.get("Total images", "")):
continue # skip placeholder/template rows
vals = {
"Total images": lead_int(row.get("Total images", "")),
"Correct images": lead_int(row.get("Correct images", "")),
"Identical": lead_int(row.get("Identical", "")),
"Cosmetic only": lead_int(row.get("Cosmetic only", "")),
"Missing images": lead_int(row.get("Missing images", "")),
}
for col in SEVERITY_MAP.values():
vals[col] = 0
for n, label in re.findall(r"(\d+)\s+(\w+)", row.get("Severity", "")):
if label.lower() in SEVERITY_MAP:
vals[SEVERITY_MAP[label.lower()]] = int(n)
current[row["Test name"]] = vals
return [t for t in tables if t]
def fmt(old, new):
if old is None or new is None:
return f"{'—' if old is None else old} -> {'—' if new is None else new}"
return f"{old} -> {new} ({new - old:+d})"
def main(argv):
if len(argv) == 2:
tables = parse_tables(open(argv[1]).read())
if len(tables) < 2:
sys.exit("Expected at least two tables in the file.")
old, new = tables[0], tables[1]
elif len(argv) == 3:
old = parse_tables(open(argv[1]).read())[0]
new = parse_tables(open(argv[2]).read())[0]
else:
sys.exit(__doc__)
print("| Test name | " + " | ".join(COLUMNS) + " |")
print("| --- |" + " --- |" * len(COLUMNS))
names = list(old) + [t for t in new if t not in old]
for name in names:
o, n = old.get(name), new.get(name)
cells = [fmt(o[c] if o else None, n[c] if n else None) for c in COLUMNS]
print(f"| {name} | " + " | ".join(cells) + " |")
if __name__ == "__main__":
main(sys.argv)
The missing images have been resolved along with some of the "Needs review (Minor), but none of the "Needs review (major/moderate)" items have. Are the remaining diffs acceptable? New in We're using the same expected results baseline as we used for the rc2 test. ilambThis matches the rc2 test results. If those diffs were acceptable there, they are here. mpas_analysisThis matches the rc2 test results, which Xylar already confirmed were acceptable. |
|
@forsyth2 thank you! The newly generated figures are the corrected ones, the 9/30 baseline plots include some cartopy0.24 impacted ones and were later fixed by cartopy 0.26, which was used by unified rcs. I think we are all good for e3sm_diags. I also think the diff for ilamb is acceptable. |
|
Great! I'll update the expected results then. |
|
With all failures accounted for, I've updated the expected results baseline. As mentioned above, the next test run will be the first to test the legacy 3.2.0 cfg in place of these older legacy cfgs, and so will produce its first set of expected results. We'll need to make sure it doesn't break. |
@chengzhuzhang Wait, are you sure? If we look at the e3sm_diags E3SMv3 diff grid above, it's showing an entirely red map for the "Actual"s... |
|
@forsyth2 thanks for catching this!! I verified with the v2 results, without noticing v3 has many more land output that show abnormal results. Let me do more troubleshooting. |
2026-10-16 Pre-E3SM Unified 1.14.0rc4 Chrysalis testThis test will test dev environments rather than an E3SM Unified RC. That is because we are trying to test bug fixes and other new development before making package RCs to be included in E3SM Unified 1.14.0rc4. We'll use the currently unmerged test improvements of the stack #887:
Note that none of those PRs touch user-facing code, so from the user perspective, this is still a Set up and run the testSet up the zppy branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
# We need to rebase the testing stack to make sure we're testing the latest `main`
# Test PR 1: #871
git checkout issue-869-improve-test-automation
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 41
# 58fdce18 Remove unnecessary function Copilot added
# ...
# 5f19eb9b Automate image checker, add per-task env descriptions, and auto-generate test report
# e45286e8 Add snow storage, snow cover, river discharge analysis (#888)
# 83cd30e1 E3SM-Unified 1.14 testing: faster diurnal climo on Perlmutter (#890)
# Good, matches the 39 commits of https://github.com/E3SM-Project/zppy/pull/871/commits
# Good, based on https://github.com/E3SM-Project/zppy/commits/main/
git push -f upstream issue-869-improve-test-automation
# Test PR 2: #886
git checkout issue-837-legacy-3.2.0
git fetch upstream issue-869-improve-test-automation
git rebase upstream/issue-869-improve-test-automation
git push -f upstream issue-837-legacy-3.2.0
git log --oneline | head -n 7
# 87ef5ca1 Add livvkit to legacy v3.2.0 testing
# ...
# 92ee104b Copy comprehensive_v3 cfg from v3.2.0
# 58fdce18 Remove unnecessary function Copilot added # From #871
# Good, matches the 6 commits of https://github.com/E3SM-Project/zppy/pull/886/commits
git log --oneline | head -n 48
# ...
# e45286e8 Add snow storage, snow cover, river discharge analysis (#888)
# 83cd30e1 E3SM-Unified 1.14 testing: faster diurnal climo on Perlmutter (#890)
# cc780ec6 Bump to 3.3.0rc1 (#882)
# Good, based on #871 based on https://github.com/E3SM-Project/zppy/commits/main/
# Test PR 3: #852
git checkout partial-update-expectations
git fetch upstream issue-837-legacy-3.2.0
git rebase upstream/issue-837-legacy-3.2.0
git push -f upstream partial-update-expectations
git log --oneline | head -n 52
# 48ffaef5 Ensure test output exists
# ...
# 54f6f34d Support partial updates of weekly/campaign expected test results
# 87ef5ca1 Add livvkit to legacy v3.2.0 testing # From #886
# ...
# 92ee104b Copy comprehensive_v3 cfg from v3.2.0
# 58fdce18 Remove unnecessary function Copilot added # From #871
# ...
# 5f19eb9b Automate image checker, add per-task env descriptions, and auto-generate test report
# e45286e8 Add snow storage, snow cover, river discharge analysis (#888) # From `main` branch
# 83cd30e1 E3SM-Unified 1.14 testing: faster diurnal climo on Perlmutter (#890)
# cc780ec6 Bump to 3.3.0rc1 (#882)
# Good, we now have commits from all 3 testing PRs, rebased off the latest `main`.Set up the test script# Now, copy the test script and cfg from the zppy repo into the directory
# that you'll be running the test script from.
mkdir -p ~/ez/zppy_main_branch_tests/test_20261016_run1
cd ~/ez/zppy_main_branch_tests/test_20261016_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Now, edit the test cfg as needed
emacs zppy_test.cfgSet up the test cfgls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Oct 9 21:04 mpas_analysis
# Oct 9 21:02 global_time_series
# Oct 9 21:02 ilamb
# Oct 9 21:00 e3sm_diags
# Oct 9 20:58 livvkit
# Oct 9 20:58 pcmdi_diagsshows the date results were promoted to be official expected results. Expected results for all tasks were promoted on 10/09, as noted here. They are the results from the E3SM-Unified 1.14.0 rc3 test. We should keep that in mind when examining the diffs of this test run, since results can differ quite a bit between dev and unified environments, due to different solvings for the dependencies. Run the test script# NOTE: This part is not listed in the docs, as the script should check this.
# To be on the safer side though, it's a good idea to check that these directories
# don't have uncommitted changes.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
cd ~/ez/zppy_main_branch_tests/test_20261016_run1
screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
cd ~/ez/zppy_main_branch_tests/test_20261016_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# [2026-10-16 13:31:10] Starting zppy integration test automation
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_run1.logD. Review the output# CTRL C # Exit tail
screen -R
# [2026-10-16 16:35:53] ✓ Markdown report: /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261016_run1/test_report_20261016_run1.md
# real 184m42.292s
# user 10m44.037s
# sys 2m23.244s
exit # Exit screen
ReportDetails20261016 zppy testSee automated testing docs page for info on setup. Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for
Environment descriptionsAn
Each task's
Automated test script results
TODO: one or more steps above did not pass -- review the full log captured from this script's stdout (for example, Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by taskNo failing image-check tests. Results analysisTODO: fill in analysis of any failures above (expected vs. unexpected, whether expected results should be updated, etc.). Results analysiscd /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261016_run1
# The report showed:
# * `test_bash_generation.py: failed`
# * `test_campaign.py: failed`
# * `test_defaults.py: failed`
# * `test_bundles.py: failed`
grep -n "FAILED tests/integration/test_" integration_test_run1.log
# 8347:FAILED tests/integration/test_bash_generation.py::test_bash_generation
# 9551:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere
# 9552:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere_override
# 9553:FAILED tests/integration/test_campaign.py::test_campaign_high_res_v1
# 9554:FAILED tests/integration/test_campaign.py::test_campaign_none - assert 256 == 0
# 9555:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle
# 9556:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle_override
# 9712:FAILED tests/integration/test_defaults.py::test_defaults - assert 256 == 0
# 9760:FAILED tests/integration/test_bundles.py::test_bundles_bash_file_content
# Yes, this matches what the report showed.
Now, for the image checker test. cd /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261016_run1
emacs image_checker_20261016_run1.o1310142It looks like the replacement of legacy cfgs (#886) crashed the whole unit test with Fixing the testsTest fixesLet's apply the test fixes. cd ~/ez/zppy
git status
# On branch partial-update-expectations
# nothing to commit, working tree clean
git checkout issue-869-improve-test-automation
# Apply changes to tests/integration/test_bundles.py
git add -A
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr871-improve-test
conda activate zppy-pr871-improve-test
pre-commit run --all-files
python -m pip install .
git commit -m "Fix path error in test_bundles.py"
git push upstream issue-869-improve-test-automation
git checkout issue-837-legacy-3.2.0
git fetch upstream issue-869-improve-test-automation
git rebase upstream/issue-869-improve-test-automation
# Apply changes to tests/integration/image_checker.py and tests/integration/test_images.py
git add -A
pre-commit run --all-files
git commit
git push -f upstream issue-837-legacy-3.2.0
git checkout partial-update-expectations
git fetch upstream issue-837-legacy-3.2.0
git rebase upstream/issue-837-legacy-3.2.0
git push -f upstream partial-update-expectationsRunning the updated testsRerunning testscd ~/ez/zppy_main_branch_tests/test_20261016_run1
# 1. Get the TAG from the original run
cat ~/.zppy_test_tag
# 20261016_run1
TAG=20261016_run1
# 2. Update the test checkout while keeping the Phase 1 utils.py edit
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20261016_run1
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git fetch upstream partial-update-expectations
git rebase --autostash upstream/partial-update-expectations
# 3. Verify the rebase worked and the Phase 1 edit survived
git log --oneline | head -n 6
# dc8ab27f Testing
# From #852:
# a71f07df Ensure test output exists
# b1bbb531 Add auto-generated update files
# f5f4b267 Change term from diags to tasks
# 92fe6799 Support partial updates of weekly/campaign expected test results
# From # 886:
# 758bcbe5 Handle missing image baselines without crashing image tests
grep -n "unique_id" tests/integration/utils.py # should show zppy_main_branch_test_${TAG}
# 30: "unique_id": "zppy_main_branch_test_20261016_run1",
# 4. Go to the directory where your copied script and cfg live
cd ~/ez/zppy_main_branch_tests/test_20261016_run1
# 5. Optional: keep the old report and image summary for comparison
mkdir -p old_results_${TAG}
mv test_report_${TAG}.md test_images_summary_${TAG}.md old_results_${TAG}/ 2>/dev/null
# 6. Edit the cfg: resume at Phase 3 with the same tag
sed -i 's/^START_PHASE=.*/START_PHASE=3/' zppy_test.cfg
sed -i "s/^EXPLICIT_TAG=.*/EXPLICIT_TAG=\"${TAG}\"/" zppy_test.cfg
grep -n "^START_PHASE\|^EXPLICIT_TAG" zppy_test.cfg # confirm
# 11:START_PHASE=3
# 13:EXPLICIT_TAG="20261016_run1"
# 7. Run it. The image checker submits to SLURM and the script waits for it, so use tmux or screen
screen
cd ~/ez/zppy_main_branch_tests/test_20261016_run1
ulimit -s unlimited
cat zppy_test.cfg # Good, has changes
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_rerun_start_phase_3.log
# [2026-10-16 18:22:27] Starting zppy integration test automation
# [2026-10-16 18:22:27] Start phase: 3
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_rerun_start_phase_3.log
# Picking up Monday 2026-10-19
cd ~/ez/zppy_main_branch_tests/test_20261016_run1
tail integration_test_rerun_start_phase_3.logscreen -ls
# No screen at all; must have been terminated by LCRC?
Report (updated)Details20261016 zppy testSee automated testing docs page for info on setup. Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for
Environment descriptionsAn
Each task's
Automated test script results
TODO: one or more steps above did not pass -- review the full log captured from this script's stdout (for example, Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by task
Results analysisNon-image-checker testscd /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261016_run1
# The report showed:
# * Output directory status files: `passed`
# * Integration tests:
# * `test_last_year.py: passed`
# * `test_bash_generation.py: failed`
# * `test_campaign.py: failed`
# * `test_defaults.py: failed`
# * `test_bundles.py: passed`
grep -n "FAILED tests/integration/test_" integration_test_rerun_start_phase_3.log
# 507:FAILED tests/integration/test_bash_generation.py::test_bash_generation - asse...
# 1711:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere - assert ...
# 1712:FAILED tests/integration/test_campaign.py::test_campaign_cryosphere_override
# 1713:FAILED tests/integration/test_campaign.py::test_campaign_high_res_v1 - assert...
# 1714:FAILED tests/integration/test_campaign.py::test_campaign_none - assert 256 == 0
# 1715:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle - assert...
# 1716:FAILED tests/integration/test_campaign.py::test_campaign_water_cycle_override
# 1872:FAILED tests/integration/test_defaults.py::test_defaults - assert 256 == 0
# Good, tests/integration/test_bundles.py::test_bundles_bash_file_content has been fixed,
# as expected.
# Other failures were earlier determined to be acceptable/expected.Image checker tests
|
|
thanks @forsyth2 the e3sm_diags diffs look as expected. And it looks like we are ready for the e3sm_diags and zppy rcs. |
|
With all failures accounted for, I've updated the expected results baseline. That means there's now a legacy v3.2.0 baseline as well now. |
2026-10-20 E3SM Unified 1.14.0rc4 Chrysalis testThis test will use E3SM Unified 1.14.0rc4. We'll also use the currently unmerged test improvements of the stack #887:
None of these PRs touch user-facing code, so it's still E3SM Unified 1.14.0rc4 that we're testing. Set up and run the testSet up the zppy branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
# We need to rebase the testing stack to make sure we're testing the latest `main`
# Test PR 1: #871
git checkout issue-869-improve-test-automation
git fetch upstream main
git rebase upstream/main
# https://github.com/E3SM-Project/zppy/pull/871/commits shows 40 commits
# Should be based on: 284eac4f Bump to 3.3.0rc2 (#891)
git log --oneline | head -n 41
# 5821e229 Fix path error in test_bundles.py
# ...
# 71e3896f Automate image checker, add per-task env descriptions, and auto-generate test report
# 284eac4f Bump to 3.3.0rc2 (#891)
git push -f upstream issue-869-improve-test-automation
# Test PR 2: #886
git checkout issue-837-legacy-3.2.0
git fetch upstream issue-869-improve-test-automation
git rebase upstream/issue-869-improve-test-automation
git push -f upstream issue-837-legacy-3.2.0
# https://github.com/E3SM-Project/zppy/pull/886/commits shows 7 commits
# Should be based on: Fix path error in test_bundles.py
git log --oneline | head -n 8
# b20c45f3 Handle missing image baselines without crashing image tests
# ...
# e72689e5 Copy comprehensive_v3 cfg from v3.2.0
# 5821e229 Fix path error in test_bundles.py
# Test PR 3: #852
git checkout partial-update-expectations
git fetch upstream issue-837-legacy-3.2.0
git rebase upstream/issue-837-legacy-3.2.0
git push -f upstream partial-update-expectations
# https://github.com/E3SM-Project/zppy/pull/852/commits shows 4 commits
# Should be based on: Handle missing image baselines without crashing image tests
git log --oneline | head -n 5
# a48dd33e Ensure test output exists
# ...
# 2b2f5c72 Support partial updates of weekly/campaign expected test results
# b20c45f3 Handle missing image baselines without crashing image tests
# Good, we now have commits from all 3 testing PRs, rebased off the latest `main`.Set up the test script# Now, copy the test script and cfg from the zppy repo into the directory
# that you'll be running the test script from.
mkdir -p ~/ez/zppy_main_branch_tests/test_20261020_run1
cd ~/ez/zppy_main_branch_tests/test_20261020_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Now, edit the test cfg as needed
emacs zppy_test.cfgSet up the test cfgls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
total 40
# Oct 19 16:09 global_time_series
# Oct 19 16:09 pcmdi_diags
# Oct 19 16:08 livvkit
# Oct 19 16:08 mpas_analysis
# Oct 19 16:05 ilamb
# Oct 19 16:03 e3sm_diagsshows the date results were promoted to be official expected results. This matches the 10/19 promotion of the results from the 10/16 run. Run the test script# NOTE: This part is not listed in the docs, as the script should check this.
# To be on the safer side though, it's a good idea to check that these directories
# don't have uncommitted changes.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
cd ~/ez/zppy_main_branch_tests/test_20261020_run1
screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
cd ~/ez/zppy_main_branch_tests/test_20261020_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# [2026-10-20 11:18:19] Starting zppy integration test automation
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_run1.logD. Review the output# CTRL C # Exit tail
screen -R
# [2026-10-20 13:09:57] ✓ Markdown report: /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261020_run1/test_report_20261020_run1.md
# real 111m37.461s
# user 1m35.509s
# sys 2m15.038s
exit # Exit screen
ReportDetails20261020 zppy testSee automated testing docs page for info on setup. SLURM jobs cancelled (DependencyNeverSatisfied)
Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for Note: this run tested the released zppy from E3SM-Unified, not the tip of the zppy baseline branch, so the
Environment descriptionsAn
Each task's
Automated test script results
TODO: one or more steps above did not pass -- review the full log captured from this script's stdout (for example, Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by task
Results analysisThe non-image checker testsThe report above noted:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts
grep "ERROR" *.status
# mpas_analysis_reference_ts_1985-1989_climo_1985-1989.status:ERROR (1)
grep -in error mpas_analysis_reference_ts_1985-1989_climo_1985-1989.o1312433
# 96:ERROR in task mpasClimatologyOceanAvg. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/logs/mpasClimatologyOceanAvg.log for details
# 274:There were errors in task mpasClimatologyOceanAvg
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/logs/mpasClimatologyOceanAvg.log
# [chr-0152:3056295:0:3056295] Caught signal 7 (Bus error: <unknown si_code>)
# 12 occurrences: BFD: Dwarf Error: Can't find .debug_ranges section.
# /lcrc/soft/climate/e3sm-unified/e3smu_1_14_0rc4/chrysalis/spack/var/spack/environments/e3sm_unified_gnu_openmpi/.spack-env/view/bin/ncclimo: line 4119: 3056295 Bus error ncra ...
# raise subprocess.CalledProcessError(process.returncode,
# subprocess.CalledProcessError: Command 'ncclimo ...This looks like it might be an LCRC issue rather than a MPAS-Analysis issue. Image checker testsglobal_time_seriesShouldn't cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts
grep . global_time_series*.status
# global_time_series_all_lnd_var_viewer_1985-1995.status:OK
# global_time_series_classic_original_8_no_ocn_1985-1995.status:OK
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 1312439
# global_time_series_classic_pdf_component_1985-1995.status:OK
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 1312440
# global_time_series_viewer_both_1985-1995.status:WAITING 1312436
# global_time_series_viewer_component_1985-1995.status:OK
# global_time_series_viewer_original_1985-1995.status:WAITING 1312437
grep "OK" global_time_series*.status
# global_time_series_all_lnd_var_viewer_1985-1995.status:OK
# global_time_series_classic_original_8_no_ocn_1985-1995.status:OK
# global_time_series_classic_pdf_component_1985-1995.status:OK
# global_time_series_viewer_component_1985-1995.status:OKWe can see that anything not involving ocean data was able to run. That's why we have some results. It seems likely then that the 183 missing images required ocean data. ilambThe diffs here appear minor as far as I can tell. livvkitThese diffs also appear to be minor. mpas_analysisThe non-legacy tests show identical diff numbers to the 10/16 run. There, I noted:
So since we updated the expected results after that run, we have simply reversed what we saw there. That is, the expected results can be updated again. As for |
|
@chengzhuzhang I think the @xylar MPAS-Analysis failed on the legacy v3.2.0 cfg with a cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts
grep "ERROR" *.status
# mpas_analysis_reference_ts_1985-1989_climo_1985-1989.status:ERROR (1)
grep -in error mpas_analysis_reference_ts_1985-1989_climo_1985-1989.o1312433
# 96:ERROR in task mpasClimatologyOceanAvg. See log file /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/logs/mpasClimatologyOceanAvg.log for details
# 274:There were errors in task mpasClimatologyOceanAvg
grep -in error /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.2.0_comprehensive_v3_output/zppy_main_branch_test_20261020_run1/v3.LR.historical_0051/post/scripts/../analysis/mpas_analysis/mvo/ts_1985-1989_climo_1985-1989/logs/mpasClimatologyOceanAvg.log
# [chr-0152:3056295:0:3056295] Caught signal 7 (Bus error: <unknown si_code>)
# 12 occurrences: BFD: Dwarf Error: Can't find .debug_ranges section.
# /lcrc/soft/climate/e3sm-unified/e3smu_1_14_0rc4/chrysalis/spack/var/spack/environments/e3sm_unified_gnu_openmpi/.spack-env/view/bin/ncclimo: line 4119: 3056295 Bus error ncra ...
# raise subprocess.CalledProcessError(process.returncode,
# subprocess.CalledProcessError: Command 'ncclimo ...I can try rerunning just that cfg and see if it happens again. |
|
@forsyth2, I've not seen that before that I remember. Let's assume it's an odd, one-time problem and just try running again. |
2026-10-09 E3SM Unified 1.14.0rc4 Chrysalis test (legacy v3.2.0)This is a retest of just the legacy v3.2.0 cfg, after the crash noted here This test will use E3SM Unified 1.14.0rc4. We'll also use the currently unmerged test improvements of the stack #887:
None of these PRs touch user-facing code, so it's still E3SM Unified 1.14.0rc4 that we're testing. Set up and run the testSet up the zppy branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
# We need to rebase the testing stack to make sure we're testing the latest `main`
# `partial-update-expectations` should still be up-to-date
git log --oneline | head -n 52 # 40+7+4=51 commits from #871, #886, #852
# a48dd33e Ensure test output exists
# ...
# 71e3896f Automate image checker, add per-task env descriptions, and auto-generate test report
# 284eac4f Bump to 3.3.0rc2 (#891)Set up the test script# Now, copy the test script and cfg from the zppy repo into the directory
# that you'll be running the test script from.
mkdir -p ~/ez/zppy_main_branch_tests/test_20261020_run2
cd ~/ez/zppy_main_branch_tests/test_20261020_run2
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .
# Now, edit the test cfg as needed
emacs zppy_test.cfgSet up the test cfgRun the test script# NOTE: This part is not listed in the docs, as the script should check this.
# To be on the safer side though, it's a good idea to check that these directories
# don't have uncommitted changes.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
cd ~/ez/zppy_main_branch_tests/test_20261020_run2
screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
cd ~/ez/zppy_main_branch_tests/test_20261020_run2
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run2.log
# [2026-10-20 14:22:32] Starting zppy integration test automation
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_run2.logD. Review the output# CTRL C # Exit tail
screen -R
# [2026-10-20 15:35:22] ✓ Markdown report: /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261020_run2/test_report_20261020_run2.md
# real 72m49.996s
# user 1m18.070s
# sys 1m53.917s
exit # Exit screen
ReportDetails20261020 zppy testSee automated testing docs page for info on setup. Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for Note: this run tested the released zppy from E3SM-Unified, not the tip of the zppy baseline branch, so the
Environment descriptionsAn
Each task's
Automated test script results
Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by task
Results analysisThe Bus error doesn't occur this time and the 3 diffs are pretty minor. So, between this run and run1, I think we can say E3SM Unified 1.14.0rc4 is working. |
|
The diffs are final-digit roundoff as we've seen previously. I don't see any difference at all in the "cosmetic-only" sample. Thanks @forsyth2! |
2026-10-30 Pre-E3SM Unified 1.14.0rc5 Chrysalis testThis test will test dev environments rather than an E3SM Unified RC. That is because we are trying to test bug fixes and other new development before making package RCs to be included in E3SM Unified 1.14.0rc5. We'll use the currently unmerged test improvements of the stack #887:
Note that none of those PRs touch user-facing code, so from the user perspective, this is still a Set up and run the testSet up the zppy branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
# We need to rebase the testing stack to make sure we're testing the latest `main`
# Test PR 1: #871
git checkout issue-869-improve-test-automation
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 41
# 826970ce Fix path error in test_bundles.py
# ...
# f4b66bc0 Automate image checker, add per-task env descriptions, and auto-generate test report
# 5d7d739b global_time_series: plot a final, shorter ts chunk when [ts] produces it (#900)
# Good, matches the 40 commits of https://github.com/E3SM-Project/zppy/pull/871/commits
# Good, based on https://github.com/E3SM-Project/zppy/commits/main/
git push -f upstream issue-869-improve-test-automation
# Confirmed no changes affect src code: https://github.com/E3SM-Project/zppy/pull/871/changes
# Test PR 2: #886
# I'm not seeing a "rebase stack" button on https://github.com/E3SM-Project/zppy/pull/886,
# so we'll just do it on the CLI.
git checkout issue-837-legacy-3.2.0
git fetch upstream issue-869-improve-test-automation
git rebase upstream/issue-869-improve-test-automation
git push -f upstream issue-837-legacy-3.2.0
git log --oneline | head -n 8
# 4472aeeb Handle missing image baselines without crashing image tests
# ...
# 99ddb5dd Copy comprehensive_v3 cfg from v3.2.0
# 826970ce Fix path error in test_bundles.py # From #871
# Good, matches the 7 commits of https://github.com/E3SM-Project/zppy/pull/886/commits
# Confirmed no changes affect src code: https://github.com/E3SM-Project/zppy/pull/886/changes
# Test PR 3: #852
git checkout partial-update-expectations
git fetch upstream issue-837-legacy-3.2.0
git rebase upstream/issue-837-legacy-3.2.0
git push -f upstream partial-update-expectations
git log --oneline | head -n 5
# 5c768fde Ensure test output exists
# ...
# 4e9b6276 Support partial updates of weekly/campaign expected test results
# 4472aeeb Handle missing image baselines without crashing image tests
# Good, matches the 4 commits of https://github.com/E3SM-Project/zppy/pull/852/commits
# Confirmed no changes affect src code: https://github.com/E3SM-Project/zppy/pull/852/changes
# Good, we now have commits from all 3 testing PRs, rebased off the latest `main`.Set up the test script# Now, copy the test script and cfg from the zppy repo into the directory
# that you'll be running the test script from.
mkdir -p ~/ez/zppy_main_branch_tests/test_20261030_run1
cd ~/ez/zppy_main_branch_tests/test_20261030_run1
cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash .
cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg .Set up the test cfgls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Oct 19 16:09 global_time_series
# Oct 19 16:09 pcmdi_diags
# Oct 19 16:08 livvkit
# Oct 19 16:08 mpas_analysis
# Oct 19 16:06 image_check_failures_comprehensive_v3_try2
# Oct 19 16:05 ilamb
# Oct 19 16:03 e3sm_diagsshows the date results were promoted to be official expected results. Expected results for all tasks were promoted on 10/19, as noted here. They are the results from the 2026-10-16 Pre-E3SM Unified 1.14.0rc4 Chrysalis test. We should keep that in mind when examining the diffs of this test run, since results can differ quite a bit between dev and unified environments, due to different solvings for the dependencies. Re: image_check_failures_comprehensive_v3_try2 -- this has been addressed here. It appears safe to go forward with this test; those files are just clutter that the test will ignore. # On branch: partial-update-expectations
git log --oneline | head -n 6
# 3fa4066b Fix weekly updater: prune stale legacy files, exclude image_check_failures
# 5c768fde Ensure test output exists
# ...
# 4e9b6276 Support partial updates of weekly/campaign expected test results
# 4472aeeb Handle missing image baselines without crashing image tests
# Now, edit the test cfg as needed
emacs zppy_test.cfgRun the test script# NOTE: This part is not listed in the docs, as the script should check this.
# To be on the safer side though, it's a good idea to check that these directories
# don't have uncommitted changes.
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy && git status
# Has uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# nothing to commit, working tree clean
cd ~/ez/zppy_main_branch_tests/test_20261030_run1
screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running.
ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen`
cd ~/ez/zppy_main_branch_tests/test_20261030_run1
cat zppy_test.cfg # Make sure changes are there
time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_run1.log
# [2026-10-30 13:42:18] Starting zppy integration test automation
# Ctrl-A D to detach from screen
screen -ls # See what screen sessions you have
tail -f integration_test_run1.logD. Review the output# CTRL C # Exit tail
screen -R
# [2026-10-30 15:57:45] ✓ Markdown report: /home/ac.forsyth2/ez/zppy_main_branch_tests/test_20261030_run1/test_report_20261030_run1.md
# real 135m26.970s
# user 11m24.307s
# sys 2m24.152s
exit # Exit screen
ReportDetails20261030 zppy testSee automated testing docs page for info on setup. SLURM jobs cancelled (DependencyNeverSatisfied)
Determine what the current expected results arePromotion date for each cfg/task under
Review changes since expected results were updatedCommits merged on each repo's expected-results baseline branch (see the "Branch tested" column when it differs from the branch this run actually tested) since that dependency's expected results were actually produced (for
Environment descriptionsAn
Each task's
Automated test script results
TODO: one or more steps above did not pass -- review the full log captured from this script's stdout (for example, Run Python testsThe image checker (
Complete summary table
Summary table -- only failing image-check tests, sorted by task
Results analysisError 1: mpas_analaysis segfault and subsequent dependency failuresDebuggingThe above report noted:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20261030_run1/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# global_time_series_1980-1990.status:WAITING 1318556
# mpas_analysis_ts_1980-1984_climo_1980-1984.status:RUNNING 1318554
# mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 1318555
grep -in error mpas_analysis_ts_1980-1984_climo_1980-1984.o1318554
# 77:srun: error: chr-0100: task 0: Segmentation fault
# 79:slurmstepd: error: *** STEP 1318554.0 ON chr-0100 CANCELLED AT 2026-10-30T14:54:29 DUE TO TIME LIMIT ***
# 80:slurmstepd: error: *** JOB 1318554 ON chr-0100 CANCELLED AT 2026-10-30T14:54:29 DUE TO TIME LIMIT ***It appears I missed the crucial
What changes are we testing?
Considering zi-#66 does in fact update ocean-relevant FIX: Add Error 2: failing test_bash_generation.pyThe above report also noted:
cd ~/ez/zppy_main_branch_tests/test_20261030_run1/
grep -in test_bash_generation.py integration_test_run1.log
# 8368:[2026-10-30 15:45:31] Running test_bash_generation.py...
# 8376:tests/integration/test_bash_generation.py F [100%]
# 8396:tests/integration/test_bash_generation.py:11: AssertionError
# 8479:FAILED tests/integration/test_bash_generation.py::test_bash_generation
# 8481:[2026-10-30 15:45:32] ⚠ test_bash_generation.py had failuresThere's just an ordering difference between: Specifically:
From Claude
Fix: None needed, we'll eventually update the expected results Error 3: image check failuresAs noted in "Error 1" above, the 12 missing images are due to a missed test step that caused MPAS-Analysis (for the Looking at one set as an example from
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20261030_run1/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# NO ERRORS
ls mpas*.o*
# mpas_analysis_mvm_ts_1985-1995_climo_1990-1995_vs_ref_ts_1985-1989_climo_1985-1989.o1318599
# mpas_analysis_reference_ts_1985-1989_climo_1985-1989.o1318597
# mpas_analysis_test_ts_1985-1995_climo_1990-1995.o1318598
grep -in error mpas*.o*
# NO MATCHES
ls global_time_series*.o*
# global_time_series_all_lnd_var_viewer_1985-1995.o1318606 global_time_series_classic_pdf_original_1985-1995.o1318604
# global_time_series_classic_original_8_no_ocn_1985-1995.o1318607 global_time_series_viewer_both_1985-1995.o1318600
# global_time_series_classic_pdf_both_1985-1995.o1318603 global_time_series_viewer_component_1985-1995.o1318602
# global_time_series_classic_pdf_component_1985-1995.o1318605 global_time_series_viewer_original_1985-1995.o1318601
grep -in error global_time_series*.o*
# Many errors
ls ../ocn/glb/ts/monthly/*/ # Confirmed present: mpaso.glb.199501-199512.nc
# mocTimeSeries_1985-1995.nc mpaso.glb.198501-198912.nc mpaso.glb.199001-199412.nc mpaso.glb.199501-199512.nc
ls atm/glb/ts/monthly/ # Confirmed present: 5yr only, no 1yr
# 5yrFix: This does appear to be related to the tail year chunking PRs. It seems like we need a fix PR. |
|
oh, no..I will look into it. Though, it might also indicate a configuration problem from the weekly config, I think it meant to use 1985-1994 for ten years, but 1985-1995 is requested. |
Addressed here |
It could be that, but the years are hard-coded in the template cfgs (e.g., https://github.com/E3SM-Project/zppy/blob/main/tests/integration/template_weekly_comprehensive_v3.cfg), not configured, so there'd need to be a commit to address that. |
|
@forsyth2 Fix for the missing "Change in ocean heat content" / "Change in sea level" plots (Error 3): E3SM-Project/zppy-interfaces#68 Cause: Fix: trim ohc/volume to the years on the atm x-axis. With the current cfg, the ocean plots come back but cover 1985-1994. Adding Note: I don't think this fix is strictly needed if the cfg is correct. When Reminder: we should fix the test cfgs to request the intended time period. Tested with Claude Code |
|
Thanks @chengzhuzhang!
Well I'd still say E3SM-Project/zppy-interfaces#68 is necessary then; without we lose backwards-compatibility. A cfg that was set up in the old manner would now fail -- as this test failure showed. I can definitely update the latest cfg ( |
Uh oh!
There was an error while loading. Please reload this page.
This testing log is specifically for testing release candidates of E3SM Unified 1.14.0. (#845 was for testing active development leading up to this release).
All reactions