Upgrade to spack-stack version 1.6.0 #856
Conversation
WenMeng-NOAA
requested review from
FernandoAndrade-NOAA and
AlexanderRichert-NOAA
|
@AlexanderRichert-NOAA Could you please review the changes of ci/spack.yaml? |
|
@FernandoAndrade-NOAA Please review and test the changes of modulefiles for R&D platforms. |
The changes in modulefiles look good to me on initial review. I can verify RTs on Hera and Orion. Hercules I can reattempt a full rebuild of baselines with 1.6.0, perhaps that may resolve previous issues encountered. Jet I can test UPP build. Thanks for working on this, I'll get started on updating and running job card scripts. |
|
@DavidHuber-NOAA please go ahead and resync your branch with develop. Could you provide a location for your RT results on Hera? |
@FernandoAndrade-NOAA There will be updates of job cards in the UPP RTs on Hera and Orion with this PR. Could you start with one test (any model) to evaluate what efforts are needed? |
|
@FernandoAndrade-NOAA The branch has been updated. I ran the regression tests here: |
|
@AlexanderRichert-NOAA Good catch, I will try reverting that. For context, I had had problems with the concretizer flagging |
Sure thing, I can start with the fv3hafs case if that's ok with you? I can compare between machines and then to David's results with my Hera run. Are you expecting any additional efforts beyond path updates resulting from the library updates? |
I believe I chose one of the very few tests that would not change with this update after comparing my results with David's for the hafs test case, I'll run for rtma and fv3r instead and see if there's any hiccups in testing beyond the expected 1.6.0 changes. I did not see any issues arise during the hafs test case @WenMeng-NOAA FYI |
|
@FernandoAndrade-NOAA That's great. Can you share with your updates in job cards? |
Sure thing they're available here: The primary updates were to the spack-stack path and module versions loaded as the job cards currently have their own explicit load commands. I will run the full RTs to compare all results and if that looks good I'll move onto making the appropriate changes on Orion. I will rerun full RTs on Hercules as well on the chance that this update resolves previous issues with testing on that machine. |
|
@WenMeng-NOAA I did not observe any changes in the full RT run on Hera if you could please verify there is no issue with my run and modifications to my job cards within the work directory at @DavidHuber-NOAA 's initial RT run without updates to the job cards would have run the tests with crtm 2.4.0 and prod_util 1.2.2 |
|
@FernandoAndrade-NOAA So from your testing, there is no baseline change, right? That's great. |
Right, I'm not seeing any errors either in the Hera run. If everything looks good to you in that Hera rundir and work directory, I'll go ahead and move onto making and testing the appropriate changes on Orion and Hercules as well. Thanks! |
|
@FernandoAndrade-NOAA Please work on testing on Orion. Thanks! |
|
@DavidHuber-NOAA Can you sync your branch with the UPP develop? |
|
@WenMeng-NOAA Done. A test build on Hercules was successful. |
WenMeng-NOAA
added
enhancement
|
The UPP RTs were completed on WCOSS2 without changed results. |
|
@WenMeng-NOAA I've rerun Hera after the sync, no changes in results. The build on Jet looks good with no errors. Hercules baselines will be fully recreated. I will create an issue as well for transitioning Gaea's configuration to the new F5 file system as the F2 file system will be retired soon and has already been disconnected from C5 nodes. The ufs-wm side is currently working on getting that back up and running. I'm seeing changes in the following test cases on Orion: rtma NATLEV similarly: gfs t00z.special.grb2f006: |
@FernandoAndrade-NOAA These changed results in simulated satellite products might come from fix files updates in crtm/2.4.0.1 on Orion. I will conduct the testing from my end. |
|
@FernandoAndrade-NOAA My test results on Orion are consistent with yours. All changes in simulated satellite products come from upgrade of crtm on Orion. I think this PR is ready for merging. |
WenMeng-NOAA
added
the
Baseline Change
Thank you for confirming! I will update Orion baselines for the affected tests tomorrow after the merge. I will also replace the baselines for Hercules to potentially resolve the Hercules fatal errors. Would you happen to know why the upgrade in crtm versions did not affect results on WCOSS2 and Hera while changing values on Orion? Is the crtm configuration considerably different on that system? |
|
I will also create an issue for the appropriate Gaea C5/F5 updates once everything is verified over on the weather model side. |
@FernandoAndrade-NOAA The changed results in Orion come from crtm fix files. We occurred the similar situation when switching hpc-stack to spack-stack. |
|
This PR is ready for merging. The new baseline is needed on Orion. |
This upgrades the libraries used by the UPP to those built with the recent release of spack-stack version 1.6.0. The libraries upgraded include
netCDF-Fortran v4.6.0 -> v4.6.1
CRTM v2.4.0 -> v2.4.0.1
sp v2.3.3 -> v2.5.0
The regression tests were run on Hera, but produced different results. Per a discussion with @WenMeng-NOAA, changes will need to be made to the job cards. Thus I am submitting this as a draft to allow that to happen.