Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update externals to what is expected to be in cesm2_3_beta17 fix some conda issues... #2385

Merged
merged 13 commits into from
Mar 5, 2024

Conversation

ekluzek
Copy link
Collaborator

@ekluzek ekluzek commented Feb 22, 2024

Description of changes

Update externals to what's currently planned for cesm2_3_beta17.

Also update manage_externals. This is only important if you add CAM to your externals.
This was important on the Dust PR, so this will allow that PR to show less differences
to master when this comes in.

Specific notes

Contributors other than yourself, if any:

CTSM Issues Fixed (include github issue #):
Fixes #2351
Fixes #2380
Fixes #2331
Fixes #1910
Fixes #1658

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No

Testing performed, if any: standard
ERI tests currently fail
This one might have trouble: ERP_P256x2_Ld30.f45_f45_mg37.I2000Clm51FatesRs.derecho_intel.clm-mimicsFatesCold
This one shows different answers so I'll fix that: SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.derecho_intel.clm-USUMB_nuop

0f884bfec Merge pull request ESCOMP#205 from jedwards4b/sunset_svn_git_access
82a5edf79 merge in billsacks:svn_testing_no_github
17532c160 Use a local svn repo for testing
9c904341a different method to determine if in tests
539952ebd remove debug print statement
cc5434fa7 fix submodule testing
1d7f28840 remove broken tests
04e94a519 provide a meaningful error message
38bcc0a8c Merge pull request ESCOMP#201 from jedwards4b/partial_match
b4466a5aa remove debug print statement
c3cf3ec35 fix issue with partial branch match

git-subtree-dir: manage_externals
git-subtree-split: 0f884bfec8e43d0c02261de858d6ec3f6d855e51
…o needed to update numpy version for it to work
… are main and develop, with the new externals the testlist wasn't working, it failed trying to setup cases
@ekluzek ekluzek added the enhancement new capability or improved behavior of existing capability label Feb 22, 2024
@ekluzek ekluzek self-assigned this Feb 22, 2024
@ekluzek
Copy link
Collaborator Author

ekluzek commented Feb 22, 2024

@samsrabin I added you as reviewer for this. The main thing to look at is py_env_create.

@ekluzek
Copy link
Collaborator Author

ekluzek commented Feb 22, 2024

@glemieux can you try using this branch to create a conda environment with py_env_create to test if #2331 works for you for the LUH tools?

@ekluzek
Copy link
Collaborator Author

ekluzek commented Feb 22, 2024

The change in answers is just because the USUMB test failed previously , and the FatesMimics test is expected to fail. So it's just the ERI tests that I need to figure out.

@ekluzek
Copy link
Collaborator Author

ekluzek commented Feb 23, 2024

I now have a cime tag that fixes the ERI tests, so they are fine. So aux_clm should be good, but I need to rerun as I just updated.

The issue now is that the python system tests are failing for the lilac tests. With a message like this for example...

Stdout:
ERROR while running:
/glade/work/erik/ctsm_worktrees/external_updates/cime/scripts/create_newcase --output-root /glade/derecho/scratch/erik/tmp/tmp_c8v_9kp/ctsm_build --case /glade/derecho/scratch/erik/tmp/tmp_c8v_9kp/ctsm_build/case --compset I2000Ctsm50NwpSpAsRs --res f10_f10_mg37 --compiler gnu --driver nuopc --project UNSET --run-unsupported --machine ctsm-build --extra-machines-dir /glade/derecho/scratch/erik/tmp/tmp_c8v_9kp/ctsm_build/machine_configuration

Compset longname is 2000_SATM_CLM50%NWP-SP_SICE_SOCN_SROF_SGLC_SWAV
Compset specification file is /glade/work/erik/ctsm_worktrees/external_updates/cime_config/config_compsets.xml
Automatically adding SESP to compset
Compset forcing is 1972-2004
ATM component is Stub atm component
LND component is clm5.0:NWP configuration with satellite phenology:
ICE component is Stub ice component
OCN component is Stub ocn component
ROF component is Stub river component
GLC component is Stub glacier (land ice) component
WAV component is Stub wave component
ESP component is Stub external system processing (ESP) component
Pes     specification file is /glade/work/erik/ctsm_worktrees/external_updates/cime_config/config_pes.xml
Compset specific settings: name is RUN_STARTDATE and value is 2000-01-01
Traceback (most recent call last):
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/scripts/create_newcase", line 18, in <module>
    _main_func()
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/CIME/scripts/create_newcase.py", line 469, in _main_func
    gpu_offload=gpu_offload,
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/CIME/case/case.py", line 2505, in create
    gpu_offload=gpu_offload,
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/CIME/case/case.py", line 1350, in configure
    probed_machine = machobj.probe_machine_name()
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/CIME/XML/machines.py", line 183, in probe_machine_name
    machine = self._probe_machine_name_one_guess(nametomatch)
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/CIME/XML/machines.py", line 213, in _probe_machine_name_one_guess
    return self._probe_machine_name_one_guess_v3(nametomatch)
  File "/glade/work/erik/ctsm_worktrees/external_updates/cime/CIME/XML/machines.py", line 306, in _probe_machine_name_one_guess_v3
    return machine
UnboundLocalError: local variable 'machine' referenced before assignment

Copy link
Collaborator

@samsrabin samsrabin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that py_env_create works on both Derecho and Izumi, thanks! I saw the same "local variable 'machine' referenced before assignment" error when I tried the Python system tests on Derecho, but interestingly I didn't see that on Izumi. Maybe a clue?

cime_config/testdefs/testlist_clm.xml Show resolved Hide resolved
@ekluzek
Copy link
Collaborator Author

ekluzek commented Mar 5, 2024

Testing is fine. The test

SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.derecho_intel.clm-USUMB_nuopc

is labeled as different answers, but that's just because the baselines don't have all the files. So I consider this fine.

@ekluzek ekluzek merged commit 851cc2e into ESCOMP:b4b-dev Mar 5, 2024
2 checks passed
@ekluzek ekluzek deleted the cesm23beta17externlas branch March 5, 2024 23:36
@ekluzek ekluzek mentioned this pull request Mar 9, 2024
@ekluzek
Copy link
Collaborator Author

ekluzek commented Mar 9, 2024

I thought I had fixed the lilac issue, but in testing for #2416 I see that it's still an issue. So I need to get

make stest

working again here.

@ekluzek
Copy link
Collaborator Author

ekluzek commented Mar 12, 2024

OK, from talking with @billsacks it looks like the problem is this one filed in cime.

ESMCI/cime#4588

And @billsacks has a simple fix for that -- that will likely fix the problem coupled with some other simple changes that we talked about.

@ekluzek
Copy link
Collaborator Author

ekluzek commented Mar 12, 2024

This small change to cime solves the issue!

ESMCI/cime#4598

@ekluzek
Copy link
Collaborator Author

ekluzek commented Mar 12, 2024

This also brings up something that we will need to do sometime, but can get away with for now: #2419

billsacks added a commit to ESMCI/cime that referenced this pull request Mar 12, 2024
Fix undefined machine on cime6.0.217_httpsbranch branch

Initialize machine to None to prevent not initializing it to anything. This is cherry-picked
from #4596

Testing: Run CTSM python testing (which was failing before this change)
./scripts_regression_tests.py (currently running)
Test status: bit-for-bit

Fix for #4588 on the httpsbranch branch.

This fixes issues in CTSM discussed here:

ESCOMP/CTSM#2385 (comment)

User interface changes?: None

Update gh-pages html (Y/N)?: N
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement new capability or improved behavior of existing capability
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants