[WIP] Plumber updates

[TeaganKing]

This will address #248 In order to implement PLUMBER capabilities

[slevis-lmwg]

The changes for PLUMBER appear to parallel the existing code for NEON, so they seem logical and simple to follow.

Before approving, we probably need to run tests. I have never worked on a CDEPS PR, so I'm not familiar with relevant testing, nor with the merging process (though it's likely similar to CTSM's, RTM's, MOSART's, etc.). But also I doubt that I have the necessary permissions to merge a CDEPS PR...

[slevis-lmwg]

From meeting with @ekluzek

• All externals remain "frozen" until ./manage_externals gets removed (which is cesm tag beta18).
• Because this PR changes externals, it needs to wait until after that happens.
• People with permissions to merge this PR are Chris Fisher, Bill Sacks, and Jim Edwards.

[ekluzek]

A note on somethings we need to do before this is asked for merge:

• As above wait until cesm2_3_beta18 freeze is resolved
• Test with it in CTSM and make sure it'll work for PLUMBER2 cases
• Add a aux_cdeps test for PLUMBER2? (Not going to do, eventually should do this both for NEON and PLUMBER2)
• Run the aux_cdeps testlist and make sure it runs the same as a baseline for it (run a baseline first) (this code adds new functionality that shouldn't mess with anything else -- but to make sure it doesn't...)
• Ask a CDEPS maintainer to merge it in (a new tag is created when they do)
• Possibly coordinate this externals update with other external updates for CTSM?

[slevis-lmwg]

Agreed in today's ctsm software meeting:
@TeaganKing will notify @slevis-lmwg when he should run the aux_cdeps test-suite.

[TeaganKing]

Per conversation with Erik, we can remove the files listed in PLUMBER2 user mod directories because these will be implemented in another PR (#277 ); those do not need to be moved to CDEPS.

However, we do need to implement the dtlimit used for these various streams specifically for PLUMBER-- hence the placeholder values that I still need to ensure work properly for changing dtlimit when CLM_USRDAT_NAME is set to PLUMBER.

Variables in those user mod directories that are duplicated in the CDEPS stream can be removed once this PR is merged in.

[TeaganKing]

The dtlimit is updated as expected when running CTSM when I do an xmlchange to set CLM_USRDAT_NAME to PLUMBER. So, @slevis-lmwg , I think we can run the aux_cdeps test-suite. Note that the CTSM changes (ESCOMP/CTSM#2485 and ESCOMP/CTSM#2406) are not yet available (since they're dependent on this CDEPS PR).

[TeaganKing]

This PR introduced CLM_USRDAT_NAME as PLUMBER2 instead of PLUMBER, so I will update that now.

[slevis-lmwg]

@TeaganKing I want to confirm that I understand.
I need to combine the branches from these three PRs:
ESCOMP/CTSM#2485
ESCOMP/CTSM#2406
#262
before I start the aux_cdeps test-suite, right?

Also, a note to myself: The checklist points out that I need to generate a baseline.

[TeaganKing]

@TeaganKing I want to confirm that I understand. I need to combine the branches from these three PRs: ESCOMP/CTSM#2485 ESCOMP/CTSM#2406 #262 before I start the aux_cdeps test-suite, right?

Also, a note to myself: The checklist points out that I need to generate a baseline.

ESCOMP/CTSM#2406 is very much still in progress, and there are going to be a few changes to ESCOMP/CTSM#2485 still as well (I'll do this within the next few days). What exactly is being tested with the aux_cdeps tests? I personally tested this one just by doing an xmlchange to set CLM_USRDAT_NAME to PLUMBER2, building the case, and checking the input files.

[slevis-lmwg]

ESCOMP/CTSM#2406 is very much still in progress, and there are going to be a few changes to ESCOMP/CTSM#2485 still as well (I'll do this within the next few days). What exactly is being tested with the aux_cdeps tests? I personally tested this one just by doing an xmlchange to set CLM_USRDAT_NAME to PLUMBER2, building the case, and checking the input files.

Ok, based on this information, I think I could go ahead and submit aux_cdeps with #262 with ctsm from master (I will try ctsm5.2.007 which is the current latest).

[slevis-lmwg]

I tried and failed to generate a baseline using the latest ctsm paired with cdeps1.0.38, i.e. the same cdeps that I see in @TeaganKing's branch:
./run_sys_tests -s aux_cdeps --skip-compare -g cdeps1.0.38_ctsm5.2.008

I also tried and failed to generate a baseline using the latest ctsm paired with cdeps1.0.34, i.e. the default cdeps for ctsm5.2.008:
./run_sys_tests -s aux_cdeps --skip-compare -g cdeps1.0.34_ctsm5.2.008

The former seems less surprising, if e.g. there are incompatibilities between ctsm5.2.008 and cdeps1.0.38.

The latter though means that I have a problem with aux_cdeps (environment or other?) or that aux_cdeps has a problem (in which case it should fail for others, as well).

@TeaganKing at this point I will need help from @ekluzek with this. I will raise the issue at Monday's stand-up.

[slevis-lmwg]

I encountered the same problem this morning even with aux_clm and ctsm_sci. This helped me realize that the problem may be as simple as setting an account number that hasn't expired. I will try this again today or tomorrow.

UPDATE 1:
I submitted the same two tests. I expect that at least the cdeps1.0.34 should work and generate a baseline.

UPDATE 2:
Worked out the opposite from what I expected:

• ctsm5.2.008 with default cdeps1.034 gave three failures.
• ctsm5.2.008 with cdeps1.0.38 passed. This is convenient because the current version of the branch in this PR modifies cdeps1.0.38.

UPDATE 3:
Submitted aux_cdeps comparing this branch to the baseline (tests_0703-140019de).
./run_sys_tests -s aux_cdeps --skip-generate -c cdeps1.0.38_ctsm5.2.008