Replaces CLUBB with new version of CLUBB (V2)

[singhbalwinder]

The v1 version of CLUBB left pockets of supersaturation for the microphysics. To mitigate this, the
v2 version of CLUBB does CLUBB's saturation adjustment immediately before microphysics when
ipdf_call_placement = ipdf_pre_advance_fields.

CLUBB_v2 includes performance optimizations that may lead to speedups, but also may be vitiated
by a performance degradation in the CLUBB interface that was introduced in order to change the
position of the saturation adjustment.

CLUBB_v2 also adds new options that may be interest when tuning E3SMv2:

• CLUBB's gustiness can be enhanced if C_wp2_splat is set to non-zero values. This may enhance the benefits found with Po-Lun Ma connection of CLUBB's gustiness and surface processes.

• CLUBB_v2 can now prognose momentum fluxes if l_predict_upwp_vpwp = .true. This is designed to reduce ocean surface stress in cumulus regions.

[NBFB]

[singhbalwinder]

Design Doc: https://acme-climate.atlassian.net/wiki/spaces/ED/pages/1001455620/W9+CLUBB+v2

I have recently rebased this branch. I am running some tests and will soon update the PR description with the new diagnostics info.

[rljacob]

telecon: encountering issues with testing on different machines.

[singhbalwinder]

I have run the model for 6 years and diagnostics from 2nd to 6th year are at: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/bsingh/2-6yr/SMS.ne30_ne30.FC5AV1C-L.anvil_intel.clbv2_rbsd_1_0cb6bd4-SMS.ne30_ne30.FC5AV1C-L.anvil_intel.def_mstr_b5aaa6b7d228_clbv2_rbsd_1/

Tagging @golaz and @polunma so that they can take a look at the diagnostics.

[singhbalwinder]

This PR is ready to merge after it is approved.

[rljacob]

@golaz please approve this.

[golaz]

@singhbalwinder : can you document this in a little more details on Confluence? If already done, please add link here. In particular I'd like to see:

• Details about simulations (both base and new CLUBB version): code base, script, output location, ...
• E3SM Diags for model vs obs (both simulations) as well as model vs model.

[singhbalwinder]

Thanks @golaz ! @vlarson is working on creating a design document. @vlarson or I will add a link here once it is ready.

For the diagnostics, would it be okay to run AMWG? I have those setup so it will be easy to run those with obs.

[vlarson]

Thanks @golaz ! @vlarson is working on creating a design document. @vlarson or I will add a link here once it is ready.

For the diagnostics, would it be okay to run AMWG? I have those setup so it will be easy to run those with obs.

@golaz , @singhbalwinder , I have started a design doc here. Please let me know what I need to add.

[golaz]

@vlarson : thanks for working on the design document.

@singhbalwinder: I can live with AMWG figures for now, but it would be good if you learned how to use E3SM Diags in the future. I think it's pretty straightforward.

[wlin7]

Merged to master and submitted requests to bless DIFFs for sandiatoss3 (37), melvin(7) and anvil(2).

[rljacob]

2 PET tests failed. Those will require a closer look:
PET_Ln9_PS.ne30_oECv3_ICG.A_WCYCL1850S.sandiatoss3_intel.allactive-mach-pet
PET_Ln5.ne4_ne4.FC5AV1C-L.sandiatoss3_intel.allactive-mach-pet

[wlin7]

@singhbalwinder , are you aware whether this PR could cause threading to be non-BFB? The two PET tests apparently say so. (DIFF between base and single-thread, in addition to DIFF from latest baseline)

@rljacob , at this point, should the merge be reverted? It may take a while to investigate.

[wlin7]

@singhbalwinder , I just checked slack and noticed you were doing tests on this threading issue. Thanks.

[wlin7]

Reverted the commit on master and next.
Resubmitted the requests to bless the previous DIFFs on sandiatoss3, melvin and anvil.

[rljacob]

@wlin7 you can't submit the bless until after the tests run with the new code. After tonight.

[wlin7]

@rljacob , @wade did comment that he can't bless anything else today. I will resubmit the request based on the test results tomorrow. That said, I don't understand why it can't be done with the master after this PR is reverted. The master is essentially back to what it was yesterday. The DIFFs that would come up tomorrow (for next and master) are expected to be the same as we saw today for next tests. I was thinking with re-blessing, the baselines that were in place before today's blessing would all be restored. And with baseline restored, we would be in better place to test the PRs going into next today.

[rljacob]

The bless command is script that works in one direction: it copies output from the run of testing on "next" to the baselines. And we have just one set of baselines that both next and master are compared against.

[wlin7]

I see now. Thanks for the explanation, @rljacob.
I thought master would be used during blessing to update the baseline because normally we merge the PR to master before requesting to bless.

[singhbalwinder]

Thanks to Zhun Guo and @vlarson , the PET test is now passing. I am now running the fully coupled PET test to ensure that I get a PASS there as well. After that, it should be good to go.

[singhbalwinder]

I got PASS on both the tests. I think it is good to go now.

[rljacob]

Its to late to affect tonights test run.

[wlin7]

Great work, @singhbalwinder. Thanks. Nightly tests probably have started. I will wait until tomorrow.

[wlin7]

Rob, is there any other non-BFB scheduled for Thursday? If not and next open, i will merge it in the morning and ask our integrators to hold on.