cam-se fails if CLDICE, CLDLIQ, or Q (or PS) is not in the state vector

[kdraeder]

Describe the bug

1. List the steps someone needs to take to reproduce the bug.
   Build a cam-se filter.
   Set model_nml:state_variables to a set of variables that is "missing" one or more of the moisture variables.
   Assimilate any appropriate observation (Feature request: CAM SE Manhattan #343 may provide files that are helpful for setting up an assimilation).
2. What was the expected outcome?
   The existence of state_variables in the namelist implies that the user is free to define the model state,
   so the expected outcome is that the observations will be assimilated.
3. What actually happened?
   All observations fail in the forward operator (QC=4).

Error Message

There is not enough diagnostic messaging from model_mod to learn the cause of the failure.
Adding messages about the various istatus contents leads to the discovery that build_dry_mass_pressure_columns requests values for QTY_SPECIFIC_HUMIDITY, QTY_CLOUD_LIQUID_WATER,
and QTY_CLOUD_ICE. If one of those fails, the failure code is immediately returned to the calling routine
without any diagnostic message. The calling routine converts the code to a generic failure code,
which eventually is converted into QC=4, which is not very informative, given the large number of ways
that the forward operator can fail.

Which model(s) are you working with?

cam-se
A similar issue applies to cam-fv too, because PS is required (in both).
No one has ever excluded PS and reported a problem.

Version of DART

Which version of DART are you using?
Any version after the refactoring which uses cam-common-code and after build_dry_mass_pressure_columns
was added. (I don't see it mentioned in the git log output, so I don't know when it was installed.)
The version I modified and sent to Nuo Chen for testing is v9.12.refactored-1915-gb3e00330f.
If a cleaner version number is needed, I'll work on finding one.

Have you modified the DART code?

I added diagnostic prints, but did not change any calculations or procedure calls.
There are several possible solutions to this problem:

1. Tell users in documentation that state_variables must include those 3 moisture variables. This is minimal effort, but unreliable.
2. Add code to check for the presence of the moisture variables and fail if one is missing.
3. Add printed messages (controlled by debug_level) that list useful debugging information: subroutine, istatus, related variables,... This can save hours to days of debugging. If this is part of the chosen solution, I can provide a version that does some of this.
4. Define (hardcode) a minimal set of state variables which are always in the state and allow users to add variables through the namelist.

A question related to 1. is whether those 3 variables are all that are required to correctly calculate the dry mass pressures.

Build information

Please describe:

1. The machine you are running on: any machine
2. The compiler you are using: any compiler

[hkershaw-brown]

Hey Kevin, v9.12.refactored-1915-gb3e00330f means 1915 commits after version v9.12 so we should chat and get you working on the main branch again.

[hkershaw-brown]

the dry mass pressure came in here: 1982c75
There are some notes about the 'precise' setting here:
#343 (comment)

[hkershaw-brown]

Thanks for reporting this, Kevin (and Nuo). Before doing a pull request to fix for this, I'd like to get a spec for what the dry_mass_vertical_coordinate options are, what namelist options are needed and what state qtys are needed for the namelist options. Ideally we would have done this when the original code was written, but the second best time to do this is now.

[kdraeder]

@hkershaw-brown thanks for digging up the SHA.
I didn't dig into the dry mass code enough to be able to write a spec yet, and I probably won't be able to work on this until April 15.
Someone else might be able to do it sooner or faster.
Which solution looks best to you?

[hkershaw-brown]

Spec first.

[kdraeder]

I've written a spec and put it in a G-doc in DAReS team > Spec > Handling calculation of total pressure in cam-se

There may be some confusion, and some connection, between this issue and
one that @jlaucar says has been lurking for several years.
This issue (#662) seems straightforward to me; if a user leaves out any of the moisture fields
from the state definition, then all of the obs will get QC=4.
Jeff's related issue may be mentioned in the cam-se/model_mod:

! Note that other tracers in the dry mass cam/se have a dry mixing ratio, not specific mixing ratio (although
! Peter plans to change this for consistency).
!SENote: The following should all be 0, but A's at levels 2, 3, 4, and 5 are not.
! This is confirmed in the caminput.nc file and is a PROBLEM.
! Some tests on the A and B coefficients
!do k = 1, nlevels
   !write(*, *) k, (grid_data%hyai%vals(k) + grid_data%hyai%vals(k+1))/2.0_r8 - grid_data%hyam%vals(k)
   !write(*, *) k, (grid_data%hybi%vals(k) + grid_data%hybi%vals(k+1))/2.0_r8 - grid_data%hybm%vals(k)
!enddo