Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove duplicative allsoils filter for FATES SatellitePhenology call #1515

Merged
merged 17 commits into from
Jul 18, 2022

Conversation

glemieux
Copy link
Collaborator

@glemieux glemieux commented Oct 8, 2021

Description of changes

Addresses #1488

This also incorporates a small name change to the Fates testmod and the fates-sp usermod per NGEET/fates#854.

Specific notes

Contributors other than yourself, if any:

CTSM Issues Fixed (include github issue #):

Are answers expected to change (and if so in what way)?

Any User Interface Changes (namelist or namelist defaults changes)?

Testing performed, if any:
(List what testing you did to show your changes worked as expected)
(This can be manual testing or running of the different test suites)
(Documentation on system testing is here: https://github.com/ESCOMP/ctsm/wiki/System-Testing-Guide)
(aux_clm on cheyenne for intel/gnu and izumi for intel/gnu/nag/pgi is the standard for tags on master)

NOTE: Be sure to check your coding style against the standard
(https://github.com/ESCOMP/ctsm/wiki/CTSM-coding-guidelines) and review
the list of common problems to watch out for
(https://github.com/ESCOMP/CTSM/wiki/List-of-common-problems).

@glemieux glemieux added the code health improving internal code structure to make easier to maintain (sustainability) label Oct 8, 2021
@glemieux
Copy link
Collaborator Author

glemieux commented Oct 8, 2021

Initial testing shows that SMS SatellitePhenology testmod pass b4b against baseline:
/glade/scratch/glemieux/ctsm-tests/tests_fates-spmode-filterfix-sms. ERS test still fails COMPARE_base_rest.

@billsacks
Copy link
Member

billsacks commented Oct 8, 2021

Awesome, thanks for this cleanup.

  • Since it's uncommon to need to use the filter_inactive_and_active, can you add a comment (probably near the FATES SatellitePhenology call in clm_driver) explaining why you need to use this version that includes inactive points?

@glemieux
Copy link
Collaborator Author

glemieux commented Oct 8, 2021

@rosiealice I adapted your comment from your allsoil filter comment (that we're replacing). Do you think I accurately represent what's being attempted:
https://github.com/glemieux/ctsm/blob/be5d45c74095560fc067331348942a4cddcb8c17/src/main/clm_initializeMod.F90#L635-L642

@billsacks
Copy link
Member

billsacks commented Oct 8, 2021

Thanks a lot @glemieux . For the future, we prefer if checking off checkboxes is left to the reviewer (or whoever added the checkbox in the first place): see https://github.com/ESCOMP/CTSM/wiki/Pull-request-review-workflow and the links from there for details. (No worries about this: I fully expect to need to tell everyone this one or more times before it sticks, but it's helpful for me as a reviewer to be able to use checkboxes like this as a list of requests I have made, which I can then check off as I'm satisfied that they have been addressed. Please feel free to give me any feedback about how this could be more intuitive. For many contributors this is not an issue, because they wouldn't have permission to check off my checkboxes anyway, but you were able to because you have write permission to this repository.)

@glemieux
Copy link
Collaborator Author

glemieux commented Oct 8, 2021

@billsacks gotcha. I've been using task check boxes for myself on other PRs and issues so I've just been in the habit of ticking them off when I'm done without much thought. I hadn't reviewed the PR workflow doc in a while (since before you added the gh-pr-query tool) so I missed the direction on how to handle them when being assigned by reviewers.

Is there a way to lock down the edit option for comments just to the OP?

@billsacks
Copy link
Member

Is there a way to lock down the edit option for comments just to the OP?

I don't think so. Comments can be edited by the OP or anyone with write access to the repository. That's still better than GitHub's "resolve" feature, which can be triggered by the PR author regardless of their repo permissions, but it means that we need to educate anyone with write permissions on this workflow. (I have been in touch with some people at GitHub to encourage them to add first-class support for a feature like this, but I have no idea if/when that would happen. In the meantime, this admittedly imperfect system is the best I've been able to come up with, but I'm open to suggestions.)

@glemieux
Copy link
Collaborator Author

Note to self: given that this effects FATES SP mode only I think we could couple this with the eventual fix for #1485

@glemieux glemieux marked this pull request as ready for review June 1, 2022 22:00
@glemieux glemieux requested a review from ekluzek June 1, 2022 22:07
@glemieux
Copy link
Collaborator Author

glemieux commented Jun 2, 2022

Izumi aux_clm test against ctsm5.1.dev098 was b4b with the expected exception of the fates NLCOMP and FIELDLIST DIFFs due to the history variable name change:
/scratch/cluster/glemieux/ctsm-tests/tests_pr854_ctsm1515-aux_clm

Similarly, Cheyenne aux_clm test against the same baseline is b4b with same expected exceptions (including one expected failure). Additionally the is a TPUTCOMP and MEMCOMP failure:
/glade/u/home/glemieux/scratch/ctsm-tests/tests_tests_pr854_ctsm1515-aux_clm

The fates test suite is complete as well: NGEET/fates#854 (comment)

First Commit of CN Matrix Solution:
Start bringing in the use_matrixcn and use_soil_matrixcn control options for the Carbon/Nitrogen BGC model using
the matrix solution methodology developed by Dr. Yiqi Luo and his EcoLab members: Drs. Xingjie Lu,
Yuanyuan Huang and Zhengguang Du, at Northern Arizona University

Work on Anomaly forcing for SSEP scenarios:
Do some work on anomoly forcing for SSP scenarios. Update the script in tools/contrib to work with the latest
data and replicate results of Keith Oleson and Sean Swenson. Add some test cases for it.

Python Environment Management using Conda:
More work on the manage_python_env script to setup the "ctsm_py" conda environment. This now works for UCAR machines
(such as cheyenne and casper) as well as CGD machines (like izumi). This script can also be called for tools, and
the test tools mechanism is now using it rather than ncar_pylib.

./manage_python_env

SCAM Fix:
Add checkimport method for CMEPS so that single-column case can work with SCAM, from Jim Edwards.

Black Formatter for Python Code:
Run "black" python code reformatter on all code under the "python" directory. Add the commits
to .git-blame-ignore-revs so they can be ignored when "git blame" is used, if you add this
to each clone where you want this to apply:

   git config blame.ignoreRevsFile .git-blame-ignore-revs

You can do it for your worktrees, but since the file only applies to CTSM for ctsm5.1.dev100 and
forward, you'll need to do it for each CTSM clone seperately. Also there is now a github action
that checks that python code under the "python" subdirectory is black clean. If it's not the action
will fail and send you an email about it.

Make Map Data Fix:
Simple fix from Sam Rabin.

Fix for DISPLA on History Output:
Fix from Keith Oleson to set history output for DISPLA (displa_patch) to zero when vegetation is buried by snow.
@glemieux
Copy link
Collaborator Author

glemieux commented Jul 8, 2022

After conversation with @ekluzek regarding efforts to address #1794, we've decided to update the externals configuration to include the tag that will result from ESMCI/ccs_config_cesm#50. There will also be a branch tag update for time per ESMCI/ccs_config_cesm#50 (comment)

@glemieux glemieux linked an issue Jul 8, 2022 that may be closed by this pull request
glemieux added 4 commits July 12, 2022 23:53
Fix accumulation variables when changing model time step

Accumulation variables (e.g., 1-day or 10-day averages) were writing and
reading their accumulation period (expressed in time steps) to the
restart file. This caused incorrect behavior when changing the model
time step relative to what was used to create the initial conditions
file (typically a 30-minute time step). So, for example, if you are
using a 15-minute time step with an initial conditions file that
originated from a run with a 30-minute time step (at some point in its
history), then an average that was supposed to be 10-day instead becomes
5-day; an average that was supposed to be 1-day becomes 12-hour, etc.
(The issue is that the number of time steps in the averaging period was
staying fixed rather than the actual amount of time staying fixed.)

For our out-of-the-box initial conditions files, this only impacts runs
that use something other than a 30-minute time step. Typically this
situation arises in configurations with an active atmospheric model that
is running at higher resolution than approximately 1 degree. It appears
that the biggest impacts are on VOC emissions and in BGC runs; we expect
the impact to be small (but still non-zero) in prescribed phenology (SP)
runs that don't use VOC emissions.

This tag fixes this issue by no longer writing or reading accumulation
variables' PERIOD to / from the restart file: this isn't actually needed
on the restart file.

See some discussion in ESCOMP#1789 for
more details, and see
ESCOMP#1802 (comment) for
some discussion of outstanding weirdness that can result for
accumulation variables when changing the model time step. The summary of
that comment is: There could be some weirdness at the start of a run,
but at least for a startup or hybrid run, that weirdness should work
itself out within about the first averaging period. A branch or restart
run could have some longer-term potential weirdness, so for now I think
we should recommend that people NOT change the time step on a branch or
restart run. With (significant?) additional work, we could probably
avoid this additional weirdness, but my feeling is that it isn't worth
the effort right now. In any case, I feel like my proposed fix will
bring things much closer to being correct than they currently are when
changing the time step.
@glemieux
Copy link
Collaborator Author

glemieux commented Jul 16, 2022

Cheyenne aux_clm test against the latest baseline results in a number of NLCOMP and FIELDLIST differences. The latter are expected differences due to the updated fates tag, which corrects a small set of fates history variable names. The former appear to be due the ccs_config update in ccs_config_cesm0.0.37, which appears to be a correction.

The fates testmod differences are all due to fates_levcdam boundary value at the first index only which looks like junk. I believe this is acceptable as this relates to the crown damage module (NGEET/fates#788) api updates that came in with ctsm5.1.dev099 and is schedule for integration at a later date (crown damage flags are currently set inactive by default). @rgknox does this seem valid to you?

File location on Cheyenne: /glade/u/home/glemieux/scratch/ctsm-tests/tests_pr1515-aux_clm-externupdate

Izumi aux_clm tests show the same pattern of differences, although the fates_levcdam diff is not showing up.

File location on Izumi: /scratch/cluster/glemieux/ctsm-tests/tests_pr1515-aux_clm-extern_update

@glemieux
Copy link
Collaborator Author

glemieux commented Jul 18, 2022

Discussed this with @rgknox at our regular Monday morning PR triage meeting. We decided to take care of the levcdam initialization now with the necessary updates here to avoid this DIFF in the next fates api update. Ryan will push the updates soon and create a new fates-side PR for us to tag (since I already integrated NGEET/fates#854.

UPDATE: no fates-side fix necessary. Retesting the fates testmods with DIFFs.

@glemieux
Copy link
Collaborator Author

Re-running the fates tests that showed differences now reports the appropriate difference value as included in the fates parameter file from the last update: https://github.com/NGEET/fates/blob/def6b3e76f9ff3043150a777f403883b3e659374/parameter_files/fates_params_default.cdl#L800

 317  fates_levcdam   (fates_levcdam)
 318           1        2  (     2) (     1) (     2) (     2)
 319                    2   8.000000000000000E+01   0.000000000000000E+00 8.0E+01  8.000000000000000E+01 5.0E-01  8.000000000000000E+01
 320                    2   0.000000000000000E+00   0.000000000000000E+00          0.000000000000000E+00          0.000000000000000E+00
 321                    2  (     1) (     1)
 322           avg abs field values:    4.000000000000000E+01    rms diff: 5.7E+01   avg rel diff(npos):  5.0E-01
 323                                    0.000000000000000E+00                        avg decimal digits(ndif):  0.0 worst:  0.0
 324  RMS fates_levcdam                    5.6569E+01            NORMALIZED  2.8284E+00

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code health improving internal code structure to make easier to maintain (sustainability)
Projects
Status: Done (non release/external)
3 participants