Highcam #422

nancycollins · 2022-11-04T17:40:51Z

Description:

Adds a new program to remove obs above a given CAM model level from an obs_seq file. This program takes a single obs_seq file in and writes a single obs_seq file out without the high obs. In the future it could also superob dense obs, change obs errors near the surface, or any other CAM-specific processing that might be needed.

It can either be added to the cam workflow scripts, or a month of obs files could be run offline ahead of time and then the cam scripts can read in the updated files instead of the originals.

Edit: this has been moved to pull request #424
Also added an unrelated update to the clean_nml.c program to capture additional features i've added to it. Probably should be moved to its own pull request but it's a single file update so i dumped it here.

Fixes issue

#415

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

Ran with one of the reanalysis obs_seq files and it removed the high obs.

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

Main motivation was to get the CAM-SE branch.

assimilating obs at the cam model top leads to problems because the model top has a fixed upper bound. this program uses the routine inside the cam model_mod.f90 to remove obs above a given model level.

add some command line args to control how much to sort (all contents of nameslists but leave the namelists in the original order, vs sorting the namelists themselves into alphabetical order), and whether to remove comment lines outside of namelists or not.

hkershaw-brown · 2022-11-07T16:53:51Z

@nancycollins I am going to split this into two pull requests (clean_nml and high cam) and then ask you a bunch of questions on both. Stay tuned

kdraeder · 2022-11-07T17:01:43Z

@nancycollins @hkershaw-brown
What's the relationship between the proposed highcam and the recently created camdart_obs_preprocessor?
(#296 and #415)
On the surface it seems that highcam may end up doing what I thought preprocessor is intended to do,
but has a less generic name.

This reverts commit 93d4d83.

This reverts commit 0acff10.

hkershaw-brown · 2022-11-07T17:30:48Z

clean_nml moved to pull request #424

nancycollins · 2022-11-07T21:59:47Z

@nancycollins @hkershaw-brown What's the relationship between the proposed highcam and the recently created camdart_obs_preprocessor? (#296 and #415) On the surface it seems that highcam may end up doing what I thought preprocessor is intended to do, but has a less generic name.

highcam is the name of the git branch with the camdart_obs_preprocessor code in it.

to handle PR #422.

models/cam-fv/camdart_obs_preprocessor.f90

kdraeder · 2022-11-18T22:35:37Z

models/cam-fv/camdart_obs_preprocessor.f90

+                            trim(filename_out)
+call error_handler(E_MSG,'main',msgstring,source)
+
+print*, 'Number of obs in the output seq file :', get_num_key_range(seq_out)


Why the different methods for printing messages? (print and error_handler(E_MSG...)

line 271 looks like a debugging line that should be removed. i'm also a bit mystified about the "if not verbose" test at 260. it looks like that should be "if verbose". i think the idea was verbose messages might print to the screen but other messages go into the log as well as screen. my suggestion is to run it a couple times with and without verbose and decide what you think are the most useful messages for various situations - like if the output file is empty, what info would help to print out? and put those prints in the verbose sections. and if you want to use msg for everything, i'm ok with that too.

I've made some changes, which I'll push for people to check out.
I went a bit beyond what's been suggested so far.

kdraeder · 2022-11-18T22:40:35Z

models/cam-fv/camdart_obs_preprocessor.rst

+   |              |                    |                                                                                |
+   +--------------+--------------------+--------------------------------------------------------------------------------+
+   | calendar     | character(len=32)  | The string name of a valid DART calendar type. (See the                        |
+   |              |                    | :doc:`../../modules/utilities/time_manager_mod` documentation for a list of    |


links to other documentation look like ':doc:`../../../assimilation_code/'...
in the github rendering. Will it look different in the docs.dart.ucar.edu?

i'm going to leave the doc situation to others - i haven't done it enough to understand the current process, sorry.

models/cam-fv/camdart_obs_preprocessor.rst

kdraeder

@nancycollins @hkershaw-brown
I've made a few comments and suggestions.
None of them are worrisome.

nancycollins · 2022-11-21T16:52:32Z

hi kevin, thanks for the review - all good points. if you're willing to address all or some of them, i'd love for you to be the one to take ownership of this program since it's possibly going to be needed for more functions in the future. if you're busy or don't want to address some, email me and i'll look at doing some of the updates. thanks.

kdraeder · 2022-11-21T17:06:43Z

hi kevin, thanks for the review - all good points. if you're willing to address all or some of them, i'd love for you to be the one to take ownership of this program since it's possibly going to be needed for more functions in the future. if you're busy or don't want to address some, email me and i'll look at doing some of the updates. thanks.

I agree that it makes sense for me to look after it.
I'll work through the name change, leave the "use ..." contents (I hadn't thought of expansion),
and look into whether I can fix the others.

and fixed typos in the previous commit (camdart -> cam_dart changes).

Added E_WARN to the program.

hkershaw-brown

What is up with these merges from the CAM-SE branch?

cam_dart_obs_preproessor is not in the table of contents for the documentation.

hkershaw-brown · 2022-11-22T22:54:14Z

We can talk about this after thanksgiving, but this merge history is weird.

When working with a pull request you can checkout the feature branch

https://ncar.github.io/dart-developers-guide/reviewing.html

Don't accept this pull request as is.

kdraeder · 2022-11-23T00:17:29Z

I wasn't aware of the reviewing.html. Thanks for the tip. I'll try to understand it before any more github work.

I'm working in /glade/u/home/raeder/DART/Manhattan_git.
This was updated some weeks ago to have the cam-se shared code in it.
I believe that I checked out Nancy's qc8fix branch to do that.
I did some work with that, cleaned it up, then checked out main.
I made a highcam directory, went there, and
$ git pull origin highcam
I made a bunch of changes to the preprocessor, as discussed above, committed them, then
$ git push origin highcam:highcam
Apparently this was misguided.

hkershaw-brown · 2022-11-28T20:33:43Z

moved commits to high-cam

kdraeder and others added 10 commits February 2, 2022 20:07

Remove a spurious _ from CAM's fv_div24del2flag

069fa6b

Merge remote-tracking branch 'refs/remotes/origin/main' into main

0de4326

Main motivation was to get the CAM-SE branch.

Merge remote-tracking branch 'remotes/origin/CAM_SE' into main

22e2d17

Update main to get build tools

fbe4aab

add a simple obs preprocessor for cam-fv that removes high obs

a11dabb

assimilating obs at the cam model top leads to problems because the model top has a fixed upper bound. this program uses the routine inside the cam model_mod.f90 to remove obs above a given model level.

add namelist for new obs preprocessor program to input.nml

23f44b2

add namelist and rst file for cam obs preprocess program.

d57cc9b

Merge branch 'main' into highcam

a496b48

clean up some of the comment lines to be clearer

93d4d83

nancycollins requested a review from hkershaw-brown November 4, 2022 17:40

hkershaw-brown added 2 commits November 7, 2022 10:26

Revert "clean up some of the comment lines to be clearer"

7b9a945

This reverts commit 93d4d83.

Revert "update clean_nml.c program for namelists"

630f4c1

This reverts commit 0acff10.

hkershaw-brown mentioned this pull request Nov 8, 2022

update to the clean_nml.c program #424

Merged

15 tasks

Merge branch 'highcam' of github.com:NCAR/DART

56bf701

to handle PR #422.

kdraeder reviewed Nov 18, 2022

View reviewed changes

models/cam-fv/camdart_obs_preprocessor.f90 Outdated Show resolved Hide resolved

kdraeder reviewed Nov 18, 2022

View reviewed changes

models/cam-fv/camdart_obs_preprocessor.f90 Outdated Show resolved Hide resolved

kdraeder reviewed Nov 18, 2022

View reviewed changes

models/cam-fv/camdart_obs_preprocessor.rst Outdated Show resolved Hide resolved

kdraeder reviewed Nov 18, 2022

View reviewed changes

kdraeder added 2 commits November 21, 2022 10:24

Renamed files to be consistent with mpas, wrf preprocessor

078d1db

Changed camdart to cam_dart in all files

f29d5b1

kdraeder self-assigned this Nov 21, 2022

kdraeder added 2 commits November 21, 2022 14:17

Improved messages from cam_dart_obs_preprocessor

e316b45

and fixed typos in the previous commit (camdart -> cam_dart changes).

Added error messages and required files to documentation

2c0db5b

Added E_WARN to the program.

hkershaw-brown requested changes Nov 22, 2022

View reviewed changes

hkershaw-brown closed this Nov 22, 2022

hkershaw-brown deleted the highcam branch November 28, 2022 20:33

hkershaw-brown mentioned this pull request Feb 24, 2023

feature request: program to remove obs above a given CAM model level from an obs_seq file #459

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Highcam #422

Highcam #422

nancycollins commented Nov 4, 2022 •

edited by hkershaw-brown

hkershaw-brown commented Nov 7, 2022

kdraeder commented Nov 7, 2022 •

edited

hkershaw-brown commented Nov 7, 2022

nancycollins commented Nov 7, 2022

kdraeder Nov 18, 2022

nancycollins Nov 21, 2022

kdraeder Nov 21, 2022

kdraeder Nov 18, 2022 •

edited

nancycollins Nov 21, 2022

kdraeder left a comment

nancycollins commented Nov 21, 2022

kdraeder commented Nov 21, 2022

hkershaw-brown left a comment

hkershaw-brown commented Nov 22, 2022

kdraeder commented Nov 23, 2022

hkershaw-brown commented Nov 28, 2022

Highcam #422

Highcam #422

Conversation

nancycollins commented Nov 4, 2022 • edited by hkershaw-brown

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

hkershaw-brown commented Nov 7, 2022

kdraeder commented Nov 7, 2022 • edited

hkershaw-brown commented Nov 7, 2022

nancycollins commented Nov 7, 2022

kdraeder Nov 18, 2022

Choose a reason for hiding this comment

nancycollins Nov 21, 2022

Choose a reason for hiding this comment

kdraeder Nov 21, 2022

Choose a reason for hiding this comment

kdraeder Nov 18, 2022 • edited

Choose a reason for hiding this comment

nancycollins Nov 21, 2022

Choose a reason for hiding this comment

kdraeder left a comment

Choose a reason for hiding this comment

nancycollins commented Nov 21, 2022

kdraeder commented Nov 21, 2022

hkershaw-brown left a comment

Choose a reason for hiding this comment

hkershaw-brown commented Nov 22, 2022

kdraeder commented Nov 23, 2022

hkershaw-brown commented Nov 28, 2022

nancycollins commented Nov 4, 2022 •

edited by hkershaw-brown

kdraeder commented Nov 7, 2022 •

edited

kdraeder Nov 18, 2022 •

edited