Add observation preparation job for aerosols DA to workflow #2624

ypwang19 · 2024-05-23T20:22:40Z

Description

This PR added a prepaeroobs job to prepare aerosol obs files for DA. This job does quality control of the VIIRS aerosol raw observations and convert them to ioda format.

Resolves #2623

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Cycled test on Hera

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

…AA-EMC#2614) - Updating STMP and PTMP settings in host file for Orion and Hercules because they are cross mounted. - Also took the opportunity to finally update **SLURM_ACCOUNT** to **HPC_ACCOUT** in CI over rides. - Added a refactor of the `rocotostat.py` tool that is more pythonic and as a execute retry feature because the `rocotostat` utility on Orion has been failing sometimes.

* origin: Change GRIB2 parameter names and vertical levels for ocean/ice post (NOAA-EMC#2611) Add atmensanlfv3inc job (NOAA-EMC#2592) Global-workflow (AR) Generic updates for Gaea C5 (NOAA-EMC#2515) Update STMP and PTMP settings in host file for Orion and Hercules (NOAA-EMC#2614)

env/HERCULES.env

ypwang19 · 2024-05-31T13:37:55Z

@CoryMartin-NOAA @andytangborn This PR is ready for review. Any comments and suggestions are welcome.

env/HERA.env

jobs/JGLOBAL_PREP_AERO_OBS

parm/config/gfs/config.base

WalterKolczynski-NOAA · 2024-05-31T14:21:19Z

jobs/JGLOBAL_PREP_AERO_OBS

+##############################################
+
+# Generate COM variables from templates
+YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_OBS COM_CHEM_ANALYSIS


COM_OBS should be considered read-only. In development we copy obs from the dump archive to a location within our directories, but in operations we use them directly from obsproc. We can make new prep COM variables in config.com for any components that need them.

@CoryMartin-NOAA @guillaumevernieres I think there are other jobs recently that have the same issue that we just let through that will need to be updated.

I think we need to have a bigger discussion on this. There are obs for the non-atmospheric components that are not being processed through the traditional obsproc channels (in particular the aerosol and SOCA datasets). I'll send an email to follow up and schedule a meeting.

My solution would be to create a COM_<component>_PREP_TMPL (or similar; waves already has a prep) to store pre-processed stuff.

CoryMartin-NOAA

Some comments

CoryMartin-NOAA · 2024-05-31T13:55:59Z

parm/config/gfs/config.prepaeroobs

+
+export OBSPROCYAML="${PARMgfs}/gdas/aero/obs/lists/gdas_aero_obsproc.yaml.j2"
+export OBSPROCEXE="${EXECgfs}/gdas_obsprovider2ioda.x"
+export VIIRS_DATA_DIR="/scratch2/NCEPDEV/stmp3/Yaping.Wang/VIIRS/AWS/"


we'll have to change this to a more "official" location for at least the CI test period before the PR can be successfully merged in.

parm/config/gfs/config.prepaeroobs

jobs/JGLOBAL_PREP_AERO_OBS

CoryMartin-NOAA · 2024-05-31T14:07:38Z

ush/python/pygfs/task/aero_prepobs.py

+        List needed raw obs files.
+        Copy the raw obs files to $DATA/obs.
+        Link over the needed executable.
+        Generate corrosponding YMAL file.


Suggested change

Generate corrosponding YMAL file.

Generate corresponding YMAL file.

CoryMartin-NOAA · 2024-05-31T14:09:07Z

ush/python/pygfs/task/aero_prepobs.py

+        """
+        self.task_config.COM_OBS_CHEM = os.path.join(self.task_config.COM_OBS, 'chem')
+        if os.path.exists(self.task_config.COM_OBS_CHEM):
+            rmdir(self.task_config.COM_OBS_CHEM)


are we positive we want to remove the directory if it exists?

ush/python/pygfs/task/aero_prepobs.py

CoryMartin-NOAA · 2024-05-31T14:17:44Z

ush/python/pygfs/task/aero_prepobs.py

+                HH = fshort[0][26:28]
+                MM = fshort[0][28:30]
+                fstart = to_datetime(yyyy + '-' + mm + '-' + dd + 'T' + HH + ':' + MM + 'Z')
+                if sensor in file:


I think this might need to be more robust, for example, what if the directory path is:
/work2/noaa/cmartin/exp/use-n20/data/n21.nc, it will be true for both n20 and n21.

Still think this comment stands, but perhaps I'm worrying too much.
Can you do if sensor in os.path.basename(file) or something like that?

WalterKolczynski-NOAA · 2024-05-31T14:27:53Z

ush/python/pygfs/task/aero_prepobs.py

+                dd = fshort[0][24:26]
+                HH = fshort[0][26:28]
+                MM = fshort[0][28:30]
+                fstart = to_datetime(yyyy + '-' + mm + '-' + dd + 'T' + HH + ':' + MM + 'Z')


Suggested change

fstart = to_datetime(yyyy + '-' + mm + '-' + dd + 'T' + HH + ':' + MM + 'Z')

fstart = to_datetime(f'{yyyy}-{mm}-{dd}T{HH}:{MM}Z')

ush/python/pygfs/task/aero_prepobs.py

jobs/JGLOBAL_PREP_OBS_AERO

CoryMartin-NOAA · 2024-06-05T19:39:26Z

@ypwang19 can you merge in develop and resolve the conflicts? I'll review what is here in the meantime

DavidHuber-NOAA · 2024-06-14T12:55:29Z

parm/config/gfs/config.resources

+    export npe_prepobsaero=1
+    export nth_prepobsaero=1
+    export npe_node_prepobsaero=1
+    export memory_prepobsaero="96GB"


Does this prep job really need this much memory?

Probably not, but it is somewhat intensive, but this is probably an order of magnitude too large, I can test next week with less memory

This can be addressed later.

CoryMartin-NOAA · 2024-06-14T12:57:00Z

FYI @ypwang19 is now on maternity leave, so I'll have to pick up this PR to get it finished, so there may be some delay in responses to reviews.

env/S4.env

env/WCOSS2.env

ush/python/pygfs/task/aero_prepobs.py

DavidHuber-NOAA

Looks pretty good overall, just one fix for WCOSS2 and a few suggestions.

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

CoryMartin-NOAA · 2024-06-14T14:43:41Z

@aerorahul @DavidHuber-NOAA @WalterKolczynski-NOAA Yaping is now on maternity leave. What do you suggest in terms of getting this PR over the finish line? My suggestions in no particular order are:

Merge as is (it's possible everything is fine, may need to stage some obs)
Merge this except the task definition stuff so that nothing breaks
Close the PR and I open a new one in my fork so I can make necessary edits
Other options?

DavidHuber-NOAA · 2024-06-14T14:46:43Z

parm/config/gfs/config.base

@@ -178,6 +178,7 @@ export DO_WAVE="NO"
 export DO_OCN="NO"
 export DO_ICE="NO"
 export DO_AERO="NO"
+export DO_PREP_OBS_AERO="YES"


@CoryMartin-NOAA How about setting this to NO for now?

Yes I think that is all that is needed to ensure it doesn't break anything

DavidHuber-NOAA

There are a couple of remaining issues, but those can be addressed later IMO. Once an issue is created, I'm good with merging this.

DavidHuber-NOAA · 2024-06-14T14:53:14Z

jobs/JGLOBAL_PREP_OBS_AERO

+# Set variables used in the script
+##############################################
+
+export COMIN_OBS="${DATA}"


This will likely need to be updated, but that can wait since the new job is now off by default. Can you create an issue for this @CoryMartin-NOAA?

DavidHuber-NOAA · 2024-06-14T14:54:06Z

parm/config/gfs/config.prepobsaero

+
+export OBSPROCYAML="${PARMgfs}/gdas/aero/obs/lists/gdas_aero_obsproc.yaml.j2"
+export OBSPROCEXE="${EXECgfs}/gdas_obsprovider2ioda.x"
+export VIIRS_DATA_DIR="/scratch2/NCEPDEV/stmp3/Yaping.Wang/VIIRS/AWS/"


Perhaps this could be wrapped into the new issue.

DavidHuber-NOAA · 2024-06-14T14:54:23Z

parm/config/gfs/config.resources

+    export npe_prepobsaero=1
+    export nth_prepobsaero=1
+    export npe_node_prepobsaero=1
+    export memory_prepobsaero="96GB"


This can be addressed later.

aerorahul · 2024-06-14T15:07:30Z

ush/python/pygfs/task/aero_prepobs.py

+        try:
+            for file in allfiles:
+                basename = os.path.basename(file)
+                pattern = r"s(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{3})"


pattern could be defined outside the for loop.

aerorahul · 2024-06-14T15:08:35Z

ush/python/pygfs/task/aero_prepobs.py

+            logger.info("Found %d matching files.", len(matching_files))
+        except FileNotFoundError:
+            logger.error("The specified file/directory does not exist.")
+            raise


Should raise FileNotFoundError(message)

* origin/develop: Add observation preparation job for aerosols DA to workflow (NOAA-EMC#2624) Remove ocean daily files (NOAA-EMC#2689) Update Jenkinsfile Add Hercules-EMC to the Jenkins configurable parameter list (NOAA-EMC#2685) Update gdas.cd and gsi_utils hashes (NOAA-EMC#2641) Add ability to use GEFS replay ICs (NOAA-EMC#2559) Replace `sleep` with `wait_for_file` (NOAA-EMC#2586) Add COM template for JEDI obs (NOAA-EMC#2678) Link both global-nest fix files and non-nest ones at the same time (NOAA-EMC#2632) Update ufs-weather-model (NOAA-EMC#2663) Add ability to process ocean/ice products specific to GEFS (NOAA-EMC#2561) Update cleanup job to use COMIN/COMOUT (NOAA-EMC#2649) Add overwrite to creat experiment in BASH CI (NOAA-EMC#2676) Add handling to select CRTM cloud optical table based on cloud scheme and update calcanal_gfs.py (NOAA-EMC#2645) Refs NOAA-EMC#2475

Add a prepaeroobs job to prepare aerosol obs files for DA. This job does quality control of the VIIRS aerosol raw observations and convert them to ioda format. Resolves #2623 --------- Co-authored-by: ypwang19 <yaping.wang@users.noreply.github.com> Co-authored-by: TerrenceMcGuinness-NOAA <terrence.mcguinness@cox.net> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

initial commit to add the prepaeroobs job

f7ea78c

ypwang19 marked this pull request as draft May 23, 2024 20:23

ypwang19 and others added 11 commits May 24, 2024 13:49

update yaml file name

9d5aeb7

Merge branch 'develop' into feature/aero_prepobs

a8faff2

add option to trun on/off prepaeroobs

bdc5338

add option to turn on/off prepaeroobs

c014ae1

fix commit

5c31d4a

fix previous commit

1b7482a

fix previous commit

c224002

fix coding norms issues

8beebee

update env

18f11fb

github-advanced-security bot found potential problems May 30, 2024

View reviewed changes

env/HERCULES.env Fixed Show fixed Hide fixed

ypwang19 added 3 commits May 30, 2024 20:00

fix shellcheck

7a631d9

archive obs file

8f78765

fix coding norms check

25616c4

ypwang19 marked this pull request as ready for review May 30, 2024 21:13

revert

8bc479a

CoryMartin-NOAA reviewed May 31, 2024

View reviewed changes

env/HERA.env Outdated Show resolved Hide resolved

WalterKolczynski-NOAA requested changes May 31, 2024

View reviewed changes

CoryMartin-NOAA reviewed May 31, 2024

View reviewed changes

WalterKolczynski-NOAA reviewed May 31, 2024

View reviewed changes

ush/python/pygfs/task/aero_prepobs.py Outdated Show resolved Hide resolved

update jobs name to prepobsaero

03324f5

github-advanced-security bot found potential problems May 31, 2024

View reviewed changes

jobs/JGLOBAL_PREP_OBS_AERO Fixed Show fixed Hide fixed

jobs/JGLOBAL_PREP_OBS_AERO Fixed Show fixed Hide fixed

jobs/JGLOBAL_PREP_OBS_AERO Fixed Show fixed Hide fixed

fix shellcheck

3db2938

specify location of sensor in filename

8fa0ecd

add double quote

d0aa0da

ypwang19 requested a review from WalterKolczynski-NOAA June 6, 2024 19:43

Merge branch 'develop' into feature/aero_prepobs

f5bd885

DavidHuber-NOAA self-requested a review June 14, 2024 12:52

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

env/S4.env Show resolved Hide resolved

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

env/WCOSS2.env Outdated Show resolved Hide resolved

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

ush/python/pygfs/task/aero_prepobs.py Outdated Show resolved Hide resolved

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

ush/python/pygfs/task/aero_prepobs.py Outdated Show resolved Hide resolved

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

ush/python/pygfs/task/aero_prepobs.py Show resolved Hide resolved

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

ush/python/pygfs/task/aero_prepobs.py Outdated Show resolved Hide resolved

DavidHuber-NOAA requested changes Jun 14, 2024

View reviewed changes

CoryMartin-NOAA and others added 5 commits June 14, 2024 14:34

Merge branch 'develop' into feature/aero_prepobs

adbe97c

Update ush/python/pygfs/task/aero_prepobs.py

a941154

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update env/WCOSS2.env

cac1e3d

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/python/pygfs/task/aero_prepobs.py

75b6d6b

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/python/pygfs/task/aero_prepobs.py

5dd514b

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

DavidHuber-NOAA reviewed Jun 14, 2024

View reviewed changes

Update config.base

8d0ce42

DavidHuber-NOAA approved these changes Jun 14, 2024

View reviewed changes

CoryMartin-NOAA mentioned this pull request Jun 14, 2024

Clean up prep aero obs job #2691

Open

aerorahul reviewed Jun 14, 2024

View reviewed changes

Merge branch 'develop' into feature/aero_prepobs

7f6d171

DavidHuber-NOAA merged commit 6c93b45 into NOAA-EMC:develop Jun 14, 2024
4 checks passed

aerorahul mentioned this pull request Jun 14, 2024

Update archive job to use COMIN/COMOUT #2668

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add observation preparation job for aerosols DA to workflow #2624

Add observation preparation job for aerosols DA to workflow #2624

ypwang19 commented May 23, 2024 •

edited

Loading

ypwang19 commented May 31, 2024

WalterKolczynski-NOAA May 31, 2024

CoryMartin-NOAA May 31, 2024

WalterKolczynski-NOAA Jun 7, 2024

CoryMartin-NOAA left a comment

CoryMartin-NOAA May 31, 2024

CoryMartin-NOAA May 31, 2024

CoryMartin-NOAA May 31, 2024

CoryMartin-NOAA May 31, 2024

CoryMartin-NOAA Jun 5, 2024

WalterKolczynski-NOAA May 31, 2024

CoryMartin-NOAA commented Jun 5, 2024

DavidHuber-NOAA Jun 14, 2024 •

edited

Loading

CoryMartin-NOAA Jun 14, 2024

DavidHuber-NOAA Jun 14, 2024

CoryMartin-NOAA commented Jun 14, 2024

DavidHuber-NOAA left a comment

CoryMartin-NOAA commented Jun 14, 2024

DavidHuber-NOAA Jun 14, 2024

CoryMartin-NOAA Jun 14, 2024

DavidHuber-NOAA left a comment

DavidHuber-NOAA Jun 14, 2024

DavidHuber-NOAA Jun 14, 2024

DavidHuber-NOAA Jun 14, 2024

aerorahul Jun 14, 2024

aerorahul Jun 14, 2024

	Generate corrosponding YMAL file.
	Generate corresponding YMAL file.

	fstart = to_datetime(yyyy + '-' + mm + '-' + dd + 'T' + HH + ':' + MM + 'Z')
	fstart = to_datetime(f'{yyyy}-{mm}-{dd}T{HH}:{MM}Z')

Add observation preparation job for aerosols DA to workflow #2624

Add observation preparation job for aerosols DA to workflow #2624

Conversation

ypwang19 commented May 23, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

ypwang19 commented May 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CoryMartin-NOAA commented Jun 5, 2024

DavidHuber-NOAA Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CoryMartin-NOAA commented Jun 14, 2024

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA commented Jun 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ypwang19 commented May 23, 2024 •

edited

Loading

DavidHuber-NOAA Jun 14, 2024 •

edited

Loading