v2023-06-BCs, drop TROPOMI albedo filters, and Rocky 8 on Cannon #130

nicholasbalasus · 2023-06-11T14:41:27Z

Name and Institution

Name: Nick Balasus
Institution: Harvard University

Describe the update

This update does the following:

Allows the IMI to run on Cannon after the upgrade from CentOS 7 to Rocky 8 (used Bob's new environment files from Update Harvard Cannon environment files and Conda files for RockyLinux #114 but left our imi_env.yml as is).
Removes the albedo filters for TROPOMI data.
Updates the boundary condition generation scripts so that if you generate boundary conditions up until day x, TROPOMI and GC data is used until day x+15 to make our temporal smoothing valid (and to make the BCs more reproducible). I generate out to 20 days past just to be careful (and because we will probably be doing this on a monthly basis).

laestrada

This looks good, but I have added some comments about a few things related to maintaining use for non-cannon users. Specifically adding a few more config variables for writing the boundary conditions.

laestrada · 2023-06-11T22:40:19Z

run_imi.sh

@@ -50,6 +50,9 @@ if ! "$isAWS"; then
    printf "\nActivating conda environment: ${CondaEnv}\n"
    eval "$(conda shell.bash hook)"
    conda activate $CondaEnv
+
+    # Set GEOS-Chem environment
+    GEOSChemEnv="$(pwd -P)/envs/Harvard-Cannon/gcclassic.rocky+gnu12.minimal.env"


GEOSChemEnv should probably remain a config variable since this won't be the same path for all local cluster users.

Any ideas on how to do this well? It would feel kind of awkward for a Harvard user to have to change it to /path/to/their/imi/clone/envs/Harvard-Cannon/gcclassic.rocky+gnu12.minimal.env every time given that the environment already exists in their cloned version.

On second thought, I think it is fine to expect Harvard users to have it somewhere in their home directory (and they could just copy from envs/Harvard-Cannon if not). I'll make the change.

I think it is okay to have cannon users have to update the config file for their specific path and we can document this in the google doc for cannon users. I would think most users are only downloading and doing the initial set up once.

Also btw using

GEOSChemEnv: "$(pwd -P)/envs/Harvard-Cannon/gcclassic.rocky+gnu12.minimal.env"

in the yaml config file should still work.

laestrada · 2023-06-11T22:40:34Z

src/components/posterior_component/posterior.sh

@@ -122,11 +122,6 @@ run_posterior() {
    python setup_gc_cache.py $StartDate $EndDate $GCsourcepth $GCDir; wait
    printf "\n=== DONE -- setup_gc_cache.py ===\n"



Thanks for removing this!

laestrada · 2023-06-12T12:01:17Z

src/write_BCs/run_BCs.sh

@@ -44,17 +59,21 @@ if "$RunGEOSChem"; then

    # Run GEOS-Chem
    cd ${workdir}/runGCC1402/
-    cp /n/holylfs05/LABS/jacob_lab/imi/ch4/boundary-conditions/restarts/GEOSChem.Restart.${startdate:0:8}_0000z.nc4 ./Restarts/
+    cp /n/holylfs05/LABS/jacob_lab/imi/ch4/tropomi-boundary-conditions/v2023-04/restarts/GEOSChem.Restart.${startdate:0:8}_0000z.nc4 ./Restarts/


Can we add this to the config file? eg. RestartsDir: p /n/holylfs05/LABS/jacob_lab/imi/ch4/tropomi-boundary-conditions/v2023-04/restarts/ or maybe just specify the restart in full?

This way non cannon users could use these scripts.

laestrada · 2023-06-12T12:08:18Z

src/write_BCs/run_BCs.sh

-    sbatch -W -p seas_compute -t 2-00:00 --mem 64000 -c 48 --wrap "source ~/.bashrc; conda activate $CondaEnv; python write_tropomi_GC_daily_avgs.py"; wait;
-    sbatch -W -p seas_compute -t 2-00:00 --mem 64000 --wrap "source ~/.bashrc; conda activate $CondaEnv; python calculate_bias.py"; wait;
-    sbatch -W -p seas_compute -t 2-00:00 --mem 64000 --wrap "source ~/.bashrc; conda activate $CondaEnv; python write_boundary.py"; wait;
+    sbatch -W -p huce_cascade,huce_bigmem,bigmem -t 2-00:00 --mem 190000 -c 48 --wrap "source ~/.bashrc; conda activate $CondaEnv; python write_tropomi_GC_daily_avgs.py"; wait;


Any objections to also adding the list of partitions to the config file?

laestrada · 2023-06-12T12:14:57Z

src/write_BCs/run_BCs.sh


 if "$RunGEOSChem"; then
    # Load modules
-    source ${imidir}/envs/Harvard-Cannon/gcc.gfortran10.2_cannon.env
+    source ${imidir}/envs/Harvard-Cannon/gcclassic.rocky+gnu12.minimal.env


can we make this a config vara? GEOSChemEnv?

nicholasbalasus · 2023-06-12T15:30:37Z

Comments addressed @laestrada

laestrada

Looks great, thanks Nick!

sabourbaray · 2023-06-12T19:04:05Z

Hi, I'm curious what the reasoning is for removing the albedo filters? Are there new recommendations for filtering? Thanks.

djvaron · 2023-06-12T20:12:56Z

Hi, I'm curious what the reasoning is for removing the albedo filters? Are there new recommendations for filtering? Thanks.

Hi Sabour, thanks for the question. There are a few reasons.

The latest TROPOMI product deals better with spectral variability in albedo and includes a post-processing albedo correction for low-albedo errors. And since the QA filtering already removes retrievals with very low SWIR albedo (a < 0.02 iirc), SRON recommended removing the additional SWIR filter we previously used (a < 0.05).

For the blended albedo filter (ba > 0.85), Lorente et al. (2021) showed that it removes retrievals in snowy scenes that would be subject to artifacts, but we've also seen it remove non-snowy retrievals at different latitudes and times of year. We aren't aware of any careful analysis of the filter's global effects, so we decided to remove it when constructing our global boundary condition fields.

What do you think? Do you have concerns about this for applications over Canadian domains? We're open to suggestions!

sabourbaray · 2023-06-12T21:02:34Z

Hi Daniel,

Thanks for the update.

The latest TROPOMI product deals better with spectral variability in albedo and includes a post-processing albedo correction for low-albedo errors. And since the QA filtering already removes retrievals with very low SWIR albedo (a < 0.02 iirc), SRON recommended removing the additional SWIR filter we previously used (a < 0.05).

That makes sense. In our case, allowing data with SWIR albedo 0.02 < a < 0.05 will give us more data to work with in upper latitudes.

For the blended albedo filter (ba > 0.85), Lorente et al. (2021) showed that it removes retrievals in snowy scenes that would be subject to artifacts, but we've also seen it remove non-snowy retrievals at different latitudes and times of year. We aren't aware of any careful analysis of the filter's global effects, so we decided to remove it when constructing our global boundary condition fields.

What do you think? Do you have concerns about this for applications over Canadian domains? We're open to suggestions!

This is helpful to know. I can include some sensitivity inversions in my work with and without the blended albedo filter. In my fork (under operational-eccc/v1.2), I'm using the surface classification to experiment with filters:

& ((tropomi_data["surface_classification"].astype(int) & 0x03) == 0)

This is likely too aggressive, but it helps us test without data that is contaminated by small lakes scattered in the domain. I think an ideal setup would be something in between, where we can select for data that is >80% land. This has been explored by de Foy et al. (2023), but it involves creating a high resolution water mask.

The surface classification variable also has a bit for "Land+Snow_Or_Ice", so something like:

& ((tropomi_data["surface_classification"].astype(int) & 0xF9) != 184)

Could also work.

nicholasbalasus added 6 commits June 1, 2023 15:15

remove albedo filters and ensure BCs use 15 days past final BC date

7497dd6

run 20 days past to be cautious; change partitions used

4ff3c24

updates for RockyLinux

8598607

remove NCOEnv and GEOSChemEnv from config file

d2bcd67

longer GC run time

6d8757f

remove NCOEnv code

72a532f

nicholasbalasus requested a review from laestrada June 11, 2023 14:41

laestrada reviewed Jun 12, 2023

View reviewed changes

address Lucas comments

4be1321

laestrada approved these changes Jun 12, 2023

View reviewed changes

laestrada mentioned this pull request Jun 12, 2023

Update Harvard Cannon environment files and Conda files for RockyLinux #114

Closed

laestrada merged commit d1e7081 into dev Jun 13, 2023

laestrada deleted the feature/v2023-06-BCs branch June 13, 2023 12:18

msulprizio added this to the IMI 1.2.1 milestone Aug 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2023-06-BCs, drop TROPOMI albedo filters, and Rocky 8 on Cannon #130

v2023-06-BCs, drop TROPOMI albedo filters, and Rocky 8 on Cannon #130

nicholasbalasus commented Jun 11, 2023

laestrada left a comment

laestrada Jun 11, 2023

nicholasbalasus Jun 12, 2023

nicholasbalasus Jun 12, 2023

laestrada Jun 12, 2023

laestrada Jun 11, 2023

laestrada Jun 12, 2023

laestrada Jun 12, 2023

laestrada Jun 12, 2023

nicholasbalasus commented Jun 12, 2023

laestrada left a comment

sabourbaray commented Jun 12, 2023

djvaron commented Jun 12, 2023

sabourbaray commented Jun 12, 2023 •

edited

		@@ -122,11 +122,6 @@ run_posterior() {
		python setup_gc_cache.py $StartDate $EndDate $GCsourcepth $GCDir; wait
		printf "\n=== DONE -- setup_gc_cache.py ===\n"

v2023-06-BCs, drop TROPOMI albedo filters, and Rocky 8 on Cannon #130

v2023-06-BCs, drop TROPOMI albedo filters, and Rocky 8 on Cannon #130

Conversation

nicholasbalasus commented Jun 11, 2023

Name and Institution

Describe the update

laestrada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicholasbalasus commented Jun 12, 2023

laestrada left a comment

Choose a reason for hiding this comment

sabourbaray commented Jun 12, 2023

djvaron commented Jun 12, 2023

sabourbaray commented Jun 12, 2023 • edited

sabourbaray commented Jun 12, 2023 •

edited