[BUG/ISSUE] Problems using RCP emissions

[jennyfisher]

Describe the bug

We are having difficulty running GEOS-Chem v12 with RCP emissions (tested with RCP4.5, but I imagine it will be the same for all). We have tried using v12.2 and also v12.8.0.

The first problem is specific to v12.8.0 and has to do with the time cycling flag in the HEMCO_Config.rc. It is set to ID which causes HEMCO to die almost immediately saying there is no such time cycling attribute (or something like that). If I change the flag back to I (as it was in our v12.2 code), we get past this error.

However, then we have another problem. HEMCO dies with the following:

-------------------------------------------------------------------------------
 Calculating emissions for species ALK4
-------------------------------------------------------------------------------
HEMCO: Entering GET_CURRENT_EMISSIONS ( 3)
      
 Evaluate field RCP45_ALK4__A
HEMCO: Leaving GET_CURRENT_EMISSIONS ( 3)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
HEMCO ERROR: Negative emissions in: RCP45_ALK4__A. To allow negatives, edit settings in the configuration file. 
ERROR LOCATION: HCO_CalcEmis (HCO_CALC_MOD.F90)
ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

After setting verbose and warnings to 3, we see that emissions of RCP45_ALK4__A (and indeed all RCP emissions) are getting set to negative values:

    -->Array sum       :  -3.1680220E+34
    -->Array min & max :  -9.9999998E+30  0.0000000E+00

However, I've double checked the input files using cdo and they look fine:

[jaf574@gadi-login-03 RCP_45]$ cdo infon RCPs_anthro_butanes_2005-2100_27424_kgC.nc 
    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter name
     1 : 2005-01-01 00:00:00       0   259200       0 :      0.0000  4.3015e-13  5.8777e-10 : ACCMIP        
     2 : 2010-01-01 00:00:00       0   259200       0 :      0.0000  4.2124e-13  5.1093e-10 : ACCMIP        
     3 : 2020-01-01 00:00:00       0   259200       0 :      0.0000  4.0342e-13  3.5725e-10 : ACCMIP        
     4 : 2030-01-01 00:00:00       0   259200       0 :      0.0000  4.5282e-13  5.0675e-10 : ACCMIP        
     5 : 2040-01-01 00:00:00       0   259200       0 :      0.0000  4.5816e-13  5.9245e-10 : ACCMIP        
     6 : 2050-01-01 00:00:00       0   259200       0 :      0.0000  4.1945e-13  6.1433e-10 : ACCMIP        
     7 : 2060-01-01 00:00:00       0   259200       0 :      0.0000  3.9967e-13  6.5590e-10 : ACCMIP        
     8 : 2070-01-01 00:00:00       0   259200       0 :      0.0000  3.7171e-13  7.6631e-10 : ACCMIP        
     9 : 2080-01-01 00:00:00       0   259200       0 :      0.0000  3.3557e-13  9.4557e-10 : ACCMIP        
    10 : 2090-01-01 00:00:00       0   259200       0 :      0.0000  3.1400e-13  7.4416e-10 : ACCMIP        
    11 : 2100-01-01 00:00:00       0   259200       0 :      0.0000  2.9243e-13  5.4276e-10 : ACCMIP        

I thought it might be something to do with the interpolation, so I tried setting the flag to R instead of I. This time the run did not die with the negative values, but the HEMCO.log still shows the same large negatives for Array sum and Array min. The HEMCO_Diagnostics files then shows zero for all anthropogenic emissions.

We also tried compiling with FPE=y and BOUNDS=y and nothing changed.

To Reproduce

Run v12 out of the box with RCP_45 set to true and all other anthropogenic emissions turned off. In our case we are also running with Emission Year set to 2030, but I tried with 2007 and everything was the same.

Expected behavior

Run with RCP4.5 and get positive emissions!

Error messages

See above.

Required information

• GEOS-Chem version 12.8.0
• ifort 19.0.5.281
• netCDF 4.7.1
• NCI Gadi cluster
• "out of the box" code

Input and log files to attach

• The lastbuild file
• The input.geos file
• The HEMCO_Config.rc file
• The GEOS-Chem "Classic" log file
• The HEMCO.log file
  (These are the versions where I set the flag to R not ID or I.)
  forGCST.zip

[jennyfisher]

Hi @msulprizio @yantosca @christophkeller -- any thoughts on this issue? Thanks!

[yantosca]

I am also looping in @jimmielin. There have been a lot of updates to HEMCO recently (to facilitate connection with other models) so perhaps something got clobbered.

[jennyfisher]

Just a very minor addition to what I wrote before - using the C flag for the RCP files goes back to the original behaviour where the run dies because of the negative emissions (as it should!).

[yantosca]

Thanks for writing. I tried to replicate your issue in 12.9.1 and found a couple of other problems:

1. There was a bug in HEMCO module hco_config_mod.F90. We now use the "ID" flag (interpolate discontinuous data) instead of the older "I" flag (which is now used to interpolate regularly-spaced time data). But we had listed an option for ID in one IF block but forgot it in a second IF block. This is now fixed by commit: 4fff9da.

2. I looked at the HEMCO/RCP/v2015-02/ files and I saw that they weren't COARDS compliant. The isCoards script returns:

===========================================================================
Filename: RCPs_anthro_BC_2005-2100_27424.nc
===========================================================================

The following items adhere to the COARDS standard:
---------------------------------------------------------------------------
-> Dimension "lon" adheres to standard usage
-> Dimension "lat" adheres to standard usage

The following items DO NOT ADHERE to the COARDS standard:
---------------------------------------------------------------------------
-> Dimension "date" is non-standard usage,
   which will cause GCHP to fail during file read.
   PLEASE REMOVE THIS DIMENSION AND ALL VARIABLES THAT USE IT!!!

So I renamed the date dimension and date index variable to time, and we now get this output from isCoards:

===========================================================================
Filename: RCPs_anthro_BC_2005-2100_27424.nc
===========================================================================

The following items adhere to the COARDS standard:
---------------------------------------------------------------------------
-> Dimension "time" adheres to standard usage
-> Dimension "lat" adheres to standard usage
-> Dimension "lon" adheres to standard usage
-> ACCMIP(time,lat,lon)
-> ACCMIP:long_name = "ACCMIP, anthro_BC: kilograms of species per square metre and per second" 
-> ACCMIP:units = "kg/m2/s" 
-> ACCMIP:long_name = NaNf 
-> lat(lat)
-> lat is monotonically increasing
-> lat:axis = "Y" 
-> lat:long_name = "Latitude" 
-> lat:units = "degrees_north" 
-> lon(lon)
-> lon is monotonically increasing
-> lon:axis = "X" 
-> lon:long_name = "Longitude" 
-> lon:units = "degrees_east" 
-> time(time)
-> time is monotonically increasing
-> time:calendar = "standard" 
-> time:long_name = "Time" 
-> time:units = "days since 2005-01-01" 
-> conventions: "CF-1.0" 
-> history: "Wed Jul 15 13:46:03 2020: ncks --mk_rec_dmn time RCPs_anthro_BC_2005-2100_27424.nc RCPs_anthro_BC_2005-2100_27424.nc\nFri Jul 17 15:08:28 2015: cdo -s invertlat RCPs_anthro_BC_2005-2100_27424.nc tmp.nc\nOriginal data from each scenario working group, provided by ECCAD-GEIA database2010 Original dataset release date\n2010 Data online on ECCAD site\n" 
-> title: "anthro_BC (Black Carbon) - RCPs emission scenarios - Data between year 2005 and 2100 from RCPs (Scenario 4.5) distributed by ECCAD" 
-> references: "Original data site: http://accmip-emis.iek.fz-juelich.de/data/accmip/gridded_netcdf/ \nDocumentation: http://accmip-emis.iek.fz-juelich.de/data/accmip/gridded_netcdf/release_history_gridded_netcdf.txt" 

The following items DO NOT ADHERE to the COARDS standard:
---------------------------------------------------------------------------

The following optional items are RECOMMENDED:
---------------------------------------------------------------------------
-> Consider adding ACCMIP:missing_value
-> Consider adding ACCMIP:add_offset
-> Consider adding ACCMIP:scale_factor
-> Consider adding time:axis = "T"
-> Consider adding the "format" global attribute

So now, nothing is in the files that would violate the netCDF COARDS standard. I have uploaded the corrected files to HEMCO/RCP/v2020-07. You will need to change `

(((RCP_45
0 RCP45_CH4     $ROOT/RCP/v2015-02/RCP_45/RCPs_anthro_CH4_2005-2100_27424.nc                            ACCMIP 2005-2100/1/1/0 ID xy kg/m2/s  CH4   -     1 1
0 RCP45_NOx     $ROOT/RCP/v2015-02/RCP_45/RCPs_anthro_NOx_2005-2100_27424.nc                            ACCMIP 2005-2100/1/1/0 ID xy kg/m2/s  NO    -     1 1
... etc ...

to

(((RCP_45
0 RCP45_CH4     $ROOT/RCP/v2020-07/RCP_45/RCPs_anthro_CH4_2005-2100_27424.nc                            ACCMIP 2005-2100/1/1/0 ID xy kg/m2/s  CH4   -     1 1
0 RCP45_NOx     $ROOT/RCP/v2020-07/RCP_45/RCPs_anthro_NOx_2005-2100_27424.nc                            ACCMIP 2005-2100/1/1/0 ID xy kg/m2/s  NO    -     1 1
...etc...

in your HEMCO_Config.rc file.

We will push this fix to 12.9.1. In the meantime, modify your code so according to 4fff9da and see if it runs properly.

[yantosca]

With these fixes in GEOS-Chem 12.9.1, I was able to get a short simulation to run to completion. I don't know if that addresses the negative value issue though.

[yantosca]

This has also been fixed in the HEMCO repository with commit: geoschem/HEMCO@2d44eda

[jennyfisher]

Thanks Bob. I am trying to download the new files but I seem to be unable to access geoschemdata.computecanada.ca (either from my server or direct using URL). The status page I found (https://status.computecanada.ca/) implies there are no downtime issues - but I can't connect. Do you know if there has been a change to the data access? Thanks!

[jennyfisher]

In the meantime, I was able to download the files from the Harvard FTP, and with the updated files plus changes to the HEMCO code, it seems to be working properly! Negative values are gone, reasonable values are present, and HEMCO diagnostics are saving normal looking distributions to the diagnostics files. Thanks!! 👍

[msulprizio]

I was just able to transfer the new RCP files over to Compute Canada via Globus. If you have access to Globus, that may be a better way to transfer files from Compute Canada. We still need to add documentation for the Globus endpoint to the wiki, so let me know if you're interested and I can get right on that.

I also can't access http://geoschemdata.computecanada.ca/ExtData/ via a web browser. @Jun-Meng are you aware of any issues on their end?

[yantosca]

Also note, we have updated the configuration files in GCHPctm with commit: a34f1ba.

[jennyfisher]

Hi Melissa,

I've never heard of Globus - can you provide more information? (I'm good with the RCP files for now, but generally use the Compute Canada http site for other data files we are missing!).

Thanks!
Jenny

[Jun-Meng]

I was just able to transfer the new RCP files over to Compute Canada via Globus. If you have access to Globus, that may be a better way to transfer files from Compute Canada. We still need to add documentation for the Globus endpoint to the wiki, so let me know if you're interested and I can get right on that.

I also can't access http://geoschemdata.computecanada.ca/ExtData/ via a web browser. @Jun-Meng are you aware of any issues on their end?

Hi Melissa and Jenny,

The Compute Canada Virtual Machine (CCVM) we are using for archiving the met-fields is under maintenance during July 14 -16. It should be normal tomorrow.

Jun

[jennyfisher]

Great, thanks for clarifying Jun!

[msulprizio]

I've never heard of Globus - can you provide more information? (I'm good with the RCP files for now, but generally use the Compute Canada http site for other data files we are missing!).

@jennyfisher I've added some text for downloading files via Globus to our wiki page. See http://wiki.seas.harvard.edu/geos-chem/index.php/Downloading_data_from_Compute_Canada#File_transfers_with_Globus.

[jennyfisher]

Thanks @msulprizio - never heard of it, but it does appear my institution has a log-in!