[BUG/ISSUE] Potentially incorrect warning about species initial values #8

JiaweiZhuang · 2018-12-08T04:23:25Z

I notice this warning message even if the restart file initial_GEOSChem_rst.c24_standard.nc exists:

 Initialized species from INTERNAL state: NO
    WARNING: using background values from species database
 Initialized species from INTERNAL state: O3
    WARNING: using background values from species database
 Initialized species from INTERNAL state: PAN
    WARNING: using background values from species database
 Initialized species from INTERNAL state: CO
    WARNING: using background values from species database

Is this a wrong warning? From the output files it seems like correct initial conditions are used.

See, for example, the log file ubuntu_timing_1day.log in #6

The text was updated successfully, but these errors were encountered:

yantosca · 2018-12-10T22:18:46Z

This is happening in Chem_GridCompMod.F90:

         ! Set spc conc to background value if rst skipped or var not there
         IF ( RC /= ESMF_SUCCESS .OR. RST == MAPL_RestartBootstrap .OR.   &
                      RST == MAPL_RestartSkipInitial ) THEN
                DO L = 1, LLPAR 
                DO J = 1, JJPAR
                DO I = 1, IIPAR
                   ! Special handling for MOH (mimicking GEOS-Chem Classic)
                   IF ( TRIM( ThisSpc%Name ) == 'MOH' ) THEN
                      ! Test for altitude (L < 9 is always in the trop)
                      IF ( L <= 9 ) THEN
                         ! Test for ocean/land boxes
                         IF ( State_Met%FRCLND(I,J) >= 0.5 ) THEN
                            ! Continental boundary layer: 2 ppbv MOH
                            State_Chm%Species(I,J,L,IND) = 2.000e-9_fp
                         ELSE
                            ! Marine boundary layer: 0.9 ppbv MOH
                            State_Chm%Species(I,J,L,IND) = 0.900e-9_fp
                         ENDIF
                      ELSE
                         ! Test for troposphere
                         IF ( State_Met%InTroposphere(I,J,L) ) THEN
                            ! Free troposphere: 0.6 ppbv MOH
                            State_Chm%Species(I,J,L,IND) = 0.600e-9_fp 
                         ELSE
                            ! Strat/mesosphere:
                            State_Chm%Species(I,J,L,IND) = 1.0E-30_FP 
                         ENDIF
                      ENDIF
                   ELSEIF ( L > LLCHEM .AND. &
                            ( .NOT. ThisSpc%Is_Advected ) ) THEN
                      ! For non-advected spc at L > LLCHEM, use small number
                      State_Chm%Species(I,J,L,IND) = 1.0E-30_FP           
                   ELSE
                      ! For all other cases, use the background value in spc db
                      State_Chm%Species(I,J,L,IND) = ThisSpc%BackgroundVV 
                   ENDIF
                ENDDO
                ENDDO
                ENDDO
                Ptr3D_R8(:,:,:) = State_Chm%Species(:,:,LM:1:-1,IND)
                IF ( MAPL_am_I_Root()) THEN
                   WRITE(*,*)  &
                   '   WARNING: using background values from species database'
                ENDIF
             ENDIF

Maybe the warning message is useless by now and we could remove it. Did this come in with the GEOS-5 updates?

lizziel · 2018-12-10T23:54:01Z

I have not been able to reproduce these warning messages for advected species using a full chemistry simulation with GCHP on the Odyssey cluster. I only get these messages for internal state variables NOT in the restart file, such as recently added species pFe and HEMCO restart variables. This indicates the warnings are being issued correctly for my particular run. This code was put into GCHP in March 2018 and was not part of a GEOS-5 update.

I notice that your log file includes messages like this for every single internal state variable:
Bootstrapping Variable: SPC_NMAO3 in initial_GEOSChem_rst.c24_standard.nc
Bootstrapping Variable: SPC_NO in initial_GEOSChem_rst.c24_standard.nc
Bootstrapping Variable: SPC_NO2 in initial_GEOSChem_rst.c24_standard.nc
Bootstrapping Variable: SPC_NO3 in initial_GEOSChem_rst.c24_standard.nc
Bootstrapping Variable: SPC_NPMN in initial_GEOSChem_rst.c24_standard.nc

If the restart file is read correctly then you should see a much shorter list, specifically this in 12.1.0, just with the restart filename swapped out:

Using parallel NetCDF for file: initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: ARCHV_DRY_TOTN in initial_GEOSChem_rst.c48_benchmark.
nc
Bootstrapping Variable: ARCHV_WET_TOTN in initial_GEOSChem_rst.c48_benchmark.
nc
Bootstrapping Variable: DEP_RESERVOIR in initial_GEOSChem_rst.c48_benchmark.n
c
Bootstrapping Variable: DRYPERIOD in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: GCCTROPP in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: GWET_PREV in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: LAI_PREVDAY in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: PARDF_DAVG in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: PARDR_DAVG in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: PFACTOR in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: SPC_LCH4 in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: SPC_LOx in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: SPC_PH2O2 in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: SPC_POx in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: SPC_pFe in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: STATE_PSC in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: T_DAVG in initial_GEOSChem_rst.c48_benchmark.nc
Bootstrapping Variable: T_PREVDAY in initial_GEOSChem_rst.c48_benchmark.nc

Based on code in Chem_GridCompMod right before the conditional in which the background value warning message is issued, it appears that the call to ESMF_AttributeGet for the 'RESTART' attribute is returning MAPL_RestartBootstrap which is one of the conditions that will lead to using background values from the species database. We know the attribute is that value due to the bootstrapping warnings that are in the log file. Those warnings, along with the restart attribute per internal state variable, were created by MAPL earlier in the execution.

         ! Determine if species in restart file
         CALL ESMF_StateGet( INTERNAL, TRIM(SPFX) // TRIM(ThisSpc%Name),  &
              trcFIELD, RC=RC )
         CALL ESMF_AttributeGet( trcFIELD, NAME="RESTART",                &
              VALUE=RST, RC=STATUS )
   
         ! Set spc conc to background value if rst skipped or var not there
         IF ( RC /= ESMF_SUCCESS .OR. RST == MAPL_RestartBootstrap .OR.   &
                  RST == MAPL_RestartSkipInitial ) THEN

Check that your restart file exists in the run directory and contains the content you think it does.

JiaweiZhuang · 2018-12-11T00:26:57Z

Found the issue.

I was using the latest GEOSCHEM_RESTARTS/v2018-11/initial_GEOSChem_rst.c24_standard.nc, which uses variable name SpeciesRst_O3, but GCHP 12.1.0 still looks for SPC_O3. Using the old restart file GEOSCHEM_RESTARTS/v2016-07/initial_GEOSChem_rst.c24_standard.nc fixes the problem.

See geoschem/GCHP#8

JiaweiZhuang · 2018-12-11T02:02:22Z

Oops the situation actually gets worse.

With the "proper" restart file v2016-07/initial_GEOSChem_rst.c24_standard.nc, the simulation always crashes at hour 1 (when the diagnostics first get written), even if I set the simulation period to 2 days. This is the default setting in ami-0a5973f14aad7413a

The simulation is able to proceed with the "improper" restart file v2018-11/initial_GEOSChem_rst.c24_standard.nc. This is the default setting in ami-0f44e999c80ef6e66.

…rash See geoschem/GCHP#8 (comment)

lizziel · 2018-12-11T15:11:48Z

We should delete any GCHP restart files that are in that folder since the older files should be used instead. I will replace them with symbolic links to the older files in case people go to that folder looking for restarts. We do not plan on changing the internal state name in GCHP so SPC_ is the correct name to use. Thanks for reporting that!

Could you provide your log file for your latest run problem?

msulprizio · 2018-12-11T15:13:16Z

Hi all! I agree with Lizzie and I meant to do that earlier. Sorry I dropped the ball. I'll fix it now.

lizziel · 2018-12-11T15:15:44Z

Okay, see this new issue and the logs are posted in geoschem/GCHP#6. Let's keep the discussion of that issue to here rather than in two places. The printing of timing values issue is unrelated.

msulprizio · 2018-12-11T15:19:30Z

The GCHP restart files are now removed from ExtData/GEOSCHEM_RESTARTS/v2018-11. It may take a while to sync to the Harvard ftp and to AWS.

lizziel · 2018-12-11T15:21:33Z

Thanks Melissa! Jiawei, try turning off all collections. If that works, then turn on one collection, e.g. SpeciesConc, and check if results are okay. If that all works please open a new issue related to writing multiple files on the cloud.

yantosca · 2018-12-11T22:31:05Z

I logged into the cloud with AMI: GCHP12.1.0_tutorial_20181210 (ami-0f44e999c80ef6e66)

When I run with all collections turned, off, I get an error in offline tracer advection at 00:10 into the run.

When I run with only SpeciesConc_avg and SpeciesConc_inst turned on, then we still get the bad termination error right after the GIGCenv time is printed out.

Will dig a bit further.

JiaweiZhuang · 2018-12-11T23:16:32Z

@yantosca Thanks for looking into this! I have the same finding, which is quite weird:

0 output: crashes at 00:10
1, 2, 3 outputs (Choose from SpeciesConc_avg, SpeciesConc_inst, StateMet_avg): proceeds and generates output files without problems. The final timing info is incomplete, but it is a separate issue.
4 outputs (SpeciesConc_avg, SpeciesConc_inst, StateMet_avg, StateMet_inst): crashes at 01:00 when writing the first output. This can be "solved" by using the incorrect restart file v2018-11, for unknown reasons.

lizziel · 2018-12-12T16:06:16Z

I suspect these are three separate issues all unrelated to the restart file. I think we should close this issue (incorrect warning about species initial values) and create one or two new issues for the others.

…

-- Lizzie Lundgren Scientific Programmer GEOS-Chem Support Team geos-chem-support@as.harvard.edu<mailto:geos-chem-support@as.harvard.edu> http://wiki.geos-chem.org/GEOS-Chem_Support_Team Please direct all GEOS-Chem support issues to the entire GEOS-Chem Support Team at geos-chem-support@as.harvard.edu<mailto:geos-chem-support@as.harvard.edu>. This will allow us to serve you better. From: Jiawei Zhuang <notifications@github.com> Reply-To: geoschem/gchp <reply@reply.github.com> Date: Tuesday, December 11, 2018 at 6:16 PM To: geoschem/gchp <gchp@noreply.github.com> Cc: "Lundgren, Elizabeth W" <elundgren@seas.harvard.edu>, Comment <comment@noreply.github.com> Subject: Re: [geoschem/gchp] Potentially incorrect warning about species initial values (#8) @yantosca<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_yantosca&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=VyyBl8eF72Uw5bhJwoVODdKCdZCT8Qcap6kbrQAfIto&s=frU-eTznxzsbHlL6rMaAmcLzdI0ov9OpXZCVlJux39k&e=> Thanks for looking into this! I have the same finding, which is quite weird: * 0 output: crashes at 00:10 * 1, 2, 3 outputs (Choose from SpeciesConc_avg, SpeciesConc_inst, StateMet_avg): proceeds and generate output files without problems. The final timing info is incomplete, but it is a separate issue. * 4 outputs (SpeciesConc_avg, SpeciesConc_inst, StateMet_avg, StateMet_inst): crashes at 01:00 when writing the first output. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geoschem_gchp_issues_8-23issuecomment-2D446399608&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=VyyBl8eF72Uw5bhJwoVODdKCdZCT8Qcap6kbrQAfIto&s=p7VTU33gjq2g7rpJY4Z7d7DabK-JaJW6DLvWENSmO84&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAnyqyEWdxgMryAMvvMBIHw3fIo3UlIHks5u4DzQgaJpZM4ZJdEl&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=VyyBl8eF72Uw5bhJwoVODdKCdZCT8Qcap6kbrQAfIto&s=fwYMuwE9itIOIZy4lQ5Z8jIupe3NLwFtiav5OqQAdY4&e=>.

JiaweiZhuang · 2018-12-12T19:16:58Z

Agreed @lizziel

JiaweiZhuang referenced this issue in geoschem/geos-chem-cloud Dec 11, 2018

Pull older GCHP restart file with correct variable name

60d3332

See geoschem/GCHP#8

JiaweiZhuang referenced this issue in geoschem/geos-chem-docker Dec 11, 2018

Fix GCHP restart file name

58cd562

See geoschem/GCHP#8

JiaweiZhuang mentioned this issue Dec 11, 2018

[BUG/ISSUE] Timing results are incomplete #6

Closed

JiaweiZhuang referenced this issue in geoschem/geos-chem-docker Dec 11, 2018

Roll back to original restart file so the simulation at least won't c…

beb7830

…rash See geoschem/GCHP#8 (comment)

JiaweiZhuang referenced this issue in geoschem/geos-chem-cloud Dec 11, 2018

Roll back to original restart file so the simulation at least won't c…

81ef271

…rash See geoschem/GCHP#8 (comment)

JiaweiZhuang closed this as completed Dec 12, 2018

JiaweiZhuang mentioned this issue Dec 12, 2018

[BUG/ISSUE] Early termination at different points depending on diagnostics configuration #10

Closed

msulprizio changed the title ~~Potentially incorrect warning about species initial values~~ [BUG/ISSUE] Potentially incorrect warning about species initial values Sep 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG/ISSUE] Potentially incorrect warning about species initial values #8

[BUG/ISSUE] Potentially incorrect warning about species initial values #8

JiaweiZhuang commented Dec 8, 2018

yantosca commented Dec 10, 2018

lizziel commented Dec 10, 2018

JiaweiZhuang commented Dec 11, 2018

JiaweiZhuang commented Dec 11, 2018

lizziel commented Dec 11, 2018

msulprizio commented Dec 11, 2018

lizziel commented Dec 11, 2018

msulprizio commented Dec 11, 2018 •

edited

lizziel commented Dec 11, 2018

yantosca commented Dec 11, 2018

JiaweiZhuang commented Dec 11, 2018 •

edited

lizziel commented Dec 12, 2018 via email

JiaweiZhuang commented Dec 12, 2018

[BUG/ISSUE] Potentially incorrect warning about species initial values #8

[BUG/ISSUE] Potentially incorrect warning about species initial values #8

Comments

JiaweiZhuang commented Dec 8, 2018

yantosca commented Dec 10, 2018

lizziel commented Dec 10, 2018

JiaweiZhuang commented Dec 11, 2018

JiaweiZhuang commented Dec 11, 2018

lizziel commented Dec 11, 2018

msulprizio commented Dec 11, 2018

lizziel commented Dec 11, 2018

msulprizio commented Dec 11, 2018 • edited

lizziel commented Dec 11, 2018

yantosca commented Dec 11, 2018

JiaweiZhuang commented Dec 11, 2018 • edited

lizziel commented Dec 12, 2018 via email

JiaweiZhuang commented Dec 12, 2018

msulprizio commented Dec 11, 2018 •

edited

JiaweiZhuang commented Dec 11, 2018 •

edited