Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NEXUS failed for both retro and NRT runs #50

Closed
JianpingHuang-NOAA opened this issue Dec 14, 2022 · 12 comments
Closed

NEXUS failed for both retro and NRT runs #50

JianpingHuang-NOAA opened this issue Dec 14, 2022 · 12 comments
Labels
bug Something isn't working

Comments

@JianpingHuang-NOAA
Copy link
Collaborator

Description:

NEXUS failed for the NRT runs, then I checked out the latest online-cmaq workflow last night and still seeing failures of running NEXUS

  1. retro example on Dogwood
    /lfs/h2/emc/ptmp/jianping.huang/emc.para/output/20221110
    nexus_emission_2022111006_s00.id_1671021056.log
    nexus_emission_2022111006_s01.id_1671021056.log
    nexus_emission_2022111006_s02.id_1671021056.log

  2. NRT runs
    /lfs/h2/emc/ptmp/jianping.huang/emc.para/output/20221213
    nexus_emission_2022121312_s01.id_1670965539.log
    nexus_emission_2022121312_s00.id_1670965539.log

Please fix the issue as soon as possible. @bbakernoaa

Thanks !
@JianpingHuang-NOAA JianpingHuang-NOAA added the bug Something isn't working label Dec 14, 2022
@chan-hoo
Copy link

@JianpingHuang-NOAA, We can't access your 'data' directory. dlogin01:/lfs/h2/emc/physics/noscrub/jianping.huang> cd data/
-bash: cd: data/: Permission denied
drwxr-sr-- 13 jianping.huang physics 4096 Dec 10 02:06 data. Please change its permission. It should be "drwxr-sr-x". Please run " chmod -R o+x data ". In addition, the latest hash was not tested completely due to the above data access issue. Please check out the hash 'e769b69'.

@JianpingHuang-NOAA
Copy link
Collaborator Author

@chan-hoo Thanks. The permission was done. What does the hash 'e769b69' represent?

@bbakernoaa
Copy link
Contributor

bbakernoaa commented Dec 14, 2022 •
edited

@JianpingHuang-NOAA I ran the 2022-12-13 12:00 case on dogwood and didn't have an issue.

12/14/22 14:58:43 UTC :: FV3LAM_wflow.xml :: WARNING: <nodesize> support is deprecated, please use <nodes> to specify the requested resources
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202212131200       nexus_emission_00                    33033172           SUCCEEDED                   0         1         979.0
202212131200       nexus_emission_01                    33033176           SUCCEEDED                   0         1         986.0
202212131200       nexus_emission_02                    33033173           SUCCEEDED                   0         1         973.0
202212131200        nexus_post_split    druby://dlogin02.dogwood.wcoss2.ncep.noaa.gov:35839          SUBMITTING                   -         0           0.0
202212131200           fire_emission                    33033174              QUEUED                   -         0           0.0
202212131200            point_source                    33033175           SUCCEEDED                   0         1         713.0
202212131200           get_extrn_ics                    33033177              QUEUED                   -         0           0.0
202212131200          get_extrn_lbcs                    33033178              QUEUED                   -         0           0.0
202212131200                make_ics                           -                   -                   -         -             -
202212131200               make_lbcs                           -                   -                   -         -             -
202212131200                aqm_lbcs                           -                   -                   -         -             -
202212131200                run_fcst                           -                   -                   -         -             -
202212131200           pre_post_stat                           -                   -                   -         -             -
202212131200            post_stat_o3                           -                   -                   -         -             -
202212131200          post_stat_pm25                           -                   -                   -         -             -

I haven't been able to use your emission directory due to permission issues and so I had copied over the emission data needed to run AQM yesterday here: /lfs/h2/emc/lam/noscrub/barry.baker/emissions/nexus

Again I didn't see any issues running this day pointing to my emission directory. You can see the output here: /lfs/h2/emc/ptmp/barry.baker/expt_dirs/aqm20221213

@JianpingHuang-NOAA
Copy link
Collaborator Author

@bbakernoaa I got a failure too when I pointed my script to your emission dir.

Here is the run log file (Dogwood)

/lfs/h2/emc/ptmp/jianping.huang/emc.para/output/20221109 > vim nexus_emission_2022110906_s02.id_1671030435.log

Can you take a look?

In addition, my nexus is located at /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.21/sorc/arl_nexus

"git log" shows
commit 5c1fa371c76a72d3d263cd27aabead914f397cc0
Merge: 8b5ef7e fe5778a
Author: Barry Baker bbakernoaa@users.noreply.github.com
Date: Tue Nov 29 15:59:42 2022 -0500

Merge pull request #21 from noaa-oar-arl/feature/nexus_split

change from xarray to netcdf4-python for all machines

commit fe5778a0464bac8a60d140e72b9279b846e97621

Is this the right version?

Thanks,

@JianpingHuang-NOAA
Copy link
Collaborator Author

@chan-hoo @bbakernoaa NEXUS ran successfully with Barry's data for the large domain on 20221213 at 00z cycle but failed to run with same data for the grid_spec_RRFS_CONUS_13km or G775. I changed the HEMCO_sa_Grid.rc
to us the same grid numbers that we ran the model for CONUS, but still failed. Is there any change that I need to make in order to run NEXUS for the CONUS domain?

@JianpingHuang-NOAA
Copy link
Collaborator Author

The revised HEMCO_sa_Grid.rc is located on Dogwood at

/lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.24/sorc/arl_nexus/config/cmaq

@JianpingHuang-NOAA
Copy link
Collaborator Author

The config.yaml file can be found at /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.24/ush (Dogwood)

@bbakernoaa
Copy link
Contributor

We shouldn’t need to change anything for the conus domain. The nexus grid is the intermediate grid and not the final one. So having a larger intermediate grid won’t matter as long as the domain is small and completely contained within.

@bbakernoaa
Copy link
Contributor

I believe this has been solved. We should close it

@JianpingHuang-NOAA
Copy link
Collaborator Author

JianpingHuang-NOAA commented Dec 15, 2022 via email

@bbakernoaa
Copy link
Contributor

bbakernoaa commented Dec 15, 2022 via email

@bbakernoaa
Copy link
Contributor

can we close this now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bbakernoaa @JianpingHuang-NOAA @chan-hoo