pcp_combine issued with GEFS forecasts

[lesnow93]

Hi MET Help,

I'm having a problem with pcp_combine in using the "-sum" option (or any options for that matter with the GEFS member I'm using). I'm running an embarrassingly old version of MET right now (v6), as I'm trying to do some additional verification for a paper using the same version I ran for generating the original verification stats using WRF model output. So hopefully my question transcends through versions of MET/METPlus.

FYI, pcp_combine works fine on hourly WRF output, but is having trouble with the 6-hourly GEFS output.

A sample command is as follows: (I also simply withholding the -field entries, but same problem)
pcp_combine -sum 20180301_120000 6 20180302_000000 12 ./forecast_APCP_12_modelGrid -pcpdir ./temp -field 'name="APCP"; level="A6";' -name APCP_12 -v 2

The error I receive is:
DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times = 20180301_120000/060000/20180302_000000/120000
DEBUG 2: Searching for 2 files with accumulation times of 060000 to sum to a total accumulation time of 120000.
ERROR :
ERROR : sum_data_files() -> Cannot find a file with a valid time of 20180302_000000 and accumulation time of 060000 matching the regular expression ".*"
ERROR :

And the contents of my ./temp folder for summing is simply 2 different GEFS model members with only APCP 6-h accumulation fields in them:
(base):=> ll ./temp
total 112
-rw-r--r-- 1 casejl ghcc 57052 Nov 12 21:24 GEFS4_1803011200_wrfout_arw_d02.grb2f060000
-rw-r--r-- 1 casejl ghcc 54994 Nov 12 21:24 GEFS4_1803011200_wrfout_arw_d02.grb2f120000

Wgrib2 on second file that is valid at 20180302_000000:
(base):=> wgrib2 GEFS4_1803011200_wrfout_arw_d02.grb2f120000
1:0:d=2018030112:APCP:surface:6-12 hour acc fcst:ENS=+4 (base:=> wgrib2 -vt GEFS4_1803011200_wrfout_arw_d02.grb2f120000
1:0:vt=2018030200

Your help is much appreciated!!

Most Sincerely,
Jonathan

---

Mr. Jonathan Case
ENSCO, Inc.
NASA Short-term Prediction Research and Transition (SPoRT) Center
320 Sparkman Dr., Room 3000B (currently teleworking)
Office: (256) 961-7504
Jonathan.Case-1@nasa.govmailto:Jonathan.Case-1@nasa.gov and case.jonathan@ensco.commailto:case.jonathan@ensco.com
https://weather.msfc.nasa.gov/sport/

---

[BinbinZhou-NOAA]

I got similar issue when tried to generate 12hr or 24hr APCP from GEFS's 6hr APCP field. I guess the reason is the product ID (pdt) definition for 6hr APCP field in GEFS members may not be well defined while PCPcombine tool is just working fine only for well defined pdt fields.
My method is, first create an empty directory, then copy required 2 (12hr APCP) or 4 (24hr APCP) previous
forecast time files into this directory. For example, for 12hr APCP at 48fhr, previous forecsat file at 42fhr and the file at 48fhr are copied first. Then using PCPcombine -sum to add the 6 hr ACP from these 2 files to obtain 12hr APCP. After the job is done, remove these 2 files for next run.
I know this method seems ugly, but working well.

Binbin Zhou
NCEP/EMC

[lesnow93]

Hello Binbin. I'm pretty sure that is how I'm currently trying to run pcp_combine, but let me attempt it manually again.
-Jonathan

[JohnHalleyGotway]

Thanks very much @BinbinZhou-NOAA for jumping in on this discussion!

I wanted to try to replicate this issue and ran the following commands to retrieve two GEFS input files from a recent run:

wget https://nomads.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.20211115/12/atmos/pgrb2bp5/gep04.t12z.pgrb2b.0p50.f006
wget https://nomads.ncep.noaa.gov/pub/data/nccf/com/gens/prod/gefs.20211115/12/atmos/pgrb2bp5/gep04.t12z.pgrb2b.0p50.f012

However, when I ran wgrib2 to inventory them, I only found ACPCP and NCPCP for accumulated convective (ACPCP) and non-convective (NCPCP) precip. However, there was no total accumulated precipitation (APCP):

wgrib2 gep04.t12z.pgrb2b.0p50.f006  | egrep -i pcp
293:59556070:d=2021111512:ACPCP:surface:0-6 hour acc fcst:ENS=+4
294:59685755:d=2021111512:NCPCP:surface:0-6 hour acc fcst:ENS=+4

@lesnow93 can you please confirm that your input files do indeed contain records of APCP? It certainly is possible I'm looking at the wrong input files!

If they do contain APCP, please point me to some publicly available data I can use to replicate this behavior.
Or you could upload data to our anonymous ftp site following these instructions.

[lesnow93]

Hi John:
I posted the GEFS4 files I'm using at https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/GEFS/.
When I first acquired the GEFS members, I combined the a and b files by concatenating them into a single grib2 file. Then I ran a couple wgrib2 commands to extract only the APCP fields prior to running them through MET/pcp_combine. The reason they have the naming convention is so it can be processed through my scripts in the same way as the WRF output files that I was verifying.

Hopefully, this gives you call the info you need to test.

Most Sincerely,
Jonathan Case
(aka @lesnow93)

[JohnHalleyGotway]

Jon,

Thanks for providing the input data. I downloaded the 2 files, stashed them in a directory named "temp", and reran the command you sent using pcp_combine from MET version 10.0.0:

MET-main_v10.0/met/bin/pcp_combine -sum 20180301_120000 6 20180302_000000 12 ./forecast_APCP_12_modelGrid -pcpdir ./temp -field 'name="APCP"; level="A6";' -name APCP_12 -v 2

It runs fine with no error:

DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times = 20180301_120000/060000/20180302_000000/120000
DEBUG 2: Searching for 2 files with accumulation times of 060000 to sum to a total accumulation time of 120000.
DEBUG 1: [1] File ./temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000 matches valid time of 20180302_000000
DEBUG 1: [2] File ./temp/GEFS4_1803011200_wrfout_arw_d02.grb2f060000 matches valid time of 20180301_180000
DEBUG 1: [1] Reading input file: ./temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000
DEBUG 1: Reading data (name="APCP"; level="A6";) from input file: ./temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000
DEBUG 1: [2] Reading input file: ./temp/GEFS4_1803011200_wrfout_arw_d02.grb2f060000
DEBUG 1: Reading data (name="APCP"; level="A6";) from input file: ./temp/GEFS4_1803011200_wrfout_arw_d02.grb2f060000
DEBUG 1: Creating output file: ./forecast_APCP_12_modelGrid
DEBUG 2: Writing output variable "APCP_12" for the "sum" of "APCP/A6".
DEBUG 1: Closing output file: ./forecast_APCP_12_modelGrid

And running plot_data_plane, produces this result:

MET-main_v10.0/met/bin/plot_data_plane forecast_APCP_12_modelGrid plot.ps 'name="APCP_12"; level="(*,*)";'

So upgrading to a newer version will produce the desired result. However, I was surprised to a find MET version 6.1 compiled on one of our project machines and was actually able to replicate this error with it:

met-6.1/bin/pcp_combine -sum 20180301_120000 6 20180302_000000 12 ./forecast_APCP_12_modelGrid -pcpdir ./temp -field 'name="APCP"; level="A6";' -name APCP_12 -v 10
...
ERROR  : sum_data_files() -> Cannot find a file with a valid time of 20180302_000000 and accumulation time of 060000 matching the regular expression ".*"

The issue seems to be correctly recognizing that these records contain an accumulation interval. I tried running plot_data_plane to retrieve/plot the APCP data:

met-6.1/bin/plot_data_plane temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000 plot.ps 'name="APCP"; level="A6";'
ERROR  : plot_data_plane -> trouble getting field "name="APCP"; level="A6";" from file "temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000"

But switching from "A6" to "L0" does work:

met-6.1/bin/plot_data_plane temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000 plot.ps 'name="APCP"; level="L0";'

I tried several things to get the MET version 6.1 pcp_combine -sum command to work but wasn't able to.
But the following -add command (using level = "L0") does work:

met-6.1/bin/pcp_combine -add \
temp/GEFS4_1803011200_wrfout_arw_d02.grb2f060000 'name="APCP"; level="L0";' \
temp/GEFS4_1803011200_wrfout_arw_d02.grb2f120000 'name="APCP"; level="L0";' \
out.nc

As long as your input GRIB2 files only contain a single accumulation interval (6-hours), then this solution is OK. But if it contained both A6 and A12, then it'd match both of those records.

I'd recommend switching to a more recent version of MET, when possible.

Thanks,
John

[lesnow93]

Working on installing met-10.0.0 now. Our system compiler is a bit older, but hopefully all will work well, per the install instructions at https://dtcenter.org/community-code/model-evaluation-tools-met/download.

[lesnow93]

Well, unsuccessful with installing met-10.0.0. The complaint is that the C++ compiler cannot create execs:
checking whether the C++ compiler works... no
configure: error: in /raid1/sport/people/casejl/MET_v10.0.0/met-10.0.0': configure: error: C++ compiler cannot create executables See config.log' for more details

[JohnHalleyGotway]

@jprestop can you please take a look at the error message that @lesnow93 is reporting while attempting to compile MET version 10.0.0 and advise?

[jprestop]

Hi @lesnow93. Could you please send us your config.log file for us to review?

[lesnow93]

Log files at: https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/LOGS/

[jprestop]

Thanks @lesnow93. In the config.log file, I see:

ld: cannot find -lpython3.6m

I see a conflict in the values for MET_PYTHON_CC and MET_PYTHON_LD. MET_PYTHON_CC refers to python3.7.m, but MET_PYTHON_LD refers to python3.6m.

MET_PYTHON_CC=-I/raid2/sport/people/casejl/python/anaconda3/include/python3.7m
MET_PYTHON_LD_value='-L/raid2/sport/people/casejl/python/anaconda3/lib -lpython3.6m -lpthread -ld
l -lutil -lrt -lm -Xlinker -export-dynamic'

I'm wondering if MET_PYTHON_LD should instead refer to python3.7m? If so, please modify that value, try rerunning, and let us know how it goes. Thanks!

[lesnow93]

Thanks for finding my careless mistake! When I copied a sample environment file, I left the library reference to python3.6m, when I have python3.7 installed. Look like MET compiled just fine now.

[lesnow93]

@JohnHalleyGotway : After successfully installing MET-v10.0.0, I was able to sum two sample 6 hour GEFS and replicate your .ps graphic procedure above, so it looks like all is well now. I will update my scripts to use MET v10 then. Hopefully, my custom .poly files from v6 will translate into v10. If I encounter any issues transitioning to MET v10 that I can't figure out, I'll create a new post on github. Many thanks to all!
-Jon Case

[JohnHalleyGotway]

Glad to hear it! I'll select our original back/forth as the "answer" for this discussion and then lock it. That's our way of encouraging users to create new discussions for new questions. We've found that it's difficult to keep track of new posts to previously answered questions.
Thanks.