calculate fractions skill score with NEP/NMEP

[JiliDong-NOAA]

Hi,

I am trying to calculate fractions skill score with NEP/NMEP in MET but failed to get any output. What I did was feeding NEP/NMEP to grid_stat with the following config options (take APCP06 as the example):

FCST_VAR1_NAME = APCP_A6_ENS_NEP_ge0.254_NBRHD25
FCST_VAR1_LEVELS = A06
FCST_VAR1_THRESH = ==0.1

OBS_VAR1_NAME = APCP
OBS_VAR1_LEVELS = A06
OBS_VAR1_THRESH = >0.254

GRID_STAT_NEIGHBORHOOD_SHAPE = SQUARE
GRID_STAT_NEIGHBORHOOD_WIDTH = 5

FCST_IS_PROB = True
FCST_PROB_IN_GRIB_PDS = False

In the related met config, I have:

output_flag = {
   fho    = NONE;
   ctc    = NONE;
   cts    = STAT;
   mctc   = NONE;
   mcts   = NONE;
   cnt    = STAT;
   sl1l2  = NONE;
   sal1l2 = NONE;
   vl1l2  = NONE;
   val1l2 = NONE;
   pct    = STAT;
   pstd   = STAT;
   pjc    = STAT;
   prc    = STAT;
   nbrctc = NONE;
   nbrcts = STAT;
   nbrcnt = STAT;
}

But I didn't see any continuous statistics output (e.g. NBRCNT, CNT) in the stat file. I wonder if I missed anything. I should mention I did have the probabilistic stats output in the above case (PCT, PJC etc.)

Thanks,

Jili

[georgemccabe]

Hi @JiliDong-NOAA, I am passing on advice from another METplus team member:

The CNT and NBRCNT output line types are computed for input scalar values (like TMP), not probabilities. Setting FCST_IS_PROB = True is preventing continuous/neighborhood stats from being computed.
Adding "prob_as_scalar = TRUE" to MET's configuration may do the trick.

From the MET User's Guide:

Set “prob_as_scalar = TRUE” to override the processing of probability data. When the “prob” entry is set as a dictionary to define the field of interest, setting “prob_as_scalar = TRUE” indicates that this data should be processed as regular scalars rather than probabilities. For example, this option can be used to compute traditional 2x2 contingency tables and neighborhood verification statistics for probability data. It can also be used to compare two probability fields directly. When this flag is set, probability values are automatically rescaled from the range [0, 100] to [0, 1].

In your METplus configuration file, you can add:

FCST_VAR1_OPTIONS = prob_as_scalar = TRUE

If that does not give you continuous statistic output, we may need the location of your use case configuration file and input data so we can take a closer look.

[georgemccabe]

Hi @JiliDong-NOAA, I am just following up on this issue. Were you able to resolve the problem?

[JiliDong-NOAA]

Hi George,

Sorry for this very late reply. Somehow I totally missed your previous post. Your suggestion works well and I am able to get the FSS using NEP/NMEP. I just would like to confirm the options are correct for precipitation equal or greater than 0.254 mm:

FCST_VAR1_NAME = APCP_A6_ENS_NEP_ge0.254_NBRHD25
FCST_VAR1_LEVELS = A06
FCST_VAR1_OPTIONS = prob_as_scalar = TRUE

OBS_VAR1_NAME = APCP
OBS_VAR1_LEVELS = A06
OBS_VAR1_THRESH = >=0.254

It is a little bit tricky as I have probability as input instead of actual precipitation. The NEP probability was calculated for precipitation ge 0.254 mm (as indicated in the name APCP_A6_ENS_NEP_ge0.254_NBRHD25). Here in the above section I have removed FCST_VAR1_THRESH but keep OBS_VAR1_THRESH = >=0.254. Could you let me know if these options look reasonable?

Thanks again for your help!

Jili

[JohnHalleyGotway]

Hi @JiliDong-NOAA, @georgemccabe asked me to take a look at this Discussion.

Seems like there's a lot of details going on here. And while I'm glad to hear that Grid-Stat ran and successfully computed an FSS output value, I do think it warrants some more clarification.

As I understand it, you are:

1. Running Ensemble-Stat (or perhaps Gen-Ens-Prod) to derive NEP and NMEP output fields.
2. Reading the APCP_A6_ENS_NEP_ge0.254_NBRHD25 variable from the NetCDF output from Ensemble-Stat into Grid-Stat and comparing against some gridded precip analysis.
3. You've set prob_as_scalar = TRUE; in the Grid-Stat config file so that the NEP data is processed as scalars instead of probabilities.
4. And you got FSS in the output.

My concern is what's happening between steps 3 and 4. Since the NEP data already contains values between 0 and 1, I suspect that you want Grid-Stat to process those as if they were the fcst fractional coverage values rather than deriving them itself, as it typically does. Please look closely at the documentation for the nbrhd dictionary. The default setting, "field = BOTH;" applies the fractional coverage derivation logic to both the fcst and obs fields. I'm guessing you only want to to the derivation to the observation data and process the input forecast probability values as is. If that is the case, please try rerunning with "field = OBS;" in the "nbrhd" dictionary and rerun.

This cannot be configured directly via METplus right now. But you can use config overrides to do it:

   GRID_STAT_MET_CONFIG_OVERRIDES = nbrhd = { field = OBS; }

Please give that a shot and let us know if/how that changes the result. Also, it'd probably be wise to run Grid-Stat at verbosity level 4 or so and then carefully inspect the log messages. Just want to confirm that it's configured to implement the logic you expect.

[JiliDong-NOAA]

Hi John,

What you described is exactly I intend to do. Following your suggestions, I got a higher FSS which should be the correct one. Thanks for the explanation and the clarification!

I would like to mark both George and your reply as answers to close the discussion but not sure if I am allowed to do that.

Jili

[georgemccabe]

Hi Jili,

It looks like we can only mark 1 response as the "answer." John's answer is found in the reply chain of my original response (which was also written with John's support!) so it will be easy to find upon review. I will go ahead an mark the answer now and lock this discussion. Thanks!