TCRMW with netcdf input

[mpm-meto]

Colleagues and I have been looking to get some of the TC tools working with netcdf and IBTraCS.
Whilst we have made some progress with this we have a few queries.

One aspect we are seeing is around support for netcdf variable names: e.g. x_wind and y_wind were not recognised as UGRD and VGRD (like is the case for other MET tools), and these have to be specified, either in the MET config or in the METplus wrapper with the OVERRIDES option. We thought both options work, but there is a caveat...

If both x_wind and x_wind_0 are present in the same file they probably can't be handled. It says it can't find x_wind_0 for example, and we wonder whether this is because there is the ability (with the overrides) to map only one wind variable (x_wind) to UGRD. The same for VGRD.

It also seems to have some issues (warning) of not being able to match lead times, which is odd.
We also don't think it's diagnosing the P level data with the netcdf level specification of "(@500,,)", for example. Whilst it doesn't complain it doesn't produce any multi-pressure level output either.

The final issue is that whilst TCRMW runs through, the output for VR and VT is incomplete even though the log file suggests that it is computing all the grid points of the grid that is specified. I've attached two figures of what the output looks like. It happily processes x_wind and y_wind 2-D output, but the VR/VT is a truncated row.

I have uploaded some sample netcdf forecast data for you to hopefully be able to replicate the behaviour we have.

[j-opatz]

Hi Marion,

I wanted to let you know that I'm not seeing the sample data that you said was uploaded; is that upload still in progress via Discussions, or was that completed on your end when you submitted this Discussion (on the 17th)? Or perhaps it was in an email that I missed?

Once I have the data, I'll be sure to take a look and confirm the behavior you're seeing. Some of these issues could be settings related, but I want to make sure we capture any potential bugs in the TC-RMW tool.

[mpm-meto]

@j-opatz I uploaded it using the google form with the discussion number. My google id is my married name. It said it uploaded the data. I did forget to upload the track, I realised. That's here. Please let me know whether you find my file or whether I need to upload again...
WP222024_GLX_2024102312.txt

[j-opatz]

Hello Marion,

I wanted to make sure I chimed in before this issue became too stale! I am looking into this still, but environment issues with my new system are causing back and forth with IT. When it's straightened out, I should be able to discuss more confidently.

All of the files I'm using were the ones you submitted to my email; I'm still not certain where the ones you provided via Google went...

I did check on a few of the issues you encountered. I can confirm the behavior you're seeing with the x_wind/x_wind_0 & y_wind/y_wind_0 variables; when I try to process x_wind as a field variable, x_wind must be the name used for the u_wind_field_name setting. If you try to mix and match, for example, x_wind as a field variable and x_wind_0 as the u_wind_field_name, MET prints

WARNING: x_wind_0 has 0 inputs
WARNING: y_wind has 1 inputs 

Is there a situation where you would desire to have one variable as the u_wind_field_name or v_wind_field_name and another variable as the field variable to be processed?

I also investigated the lead time issue. I see that the forecast file has the CF-compliant forecast_reference_time variable with a value of 480,444 and from my math, it looks like that would be 20241022_120000 (with a %Y%m%d_%H%M%S format). That seems to be proven with the file name (Fcst_20241022_1200_024_globalxvec_cutout). And from the time variable value (480,468), it seems like the lead should be calculated as 24 hours, with a valid time of 20241023_120000.

Running through the TC-RMW tool shows several WARNING messages:

DEBUG 3: Processing series entry 1: x_wind_0(@500-@750,*,*), ValidTime = 20241023_120000
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "x_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)
WARNING: 
DEBUG 3: Processing series entry 1: y_wind_0(@500-@750,*,*), ValidTime = 20241023_120000
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "y_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)
WARNING: 
...
DEBUG 3: [29 of 29] Processing track point valid at 20241030_120000 with center (lat, lon) = (17.2, 117.8).
DEBUG 3: Processing series entry 29: x_wind_0(@500-@750,*,*), ValidTime = 20241030_120000
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "x_wind_0" variable, the valid time does not match the requested valid time: (20241023_120000 != 20241030_120000)
WARNING: 
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "x_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)
WARNING: 

To me, it makes sense that any deckfile time not at 20241023_120000 (which is only the first entry in your .dat file) should raise a warning message that the forecast's lead time does not align with the valid time being processed. Do you think there should be a different logic occurring?

There are two things of interest I found. The first is that WARNING message about matching requested lead time: It seems like MET may be evaluating an empty lead time as "NA", causing a discrepancy.

The second is that I was seeing doubles: with the x_wind_0 and y_wind_0 as the field variables to be evaluated, I was seeing a "double" print of the y_wind_0:

DEBUG 3: [29 of 29] Processing track point valid at 20241030_120000 with center (lat, lon) = (17.2, 117.8).
DEBUG 3: Processing series entry 29: x_wind_0(@850,*,*), ValidTime = 20241030_120000
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "x_wind_0" variable, the valid time does not match the requested valid time: (20241023_120000 != 20241030_120000)
WARNING: 
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "x_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)
WARNING: 
DEBUG 3: Processing series entry 29: y_wind_0(@850,*,*), ValidTime = 20241030_120000
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "y_wind_0" variable, the valid time does not match the requested valid time: (20241023_120000 != 20241030_120000)
WARNING: 
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "y_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)
WARNING: 
DEBUG 3: Processing series entry 29: y_wind_0(@850,*,*), ValidTime = 20241030_120000
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "y_wind_0" variable, the valid time does not match the requested valid time: (20241023_120000 != 20241030_120000)
WARNING: 
WARNING: 
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "y_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)
WARNING: 

What very odd behavior! Did you see anything similar in your log output?

All of the things I've listed above I will discuss with the engineers to see what the root of the problem is.

I still need to look into the @ level behavior, as well as the VR/VT issue, which I'll complete as soon as my visual capability is fixed!

[JohnHalleyGotway]

@mpm-meto, thanks for sending along some sample data. I pulled the Fcst_20241022_1200_024_globalxvec_cutout.nc file and did some additional testing using the current state of the main_v12.2 branch in MET.

I have a several things to note, but let me start with the punchline. Please try rerunning using the set_attr_level config option, similar to this. That should produce better results:

data = {
   field = [
      { name = "x_wind_0"; level = "(@500,*,*)"; set_attr_level = "P500"; },
      { name = "y_wind_0"; level = "(@500,*,*)"; set_attr_level = "P500"; },
      { name = "x_wind_0"; level = "(@850,*,*)"; set_attr_level = "P850"; },
      { name = "y_wind_0"; level = "(@850,*,*)"; set_attr_level = "P850"; }
   ];
}
...
u_wind_field_name = "x_wind_0";
v_wind_field_name = "y_wind_0";

And here's the gory details:

• I was able to replicate the behavior you describe - seemingly truncated output for radial and tangential winds in the NetCDF output from TC-RMW. It matches the screenshot in your original post.

• Running at debug level 4 (e.g. -v 4), I do see log messages confirming that VR and VT are actually being computed... and being computed properly. See these types of log messages:

DEBUG 4: wind_ne_to_rt() -> center lat/lon (16.8, -122.9), range (km): 990, azimuth (deg): 358, point lat/lon (16.9012, -132.193), uv (9.75, -0.625), radial wind: 9.76587, tangential wind: -0.284349

Running through a debugger and inspecting the expected/actual dimensions of the output file, I conclude that problem lies in how the data is written, not in how it is computed.

• Specifically, VR and VT are written as 4-dimensional "pressure level data", like this:

   double VR(track_point, pressure, range, azimuth) ;
   double VT(track_point, pressure, range, azimuth) ;

But in the bad output files, that pressure dimension is missing:

   double VR(track_point, range, azimuth) ;
   double VT(track_point, range, azimuth) ;

So that explains the bad write... TC-RMW attempts to write to a 4-d variable when only 3 dimensions are defined. I'm pretty surprised this doesn't produce a runtime error!

• I investigated why the pressure level data isn't being handled properly, and the answer is in on line 163 of vx_tc_nc_util.cc:

        if (levels[j].substr(0, 1) == "P") {

Only level strings that begin with P, like P500 are counted as containing pressure level data. That's why using set_attr_level = "P500"; produces the desired result.

Here's what VR and VT look like with this sample data:

Some conclusions:

1. To work well with NetCDF inputs, we need additional unit tests to demonstrate it.
2. TC-RMW clearly needs more error checking to prevent attempting to write to 4D data to a 3D variable.
3. We should consider if/how writing VR and VT for a single level should be supported.
4. There's some "false positive" warning messages currently being written, like this:

WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &, LongArray &) -> for "x_wind_0" variable, the lead time does not match the requested lead time: (240000 != NA)

We should refine the logic to prevent those from being written. When running with forecast model data and forecast tracks, yes, the lead time information should match. But when running with analysis data and BEST tracks, no, the lead times should not, and will not match.