Confirm Units of Knots for NIDS Base Velocity Data

[julienchastang]

According to a recent e-support (ticketid 27672) there is ambiguity on whether the NIDS data, base velocity field is in knots or meters per second. @yuanho Can we have a confirmation of the units of knots for the NIDS data base velocity field in 65eda31? (We have the usual problem of knots versus meters per second being only a factor of 2 off from each other.) According to @mike-dixon, the data are probably in meters per second. I have yet to find any NEXRAD documentation that details anything about units pertaining to NIDS data.

[dopplershift]

I can confirm (from what I've done in MetPy) that products 93, 99, 154, 182 have velocity in meters per second. From the NEXRAD product interface control document:

For products 93, 99, 154, and 155 data level codes 0 and 1 correspond to "Below Threshold"
and "Range Folded", respectively. For products 93, 99, and 154 data levels 2 through 255
denote data values starting from the minimum data value in even data increments. For product
155, data levels 129 through 149 denote data values starting from the minimum data value in
even data increments. The threshold level fields are used to describe (up to) 256 levels as follows:

halfword 31 contains the minimum data value in m/s*10
halfword 32 contains the increment in m/s*10
halfword 33 contains the number of levels (0 - 255)

I have a similar document for 182 (specific to TDWR).

Unless some conversion is taking place somewhere in the code (no idea), meters per second is correct.

[julienchastang]

Thanks, Ryan, for the rapid response. For posterity's sake the document you refer to is the Common Operations and Development Environment (CODE) for the WSR-88D Open RPG (pdf) -- there's a mouthful.

[dopplershift]

Actually, the documents I'm referring to are:

• Interface Control Document for the RPG to Class 1 User (2620001W)
• Interface Control Document for the SPG to Class 1 User

These are the canonical documents describing every byte in every product generated for NEXRAD (or TDWR for NWS).

[julienchastang]

Alrighty, thanks. Out of curiosity, how come these documents are not on the web anywhere except committed to the thredds repo?

[dopplershift]

You just have to have experience knowing where to look, such as writing code to decode NEXRAD stuff in 3 languages. 😁 What you want to google for is "ROC ICD" (Radar Operations Center Interface Control Document) which returns for the first hit: http://www.roc.noaa.gov/wsr88d/BuildInfo/Files.aspx

The TDWR is trickier, but googling "ROC SPG" (SPG = supplemental product generator, used for TDWR radars) gives you this first hit with some relevant documents:
http://www.roc.noaa.gov/spg/

None of this is in the thredds repo as far as I can tell, which is a good thing IMO--the ICDs are updated with almost every major release of the ROC's radar software.

[julienchastang]

There is something amiss here that may not be related to units. Here is the metpy rendering of these data after they have been converted to knots (e.g., data = (data / 10) * 1.94384):

See here for the code to generate this image.

and here is the same plot with the IDV with the same color table:

Here are the IDV bundles to generate this images

The images and data seem to be in agreement in the middle of the data range (~40 knots), but out of whack on the low end (e.g., < 20 knots). Maybe @dopplershift and @yuanho can sort this out after Thanksgiving break.

[mike-dixon]

Hi Juilen

The metpy plot is odd in that the velocities are all positive, whereas
for radar one expects the data to go from -nyquist to +nyquist.

Mike

On 11/23/16 14:03, Julien Chastang wrote:

There is something amiss here that may not be related to units. Here
is the metpy rendering of these data after they have been converted to
knots (e.g., |data = (data / 10) * 1.94384|):

metpy
https://cloud.githubusercontent.com/assets/229955/20577779/194b580a-b181-11e6-983f-e451284c93ba.png

See here for the code to generate this image
https://gist.github.com/julienchastang/04f70c48090e48d5026664db3a8f3bfa.

and here is the same plot with the IDV with the same color table:

idv
https://cloud.githubusercontent.com/assets/229955/20577794/31a7b31c-b181-11e6-91f7-b8421098d77e.png

Here are the IDV bundles to generate this images
https://motherlode.ucar.edu/repository/entry/show?entryid=77a53daf-e626-4291-a278-312aa27c2e4b

The images and data seem to be in agreement in the middle of the data
range (~40 knots), but out of whack on the low end (e.g., < 20 knots).
Maybe @dopplershift https://github.com/dopplershift and @yuanho
https://github.com/yuanho can sort this out after Thanksgiving break.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#674 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbh_0EUSkGqfdpcJtwft6_5B7OetSsOks5rBKo8gaJpZM4Kr3Ve.

[lesserwhirls]

For fun, here is what PyArt does.

Sorry the colorbar is different (and jet to boot...). This was my first time using PyArt, but super easy to generate. Note that the pyart.io.read_nexrad_level3 code is independent of netCDF-Java and MetPy.

edit: here is the matching figure using pyart

[mike-dixon]

I think all of the data in the SW quadrant - which is probably negative - is being plotted as 0 m/s. Try using a color scale that goes from -50 to +50. Mike

…

On 11/23/16 15:51, Sean Arms wrote: For fun, here is what PyArt does (). pyart <https://cloud.githubusercontent.com/assets/67096/20581114/f8d3d008-b193-11e6-8e27-9cd580e87069.png> Sorry the colorbar is different (and |jet| to boot...). This was my first time using PyArt, but super easy to generate. Note that the |pyart.io.read_nexrad_level3| code is independent of netCDF-Java and MetPy. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#674 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGbh_62yeXyXLNLtLnZKOKPQX-znku1bks5rBMOGgaJpZM4Kr3Ve>.

[lesserwhirls]

Here is pyart with +/- 50. @mike-dixon 👍

[lesserwhirls]

New IDV screenshot, +/- 50

[lesserwhirls]

File (had to add .txt to upload to github)

KLIX_SDUS54_N0ULIX_201208301134.txt

[dopplershift]

@julienchastang While data = (data / 10) * 1.94384 may be correct in the spirit of what's in the NIDS ICD, I don't think that's handling signed-ness properly in Python. What you want to do is use the map_data attribute on Level3File which is the right way to convert the data to float (which should handled signed-ness, non-data code points, etc.):

# Pull the data out of the file object
datadict = f.sym_block[0][0]

# Turn into a floating point masked array using map_data
data = ma.array(f.map_data(datadict['data']))

# Convert from m/s to knots
data *= 1.94384

@mike-dixon As far as +/- nyquist is concerned, that's not quite correct here because we're dealing with a NEXRAD velocity product here, not raw data, so automated velocity dealiasing has been applied.

[dopplershift]

@lesserwhirls Note, zipping works too :)

[mike-dixon]

Sean - that looks correct - or at least plausible - Mike

…

On 11/23/16 16:15, Sean Arms wrote: New IDV screenshot, +/- 50 py_art_full <https://cloud.githubusercontent.com/assets/67096/20581717/08c3f32c-b198-11e6-8541-22789042520a.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#674 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGbh_5WZMH7TSngu9Uf9KwEo7RUv7jNHks5rBMkcgaJpZM4Kr3Ve>.

[dopplershift]

So if I correct the scaling as above and change to the 'RdYlGn_r' colormap to match pyart, here's what I get when I plot data in m/s, from -50 to 50:

And here's one in knots in the IDV colormap with the range from 0 to 80:

IMO, the original IDV plot is incorrect.

(See Unidata/MetPy#49 for an existing note that we need to improve MetPy's NEXRAD API to make it easy to get this decoding right.)

[mike-dixon]

Ryan - if in IDV you use a color scale that goes from -50 to +50, that plot will probably also be OK - Mike

…

On 11/23/16 17:05, Ryan May wrote: So if I correct the scaling as above and change to the |'RdYlGn_r'| colormap to match pyart, here's what I get when I plot data in |m/s|, from -50 to 50: metpy-nids-ms <https://cloud.githubusercontent.com/assets/221526/20582455/84434a88-b19e-11e6-9e2a-170dc4d49b5d.png> And here's one in knots in the IDV colormap with the range from 0 to 80: metpy-nids-idv <https://cloud.githubusercontent.com/assets/221526/20582511/fd020504-b19e-11e6-982b-54f902fb5182.png> IMO, the original IDV plot is incorrect. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#674 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGbh_zWk1MgAhPBT2Rn9QeTwnpOk0lqmks5rBNTSgaJpZM4Kr3Ve>.

[julienchastang]

This patch:

 cdm/src/main/java/ucar/nc2/iosp/nids/Nidsheader.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cdm/src/main/java/ucar/nc2/iosp/nids/Nidsheader.java b/cdm/src/main/java/ucar/nc2/iosp/nids/Nidsheader.java
index 6205681..23267ab 100644
--- a/cdm/src/main/java/ucar/nc2/iosp/nids/Nidsheader.java
+++ b/cdm/src/main/java/ucar/nc2/iosp/nids/Nidsheader.java
@@ -2519,7 +2519,7 @@ class Nidsheader{
             ctilt = pname_lookup(99, pLevel);
 
             ctitle = "HighResolution: Base Velocity";
-            cunit = "KT";
+            cunit = "m/s";
             cname = "BaseVelocityDV";
             summary = ctilt + " is a radial image of base velocity field and its range 124 nm";

yields (after you convert from m/s to knots in the IDV)

which is the same as Ryan's latest figure. Here it is again w/ state boundaries:

BTW, although the IDV user does not mention it, this is Hurricane Isaac. Can these data be additionally corroborated against what we know about the storm track on Aug 30 at 1134Z (time gleaned from the file name, I assume in UTC)? 75 mph seems high to me at that point and time in the storm.

Happy Thanksgiving everyone!

[dopplershift]

I think those values are fine given that:

1. The radar measures instantaneous winds, not sustained over a period of several seconds--so not the same as in the hurricane intensity report
2. At that distance from the radar, the beam is several hundred meters (to ~1km) above the ground, where winds will be higher than surface

[julienchastang]

To further corroborate those points, examine the following figure where I incorporated the Hurricane Isaac storm track, radar range rings, and distance tools. (The isaac.xidv bundle is also on that same ramadda folder above.) The track is colored by maximum wind intensity (though again the track does not show the units, but probably knots).

That purple area of maximum wind intensity is at 150KM from the radar site. A bit of trig (taking into the 0.5 deg angle and curvature of earth) will show this is ~3KM (~700mb) in the atmosphere. That area is also at ~175KM from the storm center on the "dirty side" of the hurricane. Therefore winds of 60 knots seem very plausible, and indeed likely.

All in all, this has been a very interesting case study. I'll be sending the PR shortly.

[dopplershift]

This should be fixed by #687 and #689.