Enhance ASCII2NC to support IABP/IPAB Arctic and Antarctic drifting buoy observations

[JohnHalleyGotway]

Describe the Enhancement

This enhancement request arose via GitHub discussion dtcenter/METplus#2312. Please see this comment from that discussion for more information about the IABP/IPAB Arctic and Antarctic drifting buoy observation dataset.

Some details to note:

• Dataset description: https://iabp.apl.uw.edu/data.html
• Sample data: https://iabp.apl.uw.edu/WebData/
• The data is stored in one ASCII file per buoy. So each file many contain months or years of data for each buoy.
• Enhance ASCII2NC to read this data source.
• In particular, extract the meteorological observations of temperature and pressure (along with any others).

Also enhance ASCII2NC to filter data by time. Currently, ASCII2NC can compute time summaries through the config file but cannot filter the input data by time. Recommend adding valid_beg and valid_end config options to support by time. pb2nc already supports this using the obs_window config option or -valid_beg and -valid_end command line options.

This will provide flexibility in how ASCII2NC is run to process the data. Users many want the input/output files the same with each file containing all obs for a single buoy. Or they may want to extract obs from all the buoys for some time window (e.g. daily or monthly perhaps). Filtering by time provides that option.

Could also consider filtering in space by supporting the mask config option, but that's not explicitly needed for this data. This feature request moved to #2754.

Time Estimate

3 days?

Sub-Issues

Consider breaking the enhancement down into sub-issues.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2784543 NOAA R2O METp SW2

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
• Need an issue for METplus to update the ascii2nc wrapper accordingly: Enhance the ASCII2NC wrapper to support setting the -valid_beg and -valid_end command line options METplus#2547

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[davidalbo]

@JohnHalleyGotway I'm trying to add @rgrumbine to this issue, but can't see it working. Any advice/suggestions/wisdom? Maybe it 'just' works, even though does not auto complete? @rgrumbine are you out there?

[rgrumbine]

Out last week, but I'm here now. The @ mentions did prompt an email to me.

[davidalbo]

great @rgrumbine. I'm actively working on modifying ascii2nc to read this data type. The goal would be to do things that make you best able to do your research. So far I've done a few things:

• wrote a python script to pull down all the files from this site: https://iabp.apl.uw.edu/WebData/. I did this for my own testing. Could provide this.
• wrote another python script to figure out which of these files are within a configurable time interval. Did this for my own testing. Could also provide this.
• began modifying ascii2nc to read files in this format
• about to add in command line options for time range as @JohnHalleyGotway suggested: -valid_beg and -valid_end

The ascii2nc design allows you to put any number of input files on the command line. If it's a lot of files, this becomes user unfriendly. Therefore if you did want to read in a large number of files to produce one output netCDF file, that would be hard (for you).

Please let me know if this is moving towards something useful for you and give me some ideas on what else to do.

[rgrumbine]

Thanks David

I'm in process of changing my sea ice drift verification to a more 'retail' approach than 'wholesale'. Drift is the accumulated distance and direction of motion (of these buoys) over N days. The operational model predicts this directly. 'Retail' meaning that I'm going to focus on scoring one buoy at a time vs. the model, as opposed to my first many years of doing all buoys with data in a given several months to several years at one big go.

n.b.: The buoy files also may contain observations of atmospheric and oceanic, and even ice, parameters. Where they exist, those are valuable and should be retained. Just that my personal first use is the drift, for which I only need the location observations and times.

My workflow for 'retail' verification will be:

• Get buoy location information for a time span
• Process it in to drift distance and direction
• Compare the model's prediction vs. these observations

Note that the current GFS-based drift model goes to day 16. GFSv17 will be to 35-45 days. Not sure this affects you, but it's a heads up for design purposes.

Since it's necessary to do the drift computation and evaluation outside of Met+ at this point, your tools to acquire and translate ascii to nc are more useful to me than embedded in Met+ -- your test codes will be helpful to me, please do send/share.

In the future some time, my codes for the drift computation and evaluation maybe? could be brought in to Met+. Mine are in C++ and maybe some Fortran.

What is 'a lot' of files? IABP and IPAB typically only have a few dozen out at a time.

[davidalbo]

What is 'a lot' of files? IABP and IPAB typically only have a few dozen out at a time.

At the website with all buoys I see 3720 buoys. Didn't know how many of these you might want to process at once (based on how big a time interval you might want to look at in part) so, blindly, thought it could possibly be a large set of files.

Do you have any examples of files that have these extra observation parameters? The documentation I found indicated there could be any subset of: Barometric Pressure, Surface Temp, and Atmospheric Temp, including none of them, in which case it's only location and time.

Can you explain the diff between day of year and position day of year? Both of these are provided.

[davidalbo]

Need to fill in the following to create individual observations in the output netCDF file. (all are required, but can in some cases be set to missing). Along with these are my guesses as to what the value should be
std::string _headerType; kind of arbitrary
std::string _stationId; the actual BuoyID
time_t _validTime; as computed from the inputs year, hour, minute, DOY (day of year)
double _latitude; from input line in file
double _longitude; from input line in file
double _elevation; set to missing
std::string _qualityFlag; set to NA
std::string _varName; 'Surface Temp' or 'Atmospheric Temp' or similar strings
int varCode; this is a grib number, set to ???
double _pressureLevel; Barametric pressure on input line. Are the units important? Are they correct in the files?
double _height; set to missing
double _value; the actual values found on the input line

Complications are that individual files can have pressure, surface temp, atmospheric temp, or any subset of these including none. As far as I know every file has BuoyID, Year, Hour, Min, DOY, POS_DOY, Lat, Lon. Example header lines:

File with all inputs:
300234011819920.dat:BuoyID Year Hour Min DOY POS_DOY Lat Lon BP Ts Ta

File with Barometric pressure and surface temp but no atmospheric temp:
300234011882260.dat:BuoyID Year Hour Min DOY POS_DOY Lat Lon BP Ts

File with surface temp but no barometric pressure or atmospheric temp:
300234011889340.dat:BuoyID Year Hour Min DOY POS_DOY Lat Lon Ts

File with no barometric pressure, no surface temp, and no atmospheric temp:
109312.dat:BuoyID Year Hour Min DOY POS_DOY Lat Lon

Many designs could be implemented for this. Putting down some thoughts/ideas:

One way to go: No matter what, every output has pressure, surface temp and atmospheric temp at all valid times, with values set to missing if not present in the file.

If a file has Ts but no Ta, only the surface temp is put to the output file.
If BP is not in a file, the pressureLevel is set to missing in the output file.

Because every observation must have a varName and value, something must go to output, even if no candidate variable/value exists, as in file 109312.dat above. What should it be?

[davidalbo]

Notes to self:
Put the time range filter into the base class.
Test what DOY returns by converting to month,day,hour,min,sec based on year, see if makes sense.
Query as to meaning of POS_DOY.

[davidalbo]

Notes on computing hour/minute using DOY input and comparing that to the individual hour/minute inputs.

The fractional part of DOY (hms) should be hours within the day as a real number, so expect
hour = hms*24
minute = fractional part of hour*60

file = 101631.dat,

Example1, off by a lot, first line:
101631 2012 12 25 254.5130 254.5130 83.55100 67.40700 1006.90 -40.55
hms = .5130
hour = .5130*24 = 12.24. (hour = 12, agrees with input hour)
minute = 0.2*60 = 14.4 so should be 14 (or 15), instead input is 25. Off by 10-11 minutes

An example that works if you truncate the minute: same file, line 10:
101631 2012 14 19 254.5970 254.5970 83.56400 67.37100 1006.30 -40.40
hms = .5970
hour = .5970*24 = 14.328. (Hour = 14, agrees with input hour).
minute = .328*60 = 19.68, which truncates to 19 which is a match to the input minute

An example that works if you round the minute: line 56, same file:
101631 2012 05 23 255.2240 255.2240 83.63200 67.67000 1008.90 -40.70
hms = .2240
hour = .2240*24 = 5.376. (Hour=5 which is a match.)
minute = .376*60 = 22.56 which rounds up to 23 to give a match to the input minute.

Note that both of the last two minute examples have a floating minute remainder > 0.5, so it does not work to truncate if <0.5 and round if > 0.5.

[davidalbo]

@JohnHalleyGotway questions/concerns before preparing a pull request:
This data has these 3 fields (any subset can be missing from a file)
BP = Barometric Pressure, if available
Ts = Surface Temperature if available
Ta = Air Temperature if available

Question1: Could I put BP into the observation as the pressure_level_hpa slot for the Observation object, or should it be a variable/value? This (optional) data is the only pressure data in these files.

Question2: Would it be better to always put Ts/Ta into the output netCDF as variable/values, with the value set to missing when not in the file, or would it be better to not put these into output netCDF unless found in the input files? But.. see concern2 below.

Input data processing concerns:

1. Multiple input files can be converted with one ascii2nc run, with various combinations of Ts/Ta/BP.
2. If none of Ts/Ta/BP are in a file, we'd have an observation with no variable, which is not allowed, so we'd have to do something or other.

I have 2 python scripts it'd be nice to put into the repo, any suggestions on where to put them?:
get_iabp_from_web.py pulls down all the IABP station data files from the web.
get_iabp_from_web.py lists the files that contain a particular time range, once they have been pulled down.

[rgrumbine]

Remember that for one of my major uses, the location of the buoy is the variable of interest. The IABP/IPAB buoys are sitting on ice floes, so this tells us two things: where it is (has drifted to) and that the ice floe hasn't melted away.

Other variables, like Ta, Ts, ... are desirable, but even with none of them, the location is a valuable observation.

Since the IABP/IPAB use flag values when an observation type is not available, I'll suggest continuing the practice here. So, have all possible observation types listed, and then not-available when they're not.

[davidalbo]

@rgrumbine That makes total sense, measuring drift via location. I agree with the choice of putting in all the obs types even when not in a particular file. It seems easiest engineering wise.

About the only thing remaining that you can help me with is the difference between DOY and POS_DOY, I need to choose one to figure out month and day. From the website: https://iabp.apl.uw.edu/IABP_Table.html

DOY = the day of the year to the decimal minute (1.0 to 365.999) of the reported data
POS_DOY = the day of the year to the decimal minute of the reported position

My guess is DOY is the time when a measurement like Ts/Ta happened. POS_DOY is the time when the lat/lon was measured. Often they are identical but not always. I'm therefore guessing you'd prefer POS_DOY. Your thoughts?

[rgrumbine]

I will ask Ignatius Rigor, lead for the IABP/IPAB, what the distinction is, exactly, between DOY and POS_DOY. iirc, the time difference between the two is ?always small (for my purposes to date). But it's best to have the definitive answer.

[rgrumbine]

Already have the authoritative answer:

"You are correct about what DOY and POS_DOY mean [I quoted your above text]. DOY refers to variables other than position, and POS_DOY is date for the position data. Some buoys only update position every 3 hours to save batteries, and sometimes a buoy may not be able to communicate with a satellite for one reason or another for a time. When this happens, the buoy is still able to take local measurements and store the information but is not able to establish a position or transmit data until a satellite link is established."

[davidalbo]

Based on your feedback @rgrumbine the code will use POS_DOY to determine month and day. Thanks for looking into it.