Compatibility with NEAD format

[BaptisteVandecrux]

Hi JAWS team,
Just wanted to ask if you had tried jaws on meteoIO output?
Cheers,
Baptiste

[czender]

Hi Baptiste,
We have not yet encountered any out in this format. Which AWS are using it?
Charlie

[BaptisteVandecrux]

Sample file:

[fixed]

[mankoff]

This format is under development by @bavay and others at SLF/WSL (can't find their GitHub handle). It's based on SMET and has been proposed to WMO as a standard.

Citation for WMO proposal:

Iosifescu Enescu, I., Bavay, M., Plattner, G.-K. (2020). Sharing and reusing meteorological data. WMO Data Conference 2020 - Earth System Data Exchange in the 21st Century (Virtual Conference).

I've tried to define the format in EBNF (text and graphics) here: https://github.com/GEUS-PROMICE/NEAD but this is a work-in-progress.

We plan to distribute PROMICE as NetCDF of course, but CSV is still a useful format with lower barrier to use than NetCDF. The advantage of NEAD over simple CSV or NetCDF is that you can embed most (all?) of the metadata supported by NetCDF, but hand-edit it - a requirement for the "raw" AWS data that comes in, although further down the processing chain no manual editing occurs and therefore ASCII formatted files are less useful.

[mankoff]

Note that if you are even considering supporting this file format, the format and required header fields are still in flux/development.

[czender]

@BaptisteVandecrux your sample data file gives a 404 file-not-found error?

[mankoff]

Here's a sample https://github.com/GEUS-PROMICE/NEAD/blob/main/summit.csv

[BaptisteVandecrux]

Dear JAWS team,

I am adapting jaws to read NEAD files, which will be the format GC-Net data will be released in the future (see sample file above). The main issue is that (in a similar way as in the original GC-Net files) it has a variable number of fields and the field description needs to be read from the header.

I have started my own branch to adapt jaws to nead: https://github.com/BaptisteVandecrux/jaws/tree/jaws_bav
I adapted jaws.py, common.py and created nead2nc.py. I am testing it at the moment and would like to have a second pair of eyes on it before I issue a pull request.

It uses @mankoff 's pyNEAD. I don't know how to handle that dependence in the distribution and installation of JAWS.

I don't know where is the most suitable place to have this discussion: here, over a pull request, by email?

Cheers
Baptiste

[czender]

Hello Baptiste,
Thanks for leading this development. @wenshanw please assist Baptiste with adding NEAD.
This would be a great extension to have for JAWS, and keep RIGB highly relevant.

There are some software requirements to be aware of:
JAWS must retain compatibility with conda-forge.
That means all dependencies that JAWS has must also be Conda-forge packages.
In particular, this means creating a conda-forge package for pyNEAD.
Please examine the conda.recipes and .yml files in the JAWS source for illustrative examples.
I have never converted a package to become a conda-forge package myself, so it will be a learning experience for everyone. The main thing to realize is that conda-forge is well documented, and I'm sure you'll find some great HOWTOs on this process.
Charlie

[BaptisteVandecrux]

Hi Charlie & Wenshan,

My version now runs fine with default setting (converting to netcdf, filling attributes, adding sza...) and with RIGB.
I would just like to make sure that I get a reasonable output:

Is it normal that the adjusted values are full of gaps? Is that when the adjusted values are not different from the original ones or when the original ones do not make sense?

I was also wondering:
How is the accuracy of rigb decreasing as we move away from the user-listed clear-sky days?
What was the usual procedure to spot clear-sky days? Just the smooth bell-shape curves for hourly fsds and fsus?
Maybe a small tutorial in the jaws doc would help.

Thanks!

[czender]

Good job getting this far. The plots are illuminating. @wenshanw is best suited to answer your questions.

[wenshanw]

Hi Baptiste,

I will look into why there are gaps in the adjusted fsds and fsus.
It's more likely the adjusted fsds/fsus does not make sense (ie, does not pass the quality control).
Are you ready to make a pull request? or is there a sample dataset?

Without spectral info or reliable cloud observations, the current way to spot clear-sky days is just the smooth bell-shaped curves.

[BaptisteVandecrux]

I have now compared the output of RIGB when processing the NEAD file and when processing the original GC-Net text file at DYE-2 for 2015-2016. The adjusted values match pretty well apart from samples that have different un-corrected fsus and fsds in the NEAD file compared to the GC-Net file (maybe different filtering or interpolating used for the new files).

I think the gaps in the adjusted values (present in both output files) are normal, we just need to know what they exactly mean.

I am only missing the conda-forge compatibility of pyNEAD to make a pull request.

In the meantime, Wenshan you can inspect and test the scripts that are here:
https://github.com/BaptisteVandecrux/jaws/tree/jaws_bav

I've added some sample files here:
https://github.com/BaptisteVandecrux/jaws/tree/jaws_bav/sample_data

[czender]

@BaptisteVandecrux this is a great example of community participation. When our funding for JAWS ended in 2019 we hoped that the community would continue to use, adapt, and improve it. Your work is a major step in that direction. We will do our best to help make pyNEAD a conda-forge package so we can merge your work into JAWS. Is the pyNEAD maintainer willing to joing the effort to make that a conda-forge package?

[mankoff]

On 2021-01-27 at 12:53 -08, Charlie Zender <notifications@github.com> wrote...

Is the pyNEAD maintainer willing to joing the effort to make that a conda-forge package?

I'm willing to help, but I'm hesitant to claim the title of "maintainer". I've put PyNEAD under the GEUS-PROMICE project (not my own GitHub repository), and I'm not using NEAD myself anymore at the moment. But I will help as needed.

[czender]

Thank you. Does pyNEAD have any upstream dependencies that are not already in the Conda-Forge repository? i.e., is pyNEAD a standalone tool that only requires the most common Python libraries?

[mankoff]

Does pyNEAD have any upstream dependencies that are not already in the Conda-Forge repository? i.e., is pyNEAD a standalone tool that only requires the most common Python libraries?

numpy, pandas, xarray

[czender]

Those are all already on conda-forge, which simplifies the problem. Now we just need a pyNEAD package for conda-forge. One option is to merge pyNEAD into JAWS. The other option is to create a conda-forge pyNEAD package. Which do you prefer?

[mankoff]

One option is to merge pyNEAD into JAWS. The other option is to create a conda-forge pyNEAD package. Which do you prefer?

Separate. I think pyNEAD should be accessible without requiring JAWS. But is conda a hard requirement right now? Given that pyNEAD is young and under active development, can we install from GitHub? Some tag (e.g. 'jaws') that we move behind master but better tested than master? Both 'requirements.txt' and conda support installing GitHub commits (tags).

[czender]

Yes, as long as conda-forge supports it, it's fine by me.