Support GRIB2 DRS 42

[rschmunk]

This seems to be a problem with 5.4-MAINT up to the current (7?) development branch.

A user sent me a GRIB2 file that netCDF-Java will open, but when one then tries to extract a variable, an "Unsupported DRS type = 42" exception is thrown by the Grib2DataReader.factory method. (But note that the same exception could be thrown by Grib2DataReader.getData.)

Comparison of the switch statements in these methods to ECMWF's data representation template list, several such templates are not handled. But for the moment, 42 is the one of concern.

A stack trace is appended (using 5.4-MAINT), but note that the line numbers in Grib2Drs and Grib2DataReader are slightly off as I inserted some logging.

A sample file that demonstrates the problem is here on Dropbox.

The user also comments that: "As far as I have understood, this file has been generated by using ECMWF's ECCODES library and it originates from MetCoOp (251) center."

--

java.lang.RuntimeException: Unsupported DRS type = 42
	at ucar.nc2.grib.grib2.Grib2Drs.factory(Grib2Drs.java:39)
	at ucar.nc2.grib.grib2.Grib2SectionDataRepresentation.getDrs(Grib2SectionDataRepresentation.java:83)
	at ucar.nc2.grib.grib2.Grib2Record.readData(Grib2Record.java:319)
	at ucar.nc2.grib.collection.GribDataReader$Grib2DataReader.readData(GribDataReader.java:474)
	at ucar.nc2.grib.collection.GribDataReader.read(GribDataReader.java:273)
	at ucar.nc2.grib.collection.GribDataReader.readDataFromCollection(GribDataReader.java:116)
	at ucar.nc2.grib.collection.GribDataReader.readData(GribDataReader.java:81)
	at ucar.nc2.grib.collection.GribIosp.readData(GribIosp.java:1085)
	at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122)
	at ucar.nc2.Variable.reallyRead(Variable.java:797)
	at ucar.nc2.Variable._read(Variable.java:736)
	at ucar.nc2.Variable.read(Variable.java:614)
	at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:459)
	at ucar.nc2.dataset.VariableDS._read(VariableDS.java:432)
	at ucar.nc2.dataset.VariableDS._read(VariableDS.java:442)
	at ucar.nc2.Variable.read(Variable.java:600)
	at ucar.nc2.Variable.read(Variable.java:546)
	at gov.nasa.giss.data.nc.array.NcArray2D.doSlice(NcArray2D.java:363)
...

[rschmunk]

Also, it seems that this issue involves use of a Lambert Conformal Conic projected grid.

Note that I posted an issue two years ago involving use of a Lambert Azimuthal Equal Area projected grid, although that manifested as an inability to even open the dataset. See #22.

[lesserwhirls]

Comparison of the switch statements in these methods to ECMWF's data representation template list, several such templates are not handled. But for the moment, 42 is the one of concern.

Usually a DRS isn't implemented due to lack of a sample file. That said, DRS 42 looks to be pretty complicated:

https://library.wmo.int/doc_num.php?explnum_id=10722 (pdf page 132)

Let me see if there is a java port of libaec out there that we could use. Otherwise, we'd either have to write our own implementation or start wrapping more C code, neither of which sound like great options.

[rschmunk]

Oh, ugh. Looking at that PDF page. 😱

[rschmunk]

Ping! I just had another request for DRS 42 from a developer at ECMWF.

[shahramn]

Thanks for chasing this up. From cycle 48R1 ECMWF data by default will use CCSDS packing ( for gridded fields )

[rschmunk]

Ping! I just had yet another query about DRS 42 from someone trying to work with ECMWF data.

[haileyajohnson]

@rschmunk it would be great to have this, but unfortunately our dev teams hands are really full right now, so unless we get some outside help on this issue, it's not going to make it to the top of the TODO list for a while.

[rschmunk]

Unfortunately, as @lesserwhirls pointed out, this one looks like a hairball.

Maybe some ECMWF person will get attached to the issue and write some code?

[Yaqiang]

GRIB2 DRS 42 support request + 1.

[NZTSL]

Another GRIB2 DRS 42 support request here, thanks.

[rschmunk]

@haileyajohnson, I hate to say it but I just heard from another user whose ability to view ECMWF data has been crippled because ECMWF is now shipping the product using data representation 42.

Edit: This looks like the same or related product as what @shahramn mentioned on Feb. 20.

[haileyajohnson]

So I 100% hear that this is becoming a major issue. The problems we previously listed though still remain. I think this is going to involve us doing some major work on compression in general in netcdf-java, which wouldn't be the worst thing. I don't know how long it would take to plan/design/develop though, so I guess my question is what is the impact to everyone +1ing this issue if it takes 3/6/9 months before we're able to ship this as a feature?

[cvasquez-github]

GRIB2 DRS 42 support request + 1

[shahramn]

You could always convert the CCSDS GRIB ( gridded ) fields back to Simple Packing. E.g.,
via the ecCodes grib_set command-line tool

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple ccsds.grib grid_simple.grib

Of course it would be fantastic if Panoply could decode such GRIBs but this is a temporary solution

[JohnLCaron]

Ive been using https://openjdk.org/jeps/434 "JEP 434: Foreign Function & Memory API" in Java 20, and it works well and seems to solve most of the problems with jni/jna. One could use it to wrap libaec.

…

On Mon, Jul 31, 2023 at 6:58 AM shahramn ***@***.***> wrote: You could always convert the CCSDS GRIB ( gridded ) fields back to Simple Packing. E.g., via the ecCodes grib_set command-line tool grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple ccsds.grib grid_simple.grib — Reply to this email directly, view it on GitHub <#753 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEVZQESMJ247YIDHRZO24TXS6TWNANCNFSM475YCZOQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[shahramn]

Dear John,
That would be fantastic! I'll buy you a beer if you do it ;)

[haileyajohnson]

That's kinda the direction we're leaning, but currently netcdf-java is committed to supporting jdk-8. What we're considering is putting new features that require higher java versions in a different subproject, and encouraging people to use a later java version to get those features.

[rongmafight]

GRIB2 DRS 42 support request + 1

[JohnLCaron]

Seems like for some, compiling libaec and running Java 20 might be worth it for getting access to DRS 42 from netcdf-java. What do you think, @DennisHeimbigner ? Im not seeing any java ports available.

[DennisHeimbigner]

According to this page: https://gitlab.dkrz.de/k202009/libaec,
it is a an implementation of the Golomb-Rice algorithm.
So I went looking for Golomb-Rice in Java an found this site:
https://github.com/tomgibara/coding/blob/master/src/main/java/com/tomgibara/coding/GolombCoding.java

So, it is barely possible that this could replace libaec.