Is it possible to open a GRIB2 message from a bytes instance?

[milancurcic]

Hi, I have a GRIB2 message as a byte array in memory (bytes type). Is it possible to open it with cfgrib without writing it to disk first?

I tried cfgrib.open_dataset and cfgrib.open_file. I also tried passing io.BytesIO instance of my byte array with no success.

This is cfgrib-0.9.10.3 with eccodes-2.16.0 on Ubuntu 20.04 LTS.

Thank you!

[juntyr]

I'd also be very interested in this - I have a custom class implementing io.IOBase but from which I cannot load a grib file at the moment since cfgrib seems to only take file system paths.

[iainrussell]

Hello,

I agree this would be a useful feature. It's not there yet, but perhaps in the meantime you could be interested in trying our new package 'earthkit-data', which does enable this. In fact, it uses cfgrib in the background, so the xarray dataset will be the same.

Warning: earthkit-data is currently beta, and not yet for operational use, but it is available on PyPi and conda-forge. It has the same dependency on ecCodes that cfgrib has.

https://github.com/ecmwf/earthkit-data
https://earthkit-data.readthedocs.io/en/latest/index.html

Here's an example of how to create a bytes object (which you already have) and create an xarray from it. I should mention that this method does not create any temporary file (that would be cheating!).

import io
import earthkit.data

f = open('tests/data/t_time_series.grib', 'rb')
buffer = f.read()
f.close()

stream=io.BytesIO(buffer)
data = earthkit.data.from_source("stream", stream, batch_size=0)
x = data.to_xarray()

If you need to pass any kwargs to cfgrib, this shows how:
https://earthkit-data.readthedocs.io/en/latest/_api/data/readers/grib/index/index.html#data.readers.grib.index.FieldList.to_xarray

Best regards,
Iain

[rabernat]

Kerchunk does this: https://github.com/fsspec/kerchunk/blob/main/kerchunk/grib2.py

import eccodes
import cfgrib

data  # type: bytes
mid = eccodes.codes_new_from_message(data)
m = cfgrib.cfmessage.CfMessage(mid)