How to build a backend for MTG L0?

[headtr1ck]

Hey there.

I want to build a xarray backend to read MTG L0 files (e.g. Sentinel 4, https://user.eumetsat.int/s3/eup-strapi-media/MTG_Generic_Format_Specification_GFS_59f7e6d891.pdf)

I think reading a single binary file should be easy to implement but I am struggling in how to combine multiple files.
The issue is that this spec can cut images at arbitrary rows, so an image might be split between two files. I guess a single file would then have some NaNs in the first and last image.

A native open_mfdataset approach would therefore duplicate the images at the file boundary that needs to be patched afterwards with some merge somehow.

But the binary format is built in a way that you could interpret it as one continuous binary stream, is there a way to let open_mfdataset know about that?

[keewis]

If all you need to do is trim off a certain amount (where fixed or to be determined from the metadata) you can use preprocess to do this. Combining an index with an appropriate merge strategy (e.g. join="left") could also work.

Though if this is a continuous binary stream that is chopped into files, then it might be easier to have the backend interpret the sequence of files as that binary stream and do everything in the backend.