incorrect data offset in HSD files

[jferencik]

Describe the bug
The satpy's HSD reader is reading data incorrectly because it fails to calculate the
byte offset where data is located correctly. This is because one element (header length) from the ERROR_INFO_BLOCK or Block 10 has a incorrect type (u2, 2 bytes ).

https://github.com/pytroll/satpy/blob/master/satpy/readers/ahi_hsd.py#L211

As per HSD ver. 1.4

https://www.data.jma.go.jp/mscweb/en/himawari89/space_segment/hsd_sample/HS_D_users_guide_en_v13.pdf

it is a 4 byte word (Page 15)

The reader works but the data is shifted 1 pixel in x(horizontal ) dimension because the data offset is 2 bytes smaller so the first pixel in the image is read from the last spare 2 bytes.

To Reproduce

given the same image in HSD and HRIT from JMA (easier with IR >7 bands because they they feature 2 km resolution ) the difference between the brightness temperature shows the spatial offset.

# Your code here

Expected behavior
Calibrated data for IR images from HSD and HRIT images should be almost identical

Actual results
Text output of actual results or error messages including full tracebacks if applicable.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Info:

• OS: [e.g. OSX, Windows, Linux]
• SatPy Version: [e.g. 0.13.0]
• PyResample Version:
• Readers and writers dependencies (when relevant): [run from satpy.config import check_satpy; check_satpy()]

Additional context
Add any other context about the problem here.

[djhoese]

You mentioned HSD 1.4, but the document you linked to says 1.3. This also says 2017. Do all current available HSD files have this 4 byte element? If not, is there a way to tell which version of the HSD format we are reading?

[jferencik]

so to clarify

HSD comes with v1.2 and 1.3...so no 1.4. My mistake
v1.3 is the current one
I checked and v1.2 and it is 4 bytes as well.

[simonrp84]

To my knowledge there is only one specification of the HSD format, the v1.1, 1.2, 1.3 refers to the document version - there have been some documentation updates but none to the file structure itself.

I will check this bug against our own HSD reader - but I think @jferencik is right, there's a one pixel offset.

[simonrp84]

Yep, can confirm that the SatPy Himawari data is shifted by one pixel relative to our own HSD reader. This is true both for current data and for historical data (i.e: Data across all document versions).
I will see if I can fix this.

[sfinkens]

@jferencik Good catch! Thanks for reporting! @simonrp84 Thanks for double-checking!

[jferencik]

@simonrp84,
I want to ask you if , by any chance you know who produces the Himawari HRIT files (is it JMA or Eumetcast???? ). I am trying to create an exact replica of HRIT files from HSD, and while I accomplished this successfully for IR channels I am having issues replicating the resampling/downsampling of VIS and NIR channels as HSD VIS/NIR channels have a higher spatial resolution than HRIT VIS/NIR channels.

[mraspaud]

I think it's JMA

[jferencik]

@mraspaud , thanks. I was thinking the same but I just wanted to be sure.
I was expecting JMA will use NearestNeighbour when downsampling high res VIS/NIR counts to 2km HRIT counts but I am unable to reproduce the process. I guess not many people want to convert a HSD to HRIT using there own code.
I have also noticed JMA software does use a more sophisticated way (correction header) when reprojecting from GEOS to LATLON but I am not sure if satpy uses this header as well. Does anybody know about this?

[simonrp84]

@jferencik Unfortunately I've never used the HRIT data from Himawari, so am unsure how it's processed.

[jferencik]

@mraspaud
do you think the info located here

https://navigator.eumetsat.int/product/EO:EUM:DAT:HIMAWARI:H8-L1C/print

is accurate:)))???

according to this link the HRIT files are produced by EUMETSAT not JMA

I simply can not replicate the conversion of HSD to HRIT for VIS channels. Everything works for IR channels though, where there is no downsampling involved.
I am going to download sample HSD and HRIT images data from JMA's website and check them to see whether me or Eumetcast is wrong:)))

[mraspaud]

I'm not sure, the best way is to ask the EUM Helpdesk for a definitive answer