G2G AGRI geotiff images show artifacts

[kathys]

Working with the November 4 G2G tarball, I am creating all default AGRI products from an FY4A dataset. The resulting infrared images show some artifacts including a non-white backing in Channel 7, and cutoff and smearing of data along the right side of the image in the infrared bands 8-14.

data: bumi/data/test_data/agri/2022_01_20
logfile: bumi:/data/users/kathys/geo/agri4a

See the montage below:

[djhoese]

I think this was just fixed by @simonrp84 in pytroll/satpy#2276

[djhoese]

Could also be this one: pytroll/satpy#2165

[simonrp84]

I have not seen the smearing in any of my processing through satpy, but the non white background should be fixed in my recent pr yes.

[djhoese]

I wonder if the smearing is an issue in the image viewer. I've seen that with some when dealing with tiled geotiffs.

[kathys]

I can confirm that you are right about the viewer. I tried displaying the image using two other viewers, and neither of them showed the smearing artifacts in the IR bands. The problem viewer is ImageMagick.

[kathys]

G2G V1.1 11/25 tarball still creates AGRI-FY4A Channel 7 non-transparent background. I do not consider this a show-stopper for the release of the tarball.

[djhoese]

@kathys if you have time, could you confirm transparent backgrounds on all ABI channel geotiffs?

[simonrp84]

Any chance you can send one of the problematic files over, or at least give the filename? Agri files are sometimes inconsistent so maybe you've got some with a different fill value or something.

[kathys]

@djhoese No non-transparent geotiffs from any of the abi instruments/bands that I have seen.

[kathys]

@simonrp84 I put the data that I am using here:

https:

[simonrp84]

@kathys Thanks, but it looks like the /fy4a/ directory is missing.

[kathys]

Sorry @simonrp84 it was not supposed to get scrubbed. I placed it here: https://bin.ssec.wisc.edu/pub/CSPP/data/

[djhoese]

FYI I am able to reproduce this when generating C07 and C08 with Geo2Grid and with Satpy. Oddly though I had to use imagemagick (a modern version) otherwise my system's default image viewer showed a black background. Very odd.

[djhoese]

Here's C07 plotted with matplotlib straight out of Satpy:

In [1]: from satpy import Scene; from glob import glob
s
In [2]: scn = Scene(reader="agri_fy4a_l1", filenames=glob("/data/satellite/agri_l1/20220120/*.HDF"))

In [3]: scn.load(["C07", "C08"])

In [4]: scn["C07"].plot.imshow().figure.show()

So the space pixels have a value of 200.

[djhoese]

One thing that's odd is why it seems to be choosing the C08 at 4km resolution even though I think it is available from the 2km file. C07 (the bad background) is from the 2km file.

[djhoese]

@simonrp84 So a couple things:

1. I think I found a bug in https://github.com/pytroll/satpy/blob/ba2f8e6ff3a96d734016d1c0fac0358baf4e9e9e/satpy/readers/fy4_base.py#L100. I think this should be >=, right?
2. If I print out the information for C08 (the good channel) then I see space pixels in the file set to 65535, the valid range set to 0-4095, and the size of the LUT is 4096 (before NaN is appended). This all makes sense; all those 65535s will be converted to NaN as they are outside the LUT. For C07, space pixels are still 65535, the valid range is 0 to 65534, but the LUT has 65536 elements. So for some reason the LUT has an entry for this value even though it should be invalid based on the data's valid_range. The data's range is never enforced because the reader assumes the LUT checks will handle that.

Summary for @kathys: Files aren't exactly wrong, but they aren't great. Some files (other files) might have this issue.

[djhoese]

Coming back to this more than a year later, it looks like the issue I identified in my previous comment was never fixed. @simonrp84 I know this is old, but does that seem like a bug to you?

For now, @kathys is going to get a new version of our Geo2Grid bundle and try running these cases again and see what we get with modern versions of the software. We may not test with the newest files for AGRI, but we can probably fix that bug in Satpy...if it is a bug.

[kathys]

Working on new version of G2G v1.2 (geo2grid-swbundle-20240110-204310). Some of the artifacts are no longer visible. But the C7 image still shows the space pixels as being something other than fill values, and the enhancement for C7 does not appear to be as good as shown in the images created using G2G versions 1.1. Here is the image from the same dataset created using G2G v1.2. BTW, a recent dataset shows the same behavior.

[kathys]

Enhancement for C7 is different in G2G version 1.2 than previous version for FY4B AGRI as well. Everything else looks fine, and the image creation time was about a minute faster.

FY4B AGRI G2G v1.1

FY4B AGRI G2G v1.2

[simonrp84]

OK, I think the problem with C07 is that the LUT for this channel contains 65536 values, and the fill value is 65535. For other channels the fill value is the same...but the LUT only contains 4096 values, and hence the fill value never matches a LUT value.

Will see if I can think of a way around that but it might take a while (lots of other tasks) so your thoughts on how to overcome it would be welcome!

[djhoese]

I'm going to try to come up with a good solution for this today. @kathys for the change in C07 enhancement in G2G 1.2, this has to do with Jim's request in #278 to be more sensitive to fires in the "C07" band of the main supported geostationary imagers.

[kathys]

Okay @djhoese. That is fine.