Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ninjogeotiff writer adds offset/scale factor when this is not meaningful #2490

Closed
gerritholl opened this issue May 23, 2023 · 1 comment · Fixed by #2491
Closed

ninjogeotiff writer adds offset/scale factor when this is not meaningful #2490

gerritholl opened this issue May 23, 2023 · 1 comment · Fixed by #2491
Assignees
@gerritholl
Labels
bug component:writers

Comments

@gerritholl
Copy link
Collaborator

Describe the bug

The ninjogeotiff writer unconditionally writes an offset and scale factor for mode L images, which is usually fine. It omits them for mode RGB or RGBA images, which is correct. However, it also adds offset and scale factor for mode P images, which is wrong. In some cases, offset and scale factor should be skipped even for mode L or LA images, for example, when writing vis_with_ir_cloud_overlay.

NinJo will interpret those images wrongly if they have an offset and scale factor.

To Reproduce

from satpy import Scene
from satpy.writers import get_enhanced_image
from sattools.io import plotdir
from satpy.utils import debug_on; debug_on()
from glob import glob
import numpy as np
fn = "/media/nas/x21308/scratch/NWCSAF/202302221700/S_NWC_CTTH_MSG4_MSG-N-VISIR_20230222T170000Z.nc"
#fn = "/media/nas/x21308/scratch/NWCSAF/202302010730/S_NWC_CTTH_MSG4_MSG-N-VISIR_20230201T073000Z.nc"
sc = Scene(filenames=[fn], reader=["nwcsaf-geo"])
sc.load(["cloud_top_height"])
#sc.load(["cloudtype"])
ls = sc.resample("nqeuro3km", radius_of_influence=15000)
ls.save_datasets(
    filename=str(plotdir(create=True) / "{platform_name}-{sensor}-{area.area_id}-{name}-{start_time:%Y%m%d%H%M}-{end_time:%H%M}.tif"),
    keep_palette=True,
    writer="ninjogeotiff",
    ChannelID="Cloud type",
    DataType="GPRN",
    PhysicUnit="N/A",
    PhysicValue="Cloud type",
    SatelliteNameID="NOWCASTING NWCSAF Europe-Atlantic",
    fill_value=0)

followed by

gdalinfo Meteosat-11-seviri-nqeuro3km-cloud_top_height-202302221709-1712.tif

Expected behavior

I expect an image that does not have any AxisIntercept (offset) or Gradient (scale factor) defined.

Actual results

[DEBUG: 2023-05-23 11:07:40 : satpy.readers.yaml_reader] Reading ('/data/gholl/checkouts/satpy/satpy/etc/readers/nwcsaf-geo.yaml',)
[DEBUG: 2023-05-23 11:07:40 : satpy.readers.yaml_reader] Assigning to nwcsaf-geo: ['/media/nas/x21308/scratch/NWCSAF/202302221700/S_NWC_CTTH_MSG4_MSG-N-VISIR_20230222T170000Z.nc']
[DEBUG: 2023-05-23 11:07:40 : matplotlib] matplotlib data path: /data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/matplotlib/mpl-data
[DEBUG: 2023-05-23 11:07:40 : matplotlib] CONFIGDIR=/home/gholl/.config/matplotlib
[DEBUG: 2023-05-23 11:07:40 : matplotlib] interactive is False
[DEBUG: 2023-05-23 11:07:40 : matplotlib] platform is linux
[DEBUG: 2023-05-23 11:07:40 : matplotlib] CACHEDIR=/data/gholl/cache/matplotlib
[DEBUG: 2023-05-23 11:07:40 : matplotlib.font_manager] Using fontManager instance from /data/gholl/cache/matplotlib/fontlist-v330.json
/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/metpy/io/nexrad.py:14: DeprecationWarning: 'xdrlib' is deprecated and slated for removal in Python 3.13
  from xdrlib import Unpacker
[DEBUG: 2023-05-23 11:07:41 : rasterio.session] Could not import boto3, continuing with reduced functionality.
[DEBUG: 2023-05-23 11:07:41 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2023-05-23 11:07:41 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2023-05-23 11:07:41 : satpy.readers.nwcsaf_nc] Reading ctth_alti.
[DEBUG: 2023-05-23 11:07:41 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-05-23 11:07:41 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-05-23 11:07:41 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-05-23 11:07:41 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-05-23 11:07:41 : satpy.readers.nwcsaf_nc] Reading ctth_status_flag.
[DEBUG: 2023-05-23 11:07:41 : satpy.readers.nwcsaf_nc] Reading ctth_conditions.
[DEBUG: 2023-05-23 11:07:41 : satpy.readers.nwcsaf_nc] Reading ctth_alti_pal.
[DEBUG: 2023-05-23 11:07:41 : satpy.readers.yaml_reader] No coordinates found for DataID(name='ctth_alti_pal', resolution=3000, modifiers=())
[DEBUG: 2023-05-23 11:07:41 : satpy.readers.nwcsaf_nc] Reading ctth_quality.
[DEBUG: 2023-05-23 11:07:41 : satpy.scene] Unloading dataset: DataID(name='ctth_alti', resolution=3000, modifiers=())
[DEBUG: 2023-05-23 11:07:41 : satpy.scene] Resampling DataID(name='cloud_top_height', resolution=3000)
/data/gholl/checkouts/pyresample/pyresample/spherical.py:300: RuntimeWarning: invalid value encountered in divide
  cart = self.cart / norm
[INFO: 2023-05-23 11:07:45 : satpy.resample] Using default KDTree resampler
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Resampling add-30e1292e75a2d0a82d4fcfa0c4a082ad
[DEBUG: 2023-05-23 11:07:45 : satpy.scene] Resampling DataID(name='ctth_status_flag', resolution=3000, modifiers=())
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Resampling ctth_status_flag
[DEBUG: 2023-05-23 11:07:45 : satpy.scene] Resampling DataID(name='ctth_conditions', resolution=3000, modifiers=())
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Resampling ctth_conditions
[DEBUG: 2023-05-23 11:07:45 : satpy.scene] Resampling DataID(name='ctth_quality', resolution=3000, modifiers=())
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Computing kd-tree parameters
[DEBUG: 2023-05-23 11:07:45 : satpy.resample] Resampling ctth_quality
[DEBUG: 2023-05-23 11:07:45 : satpy.writers] Reading ['/data/gholl/checkouts/satpy/satpy/etc/writers/ninjogeotiff.yaml']
[DEBUG: 2023-05-23 11:07:45 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/generic.yaml
[DEBUG: 2023-05-23 11:07:45 : satpy.writers] Adding enhancement configuration from file: /home/gholl/checkouts/pytroll-dwd-config/enhancements/generic.yaml
[DEBUG: 2023-05-23 11:07:45 : satpy.writers] Adding enhancement configuration from file: /data/gholl/checkouts/satpy/satpy/etc/enhancements/seviri.yaml
[DEBUG: 2023-05-23 11:07:45 : satpy.writers] Data for DataID(name='cloud_top_height', resolution=3000) will be enhanced with options:
        [{'name': 'palettize', 'method': <function palettize at 0x7f86bf5e7b00>, 'kwargs': {'palettes': [{'dataset': 'ctth_alti_pal', 'color_scale': 255}]}}]
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f86bfa278d0>
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Starting outermost env
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] No GDAL environment exists
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f86bf41a860> created
[DEBUG: 2023-05-23 11:07:45 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2023-05-23 11:07:45 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2023-05-23 11:07:45 : rasterio._env] PROJ_DATA found in environment.
[DEBUG: 2023-05-23 11:07:45 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f86bf41a860>.
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f86bfa278d0>
[DEBUG: 2023-05-23 11:07:45 : rasterio._io] Path: _UnparsedPath(path='/media/nas/x21308/plots_and_maps/2023/05/23/Meteosat-11-seviri-nqeuro3km-cloud_top_height-202302221709-1712.tif'), mode: w, driver: GTiff
[DEBUG: 2023-05-23 11:07:45 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f86bfa278d0>
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f86bf41a860> options
[DEBUG: 2023-05-23 11:07:45 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f86bf41a860>.
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Exiting outermost env
[DEBUG: 2023-05-23 11:07:45 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f86bfa278d0>
[INFO: 2023-05-23 11:07:45 : satpy.writers] Computing and writing results...
/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/dask/core.py:119: RuntimeWarning: invalid value encountered in sin
  return func(*(_execute_task(a, cache) for a in args))
/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/dask/core.py:119: RuntimeWarning: invalid value encountered in cos
  return func(*(_execute_task(a, cache) for a in args))
/data/gholl/checkouts/trollimage/trollimage/colormap.py:168: RuntimeWarning: All-NaN slice encountered
  outside_range_bin = max(np.nanmax(arr), values.max()) + 1

Metadata from gdalinfo:

Metadata:
  AREA_OR_POINT=Area
  ninjo_AxisIntercept=-2000.0
  ninjo_CentralMeridian=0.0
  ninjo_ChannelID=Cloud type
  ninjo_ColorDepth=8
  ninjo_CreationDateID=1684832865
  ninjo_DataType=GPRN
  ninjo_DateID=1677085757
  ninjo_EarthRadiusLarge=6378137.0
  ninjo_EarthRadiusSmall=6356752.314245179
  ninjo_FileName=/media/nas/x21308/plots_and_maps/2023/05/23/Meteosat-11-seviri-nqeuro3km-cloud_top_height-202302221709-1712.tif
  ninjo_Gradient=27000.0
  ninjo_HeaderVersion=2
  ninjo_Magic=NINJO
  ninjo_MaxGrayValue=255
  ninjo_MinGrayValue=255
  ninjo_PhysicUnit=N/A
  ninjo_PhysicValue=Cloud type
  ninjo_Projection=PLAT
  ninjo_ReferenceLatitude1=0.0
  ninjo_SatelliteNameID=NOWCASTING NWCSAF Europe-Atlantic
  ninjo_TransparentPixel=0
  ninjo_XMaximum=4000
  ninjo_XMinimum=1
  ninjo_YMaximum=2000
  ninjo_YMinimum=1
  TIFFTAG_DATETIME=2023:02:22 17:09:17

Environment Info:

  • OS: openSUSE Leap 15.3
  • Satpy Version: 0.40.1.dev26+g5f2e3a124
  • PyResample Version: 1.24.0

Additional context

NinJo will display those images wrongly. The writer should have an option whether to include offset and scale factor. Or maybe this could be skipped when units are set to N/A; if we have a scene where we include both 10.8, HRV, and a HRV/10.8-combination, then the former two should have them, but the latter should not.
@gerritholl gerritholl self-assigned this May 23, 2023
@gerritholl
Copy link
Collaborator Author

The NinJo technical documentation suggests that AxisIntercept and Gradient are mandatory tags. If true, they should be set to 0/1 when there is no physical interpretation for the pixels. But so far NinJo has been accepting RGBs without those tags just fine. Checking with the NinJo devs to see what should be done here.

@gerritholl gerritholl mentioned this issue May 23, 2023
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerritholl gerritholl

Labels
bug component:writers
Projects
PCW Spring 2023
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change logic for ninjogeotiff gradient/axisintercept tags gerritholl/satpy
1 participant
@gerritholl