Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

avhrr_l1b_gaclac fails to read most files from NOAA CLASS #2494

Closed
thomasdouwes opened this issue May 24, 2023 · 13 comments · Fixed by pytroll/pygac#123 or #2501
Closed

avhrr_l1b_gaclac fails to read most files from NOAA CLASS #2494

thomasdouwes opened this issue May 24, 2023 · 13 comments · Fixed by pytroll/pygac#123 or #2501

Comments

@thomasdouwes
Copy link

Describe the bug
A clear and concise description of what the bug is.

When using the avhrr_l1b_gaclac reader to load NOAA POES GAC files from NOAA CLASS, most file will fail to load when Scene.load is called.

To Reproduce

Fail:
Try to load NSS.GHRR.NK.D23133.S0714.E0907.B3003031.WI

success:
Try to load NSS.GHRR.NK.D23133.S0902.E1057.B3003132.WI

# Your code here
from satpy import Scene
from satpy.utils import debug_on

debug_on()

reader_kwargs = {'tle_dir': '.', 'tle_name': 'TLE.txt'}

filename="/path/to/file"

scn = Scene(filenames=[filename],  reader='avhrr_l1b_gaclac', reader_kwargs=reader_kwargs)

scn.load(["1"])

Expected behavior
A clear and concise description of what you expected to happen.

AVHRR channel one is loaded into the scene

Actual results
Text output of actual results or error messages including full tracebacks if applicable.

[DEBUG: 2023-05-24 22:29:43 : satpy.readers.yaml_reader] Reading ('/home/thomas/.local/lib/python3.11/site-packages/satpy/etc/readers/avhrr_l1b_gaclac.yaml',)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap.responses')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2349: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(parent)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap.handlers')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap.tests')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('zope')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
[DEBUG: 2023-05-24 22:29:43 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-05-24 22:29:43 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-05-24 22:29:43 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-05-24 22:29:43 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-05-24 22:29:43 : satpy.readers.yaml_reader] Assigning to avhrr_l1b_gaclac: ['/misc/noaa/class2/ftp.avl.class.noaa.gov/8308276947/001/NSS.GHRR.NK.D23133.S0714.E0907.B3003031.WI']
[DEBUG: 2023-05-24 22:29:43 : satpy.composites.config_loader] Looking for composites config file avhrr-3.yaml
[DEBUG: 2023-05-24 22:29:43 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2023-05-24 22:29:43 : satpy.composites.config_loader] Looking for composites config file avhrr-1.yaml
[DEBUG: 2023-05-24 22:29:43 : satpy.composites.config_loader] No composite config found called avhrr-1.yaml
[DEBUG: 2023-05-24 22:29:43 : satpy.composites.config_loader] Looking for composites config file avhrr-2.yaml
[DEBUG: 2023-05-24 22:29:43 : satpy.composites.config_loader] No composite config found called avhrr-2.yaml
[INFO: 2023-05-24 22:29:43 : pygac.klm_reader] Reading NSS.GHRR.NK.D23133.S0714.E0907.B3003031.WI
[DEBUG: 2023-05-24 22:29:43 : pygac.reader] validate header
[DEBUG: 2023-05-24 22:29:43 : pygac.klm_reader] validate header
[DEBUG: 2023-05-24 22:29:43 : pygac.gac_reader] validate header
[WARNING: 2023-05-24 22:29:43 : pygac.reader] Expected 13630 scan lines, but found 16659!
/home/thomas/.local/lib/python3.11/site-packages/pygac/reader.py:276: RuntimeWarning: Unexpected number of scanlines!
  warnings.warn("Unexpected number of scanlines!",
[DEBUG: 2023-05-24 22:29:43 : pygac.reader] Removed 1929 scanline(s) with corrupt scanline numbers
[ERROR: 2023-05-24 22:29:43 : pygac.reader] Timestamp mismatch. Cannot perform correction.
[DEBUG: 2023-05-24 22:29:43 : pygac.calibration] Read PyGAC internal calibration coefficients.
[INFO: 2023-05-24 22:29:43 : pygac.calibration] Identified calibration coefficients version "PATMOS-x, v2017r1, with provisional coefficients for MetOp-C".
/home/thomas/.local/lib/python3.11/site-packages/pygac/calibration.py:198: RuntimeWarning: Using CoeffStatus.PROVISIONAL calibration coefficients
  warnings.warn(warning, RuntimeWarning)
[WARNING: 2023-05-24 22:29:43 : pygac.calibration] Using CoeffStatus.PROVISIONAL calibration coefficients
/home/thomas/.local/lib/python3.11/site-packages/geotiepoints/geointerpolator.py:82: RuntimeWarning: invalid value encountered in arcsin
  lats = np.sign(z__) * (90 - np.rad2deg(np.arcsin(np.sqrt(x__ ** 2 + y__ ** 2) / radius)))
/home/thomas/.local/lib/python3.11/site-packages/geotiepoints/geointerpolator.py:89: RuntimeWarning: invalid value encountered in arccos
  lat_z_only = 90 - np.rad2deg(np.arccos(normalized_z))
[INFO: 2023-05-24 22:29:46 : pygac.reader] TLE filename = ./TLE.txt
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
[ERROR: 2023-05-24 22:29:46 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='longitude', resolution=1050, modifiers=())': Could not convert tuple of form (dims, data[, attrs, encoding]): ('y', array(['0716-05-19T16:39:19.820', '0716-05-19T07:34:17.820',
       '0716-05-19T07:39:54.820', ..., '0716-05-19T07:35:48.320',
       '0716-05-19T07:39:31.320', '0716-05-19T07:36:01.320'],
      dtype='datetime64[ms]')) to Variable.
Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 132, in as_variable
    obj = Variable(*obj)
          ^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 366, in __init__
    self._data = as_compatible_data(data, fastpath=fastpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 296, in as_compatible_data
    data = _possibly_convert_objects(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 225, in _possibly_convert_objects
    as_series = _as_nanosecond_precision(as_series)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 203, in _as_nanosecond_precision
    return duck_array_ops.astype(data, nanosecond_precision_dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/duck_array_ops.py", line 188, in astype
    return data.astype(dtype, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/generic.py", line 6324, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 451, in astype
    return self.apply(
           ^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 511, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 242, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 184, in astype_array
    values = values.astype(dtype, copy=copy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py", line 674, in astype
    res_values = astype_overflowsafe(self._ndarray, dtype, copy=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/tslibs/np_datetime.pyx", line 306, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 400, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 387, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 218, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 716-05-19 16:39:19

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 823, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 723, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 699, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py", line 189, in get_dataset
    res['acq_time'] = ('y', times)
    ~~~^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/dataarray.py", line 826, in __setitem__
    self.coords[key] = value
    ~~~~~~~~~~~^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/coordinates.py", line 41, in __setitem__
    self.update({key: value})
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/coordinates.py", line 169, in update
    coords, indexes = merge_coords(
                      ^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/merge.py", line 553, in merge_coords
    collected = collect_variables_and_indexes(aligned, indexes=indexes)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/merge.py", line 354, in collect_variables_and_indexes
    variable = as_variable(variable, name=name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 135, in as_variable
    raise error.__class__(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Could not convert tuple of form (dims, data[, attrs, encoding]): ('y', array(['0716-05-19T16:39:19.820', '0716-05-19T07:34:17.820',
       '0716-05-19T07:39:54.820', ..., '0716-05-19T07:35:48.320',
       '0716-05-19T07:39:31.320', '0716-05-19T07:36:01.320'],
      dtype='datetime64[ms]')) to Variable.
[INFO: 2023-05-24 22:29:46 : pygac.reader] TLE filename = ./TLE.txt
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
[ERROR: 2023-05-24 22:29:46 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='latitude', resolution=1050, modifiers=())': Could not convert tuple of form (dims, data[, attrs, encoding]): ('y', array(['0716-05-19T16:39:19.820', '0716-05-19T07:34:17.820',
       '0716-05-19T07:39:54.820', ..., '0716-05-19T07:35:48.320',
       '0716-05-19T07:39:31.320', '0716-05-19T07:36:01.320'],
      dtype='datetime64[ms]')) to Variable.
Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 132, in as_variable
    obj = Variable(*obj)
          ^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 366, in __init__
    self._data = as_compatible_data(data, fastpath=fastpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 296, in as_compatible_data
    data = _possibly_convert_objects(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 225, in _possibly_convert_objects
    as_series = _as_nanosecond_precision(as_series)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 203, in _as_nanosecond_precision
    return duck_array_ops.astype(data, nanosecond_precision_dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/duck_array_ops.py", line 188, in astype
    return data.astype(dtype, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/generic.py", line 6324, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 451, in astype
    return self.apply(
           ^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 511, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 242, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 184, in astype_array
    values = values.astype(dtype, copy=copy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py", line 674, in astype
    res_values = astype_overflowsafe(self._ndarray, dtype, copy=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/tslibs/np_datetime.pyx", line 306, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 400, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 387, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 218, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 716-05-19 16:39:19

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 823, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 723, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 699, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py", line 189, in get_dataset
    res['acq_time'] = ('y', times)
    ~~~^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/dataarray.py", line 826, in __setitem__
    self.coords[key] = value
    ~~~~~~~~~~~^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/coordinates.py", line 41, in __setitem__
    self.update({key: value})
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/coordinates.py", line 169, in update
    coords, indexes = merge_coords(
                      ^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/merge.py", line 553, in merge_coords
    collected = collect_variables_and_indexes(aligned, indexes=indexes)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/merge.py", line 354, in collect_variables_and_indexes
    variable = as_variable(variable, name=name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 135, in as_variable
    raise error.__class__(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Could not convert tuple of form (dims, data[, attrs, encoding]): ('y', array(['0716-05-19T16:39:19.820', '0716-05-19T07:34:17.820',
       '0716-05-19T07:39:54.820', ..., '0716-05-19T07:35:48.320',
       '0716-05-19T07:39:31.320', '0716-05-19T07:36:01.320'],
      dtype='datetime64[ms]')) to Variable.
/home/thomas/.local/lib/python3.11/site-packages/pygac/calibration.py:337: DeprecationWarning: Using the 'corr' argument is depricated in favor of making the units of the function result clear. Please make any unit conversion outside this function.
  warnings.warn(
/home/thomas/.local/lib/python3.11/site-packages/pygac/calibration.py:544: RuntimeWarning: invalid value encountered in log
  tsE = c2*nu_c / np.log(1.0 + nBB_num / Ne)
[INFO: 2023-05-24 22:29:49 : pygac.reader] TLE filename = ./TLE.txt
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
[ERROR: 2023-05-24 22:29:49 : satpy.readers.yaml_reader] Could not load dataset 'DataID(name='1', wavelength=WavelengthRange(min=0.58, central=0.63, max=0.68, unit='µm'), resolution=1050, calibration=<1>, modifiers=())': Could not convert tuple of form (dims, data[, attrs, encoding]): ('y', array(['0716-05-19T16:39:19.820', '0716-05-19T07:34:17.820',
       '0716-05-19T07:39:54.820', ..., '0716-05-19T07:35:48.320',
       '0716-05-19T07:39:31.320', '0716-05-19T07:36:01.320'],
      dtype='datetime64[ms]')) to Variable.
Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 132, in as_variable
    obj = Variable(*obj)
          ^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 366, in __init__
    self._data = as_compatible_data(data, fastpath=fastpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 296, in as_compatible_data
    data = _possibly_convert_objects(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 225, in _possibly_convert_objects
    as_series = _as_nanosecond_precision(as_series)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 203, in _as_nanosecond_precision
    return duck_array_ops.astype(data, nanosecond_precision_dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/duck_array_ops.py", line 188, in astype
    return data.astype(dtype, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/generic.py", line 6324, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 451, in astype
    return self.apply(
           ^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 511, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 242, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 184, in astype_array
    values = values.astype(dtype, copy=copy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py", line 674, in astype
    res_values = astype_overflowsafe(self._ndarray, dtype, copy=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/tslibs/np_datetime.pyx", line 306, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 400, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 387, in pandas._libs.tslibs.np_datetime.astype_overflowsafe
  File "pandas/_libs/tslibs/np_datetime.pyx", line 218, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 716-05-19 16:39:19

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 823, in _load_dataset_with_area
    ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 723, in _load_dataset_data
    proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/yaml_reader.py", line 699, in _load_dataset
    projectable = fh.get_dataset(dsid, ds_info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py", line 189, in get_dataset
    res['acq_time'] = ('y', times)
    ~~~^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/dataarray.py", line 826, in __setitem__
    self.coords[key] = value
    ~~~~~~~~~~~^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/coordinates.py", line 41, in __setitem__
    self.update({key: value})
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/coordinates.py", line 169, in update
    coords, indexes = merge_coords(
                      ^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/merge.py", line 553, in merge_coords
    collected = collect_variables_and_indexes(aligned, indexes=indexes)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/merge.py", line 354, in collect_variables_and_indexes
    variable = as_variable(variable, name=name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thomas/.local/lib/python3.11/site-packages/xarray/core/variable.py", line 135, in as_variable
    raise error.__class__(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Could not convert tuple of form (dims, data[, attrs, encoding]): ('y', array(['0716-05-19T16:39:19.820', '0716-05-19T07:34:17.820',
       '0716-05-19T07:39:54.820', ..., '0716-05-19T07:35:48.320',
       '0716-05-19T07:39:31.320', '0716-05-19T07:36:01.320'],
      dtype='datetime64[ms]')) to Variable.
[WARNING: 2023-05-24 22:29:49 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='1', wavelength=WavelengthRange(min=0.58, central=0.63, max=0.68, unit='µm'), resolution=1050, calibration=<1>, modifiers=())

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Info:

  • OS: Arch linux (rolling)
  • Satpy Version: 0.42.2
  • PyResample Version: 1.27.0
  • Readers and writers dependencies (when relevant): None that should affect avhrr_l1b_gaclac

Additional context
Add any other context about the problem here.

gdal_translate can successfuly read all of the files.

Even the working file produces a lot of warnings, here is the log trying to load the working file:

[DEBUG: 2023-05-24 22:30:38 : satpy.readers.yaml_reader] Reading ('/home/thomas/.local/lib/python3.11/site-packages/satpy/etc/readers/avhrr_l1b_gaclac.yaml',)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap.responses')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2349: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(parent)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap.handlers')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydap.tests')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/usr/lib/python3.11/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('zope')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
[DEBUG: 2023-05-24 22:30:38 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-05-24 22:30:38 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-05-24 22:30:38 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2023-05-24 22:30:38 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2023-05-24 22:30:39 : satpy.readers.yaml_reader] Assigning to avhrr_l1b_gaclac: ['/misc/noaa/class2/ftp.avl.class.noaa.gov/8308276947/001/NSS.GHRR.NK.D23133.S0902.E1057.B3003132.WI']
[DEBUG: 2023-05-24 22:30:39 : satpy.composites.config_loader] Looking for composites config file avhrr-3.yaml
[DEBUG: 2023-05-24 22:30:39 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2023-05-24 22:30:39 : satpy.composites.config_loader] Looking for composites config file avhrr-2.yaml
[DEBUG: 2023-05-24 22:30:39 : satpy.composites.config_loader] No composite config found called avhrr-2.yaml
[DEBUG: 2023-05-24 22:30:39 : satpy.composites.config_loader] Looking for composites config file avhrr-1.yaml
[DEBUG: 2023-05-24 22:30:39 : satpy.composites.config_loader] No composite config found called avhrr-1.yaml
[INFO: 2023-05-24 22:30:39 : pygac.klm_reader] Reading NSS.GHRR.NK.D23133.S0902.E1057.B3003132.WI
[DEBUG: 2023-05-24 22:30:39 : pygac.reader] validate header
[DEBUG: 2023-05-24 22:30:39 : pygac.klm_reader] validate header
[DEBUG: 2023-05-24 22:30:39 : pygac.gac_reader] validate header
[DEBUG: 2023-05-24 22:30:39 : pygac.reader] Removed 0 scanline(s) with corrupt scanline numbers
[DEBUG: 2023-05-24 22:30:39 : pygac.reader] Corrected 4 timestamp(s)
[DEBUG: 2023-05-24 22:30:39 : pygac.calibration] Read PyGAC internal calibration coefficients.
[INFO: 2023-05-24 22:30:39 : pygac.calibration] Identified calibration coefficients version "PATMOS-x, v2017r1, with provisional coefficients for MetOp-C".
/home/thomas/.local/lib/python3.11/site-packages/pygac/calibration.py:198: RuntimeWarning: Using CoeffStatus.PROVISIONAL calibration coefficients
  warnings.warn(warning, RuntimeWarning)
[WARNING: 2023-05-24 22:30:39 : pygac.calibration] Using CoeffStatus.PROVISIONAL calibration coefficients
/home/thomas/.local/lib/python3.11/site-packages/geotiepoints/geointerpolator.py:82: RuntimeWarning: invalid value encountered in arcsin
  lats = np.sign(z__) * (90 - np.rad2deg(np.arcsin(np.sqrt(x__ ** 2 + y__ ** 2) / radius)))
[INFO: 2023-05-24 22:30:41 : pygac.reader] TLE filename = ./TLE.txt
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
[DEBUG: 2023-05-24 22:30:41 : satpy.readers.yaml_reader] No coordinates found for DataID(name='longitude', resolution=1050, modifiers=())
[INFO: 2023-05-24 22:30:42 : pygac.reader] TLE filename = ./TLE.txt
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
[DEBUG: 2023-05-24 22:30:42 : satpy.readers.yaml_reader] No coordinates found for DataID(name='latitude', resolution=1050, modifiers=())
/home/thomas/.local/lib/python3.11/site-packages/pygac/calibration.py:337: DeprecationWarning: Using the 'corr' argument is depricated in favor of making the units of the function result clear. Please make any unit conversion outside this function.
  warnings.warn(
/home/thomas/.local/lib/python3.11/site-packages/pygac/calibration.py:544: RuntimeWarning: invalid value encountered in log
  tsE = c2*nu_c / np.log(1.0 + nBB_num / Ne)
[INFO: 2023-05-24 22:30:44 : pygac.reader] TLE filename = ./TLE.txt
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)
/home/thomas/.local/lib/python3.11/site-packages/satpy/readers/avhrr_l1b_gaclac.py:189: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  res['acq_time'] = ('y', times)

I have tried 28 different GAC files from CLASS and only two have worked, I'm not sure what about the files that work is different but it seems unlikely to me that the majority of the files I downloaded are corrupted.

I have also tried running the script in a python 3.10 conda enviroment but it still did not work.

The files are full GAC so they are too large to attach to this github issue so I have uploaded then to my website (hopefully that is alright):
working: i3b.co/NSS.GHRR.NK.D23133.S0902.E1057.B3003132.WI
not working: i3b.co/NSS.GHRR.NK.D23133.S0714.E0907.B3003031.WI
@mraspaud
Copy link
Member

Thanks for reporting this!
@sfinkens do you know what this might be about?

@sfinkens
Copy link
Member

Thanks for the report! I haven't seen this before, and we are using data from NOAA CLASS quite often. Did you uncheck the "include archive header" (or similar) option in your user preferences? Pygac is currently not able to read files with the CLASS archive header included.

@sfinkens
Copy link
Member

Sorry, I was wrong. Pygac can handle the archive header.

@thomasdouwes
Copy link
Author

I have tried with and without the archive header on another file. I have ordered the 2 files I linked in the issue again both with and without the archive header, and I will try that.

@mraspaud
Copy link
Member

Looks like the timestamps are corrupt in the failing file, we're looking at it.

@thomasdouwes
Copy link
Author

I have re-downloaded the files both with and without the archive header and now none of the downloaded files work.
Here are some observations:

new copy of the working file is larger
old: 60.4 MiB
new: 73.8MiB

new file without archive header no longer works on gdal_translate. only with archive header.

image size is different:
old: 409, 13738
new: 409, 16791

different STOP timestamp:
old: 0, day: 0, millisecond: 4294851848
new: 0, day: 0, millisecond: 0

on the "working" file gdal_translate gets the wrong channel, so channel 1 is infrared but on the new file (with archive header) channel 1 is visible.

There is also this bar on the right of the image from gdal_translate on the old file:
image1

Even through it should be the same file, it is different.
It seems even the working file is somewhat invalid.
Also the new file passes a checksum check.

@sfinkens
Copy link
Member

sfinkens commented May 25, 2023 •
edited
Loading

Here are two plots of the raw timestamps before any processing.

  1. NSS.GHRR.NK.D23133.S0902.E1057.B3003132.WI (working). You can see the "usual" timestamp corruptions that Pygac can handle.

n15_timestamps_ok

  1. NSS.GHRR.NK.D23133.S0714.E0907.B3003031.WI (not working): 😱

n15_timestamps_corrupt

I think there's nothing we can do about that, except setting all timestamps to NaT if the year is outside [1970, today] or something like that.

@sfinkens
Copy link
Member

I have re-downloaded the files both with and without the archive header and now none of the downloaded files work.

Oh...

@sfinkens
Copy link
Member

I'll ask my colleagues if they experienced something like that recently.

@mraspaud
Copy link
Member

Satpy/Pygac are dependent on valid times, so corrupt timestamps like these will be a problem.
Maybe an improvement would be to do as you say @sfinkens , put all the times for these corrupt files to NaT. But that might trigger other problems if we need to correct the navigation in some way.

@thomasdouwes thomasdouwes changed the title avhrr_l1b_gaclac fails to read most file from NOAA CLASS avhrr_l1b_gaclac fails to read most files from NOAA CLASS May 25, 2023
@sfinkens
Copy link
Member

I tested with data from my colleagues and found no issues on that day. They got the data via CLASS subscription. So maybe something went wrong with your order...

@thomasdouwes
Copy link
Author

I think I just figured it out, All the files I was downloading where 16 bits/pixel. I just ordered them again with 10 bits/pixel and now they work. I ordered again with 16 bits/pixel and the same error. That explains the "Unexpected number of scanlines!" error too. I must have accidentally selected 10 bits/pixel for the two working files. I guess avhrr_l1b_gaclac only supports 10 bits/pixel data.

@mraspaud
Copy link
Member

@thomasdouwes oh, nice find! thanks for reporting back!
Indeed, I think we always work with 10 bits (as 16bits doesn't add anything else to quality, it's just a convenience for the user). So we should probably mention this in the doc at least.

This was referenced May 31, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@mraspaud @sfinkens @thomasdouwes