Skip to content

Commit

Permalink
Merge branch 'main' into feature-insat3d-reader
Browse files Browse the repository at this point in the history
  • Loading branch information
mraspaud committed Nov 17, 2022
2 parents da08988 + c81e02e commit 321c62f
Show file tree
Hide file tree
Showing 45 changed files with 3,865 additions and 668 deletions.
83 changes: 83 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,86 @@
## Version 0.38.0 (2022/11/11)

### Issues Closed

* [Issue 2258](https://github.com/pytroll/satpy/issues/2258) - VIIRS day_microphysics array broadcast error ([PR 2260](https://github.com/pytroll/satpy/pull/2260) by [@djhoese](https://github.com/djhoese))
* [Issue 2250](https://github.com/pytroll/satpy/issues/2250) - Is reader modis_l1b available?
* [Issue 2249](https://github.com/pytroll/satpy/issues/2249) - ahi-hsd raise keyerror when load B08 in dataquery
* [Issue 2241](https://github.com/pytroll/satpy/issues/2241) - Slow loading speed of TROPOMI L2 product
* [Issue 2234](https://github.com/pytroll/satpy/issues/2234) - Scene `modifiers` keyword argument has no effect ([PR 2235](https://github.com/pytroll/satpy/pull/2235) by [@djhoese](https://github.com/djhoese))
* [Issue 2233](https://github.com/pytroll/satpy/issues/2233) - 'cached_property' from 'functools' - seems incompatible with python3.7
* [Issue 2228](https://github.com/pytroll/satpy/issues/2228) - Question: Why the ellipsoid of a geostationary satellite image is not typical WGS84?
* [Issue 2227](https://github.com/pytroll/satpy/issues/2227) - CF writer output wrong for area with geographic CRS ([PR 2236](https://github.com/pytroll/satpy/pull/2236) by [@gerritholl](https://github.com/gerritholl))
* [Issue 2215](https://github.com/pytroll/satpy/issues/2215) - Abi l2 nc reader can't handle AOD product ([PR 2216](https://github.com/pytroll/satpy/pull/2216) by [@mraspaud](https://github.com/mraspaud))
* [Issue 2208](https://github.com/pytroll/satpy/issues/2208) - hy2_scat_l2b_h5 reader does not work any more due to space in `valid range` attribute ([PR 2268](https://github.com/pytroll/satpy/pull/2268) by [@TAlonglong](https://github.com/TAlonglong))
* [Issue 2206](https://github.com/pytroll/satpy/issues/2206) - If you do `python -c "import scipy.sparse"` do you get a similar error?
* [Issue 2202](https://github.com/pytroll/satpy/issues/2202) - AttributeError: 'AreaDefinition' object has no attribute 'crs'
* [Issue 2192](https://github.com/pytroll/satpy/issues/2192) - Available Readers Problem
* [Issue 2189](https://github.com/pytroll/satpy/issues/2189) - Supported readers page does not include many readers ([PR 2191](https://github.com/pytroll/satpy/pull/2191) by [@BENR0](https://github.com/BENR0))
* [Issue 2183](https://github.com/pytroll/satpy/issues/2183) - Reading bzipped Seviri HRIT segment crashes when reading the data from disk ([PR 2185](https://github.com/pytroll/satpy/pull/2185) by [@mraspaud](https://github.com/mraspaud))
* [Issue 2170](https://github.com/pytroll/satpy/issues/2170) - satpy_cf_nc Reader Fails to Read Data Written by cf Writer ([PR 2176](https://github.com/pytroll/satpy/pull/2176) by [@mraspaud](https://github.com/mraspaud))
* [Issue 2154](https://github.com/pytroll/satpy/issues/2154) - module 'ntpath' has no attribute 'sep'"
* [Issue 2111](https://github.com/pytroll/satpy/issues/2111) - Archived GOES datasets not loading. AttributeError: 'area' object has no attribute 'crs'
* [Issue 1929](https://github.com/pytroll/satpy/issues/1929) - Two test failures in test_goes_imager_nc.py with Python 3.10
* [Issue 1672](https://github.com/pytroll/satpy/issues/1672) - Add AreaDefinition support to the 'satpy_cf_nc' reader ([PR 1695](https://github.com/pytroll/satpy/pull/1695) by [@BENR0](https://github.com/BENR0))

In this release 20 issues were closed.

### Pull Requests Merged

#### Bugs fixed

* [PR 2262](https://github.com/pytroll/satpy/pull/2262) - Fix ratio sharpening not sharing invalid mask between bands ([556](https://github.com/ssec/polar2grid/issues/556))
* [PR 2260](https://github.com/pytroll/satpy/pull/2260) - Fix VIIRS L1B I-band angle names being inconsistent with VIIRS SDR ([2258](https://github.com/pytroll/satpy/issues/2258))
* [PR 2257](https://github.com/pytroll/satpy/pull/2257) - Fix failure creating directory if it already exists.
* [PR 2246](https://github.com/pytroll/satpy/pull/2246) - Fix MODIS reader tests failing with new geotiepoints
* [PR 2239](https://github.com/pytroll/satpy/pull/2239) - Fix incorrect rows_per_scan in 'acspo' reader ([498](https://github.com/ssec/polar2grid/issues/498))
* [PR 2236](https://github.com/pytroll/satpy/pull/2236) - CF-compliant storage for lon/lat case ([2227](https://github.com/pytroll/satpy/issues/2227))
* [PR 2235](https://github.com/pytroll/satpy/pull/2235) - Fix Scene.load modifiers keyword argument having no effect ([2234](https://github.com/pytroll/satpy/issues/2234))
* [PR 2232](https://github.com/pytroll/satpy/pull/2232) - Make longitude masker & filler single band
* [PR 2221](https://github.com/pytroll/satpy/pull/2221) - Fix ici after mws merge
* [PR 2220](https://github.com/pytroll/satpy/pull/2220) - Fix CLAVR-x configuration in 'awips_tiled' writer to be backwards compatible
* [PR 2216](https://github.com/pytroll/satpy/pull/2216) - Fix coord renaming for AOD product in 'abi_l2_nc' reader ([2215](https://github.com/pytroll/satpy/issues/2215))
* [PR 2210](https://github.com/pytroll/satpy/pull/2210) - Fix VIIRS EDR Active Fires reader for new format and fix fine/coarse 1D swath handling ([458](https://github.com/ssec/polar2grid/issues/458))
* [PR 2190](https://github.com/pytroll/satpy/pull/2190) - Fix some enhancements producing dask arrays wrapped in dask arrays
* [PR 2185](https://github.com/pytroll/satpy/pull/2185) - Refactor HRIT readers to be smarter about compression and reading data ([2183](https://github.com/pytroll/satpy/issues/2183))
* [PR 2177](https://github.com/pytroll/satpy/pull/2177) - Fix plugins not working with new versions of setuptools
* [PR 2176](https://github.com/pytroll/satpy/pull/2176) - Fix cf write-read roundtrip ([2170](https://github.com/pytroll/satpy/issues/2170))
* [PR 2166](https://github.com/pytroll/satpy/pull/2166) - Correct the sun azimuth angle range within satpy.

#### Features added

* [PR 2230](https://github.com/pytroll/satpy/pull/2230) - Add support for compressed FSFiles to HRIT readers
* [PR 2209](https://github.com/pytroll/satpy/pull/2209) - Update seadas_l2 reader to handle alternative NetCDF file format ([457](https://github.com/ssec/polar2grid/issues/457))
* [PR 2207](https://github.com/pytroll/satpy/pull/2207) - Add SEVIRI level 2 AMV BUFR
* [PR 2203](https://github.com/pytroll/satpy/pull/2203) - Fix experimental dependency stdlibc++ issues in CI
* [PR 2198](https://github.com/pytroll/satpy/pull/2198) - Add warning for SEVIRI native reader in case of bad data
* [PR 2187](https://github.com/pytroll/satpy/pull/2187) - adding a reader for ATMS level1b data
* [PR 2185](https://github.com/pytroll/satpy/pull/2185) - Refactor HRIT readers to be smarter about compression and reading data ([2183](https://github.com/pytroll/satpy/issues/2183))
* [PR 2175](https://github.com/pytroll/satpy/pull/2175) - Add utility function to compute the relative azimuth angle.
* [PR 2164](https://github.com/pytroll/satpy/pull/2164) - Add low level moisture composite
* [PR 2125](https://github.com/pytroll/satpy/pull/2125) - Add reader for FY-4B / GHI data
* [PR 2120](https://github.com/pytroll/satpy/pull/2120) - Add reader for MWS onboard EPS-SG-A
* [PR 2118](https://github.com/pytroll/satpy/pull/2118) - Add a reader for EPS-SG Ice Cloud Imager
* [PR 1695](https://github.com/pytroll/satpy/pull/1695) - Add `get_area_def` to cf reader ([1672](https://github.com/pytroll/satpy/issues/1672))

#### Documentation changes

* [PR 2247](https://github.com/pytroll/satpy/pull/2247) - Document behaviour on default enhancement
* [PR 2225](https://github.com/pytroll/satpy/pull/2225) - Update writer table docs
* [PR 2200](https://github.com/pytroll/satpy/pull/2200) - Remove mention of fallback to GDAL in geotiff writer
* [PR 2195](https://github.com/pytroll/satpy/pull/2195) - Add additional logging information about enhancements being used
* [PR 2191](https://github.com/pytroll/satpy/pull/2191) - Fix automatic reader table not listing readers with missing dependencies ([2189](https://github.com/pytroll/satpy/issues/2189))

#### Clean ups

* [PR 2268](https://github.com/pytroll/satpy/pull/2268) - Cleanup hy2 reader ([2208](https://github.com/pytroll/satpy/issues/2208))
* [PR 2252](https://github.com/pytroll/satpy/pull/2252) - Create dependabot.yml
* [PR 2240](https://github.com/pytroll/satpy/pull/2240) - Refactor RGB ratio sharpening again for better performance
* [PR 2205](https://github.com/pytroll/satpy/pull/2205) - Update URL to rasterio repository in CI

In this release 39 pull requests were closed.


## Version 0.37.1 (2022/08/15)

### Issues Closed
Expand Down
3 changes: 2 additions & 1 deletion doc/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@ def __getattr__(cls, name):
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage',
'sphinx.ext.doctest', 'sphinx.ext.napoleon', 'sphinx.ext.autosummary', 'doi_role',
'sphinx.ext.viewcode', 'sphinxcontrib.apidoc']
'sphinx.ext.viewcode', 'sphinxcontrib.apidoc',
'sphinx.ext.mathjax']

# API docs
apidoc_module_dir = "../../satpy"
Expand Down
6 changes: 6 additions & 0 deletions doc/source/readers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,12 @@ satpy cf nc readers
hdf5 based readers
------------------

.. automodule:: satpy.readers.agri_l1
:noindex:

.. automodule:: satpy.readers.ghi_l1
:noindex:

Arctica-M N1 HDF5 format reader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
61 changes: 61 additions & 0 deletions satpy/composites/agri.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Copyright (c) 2015-2022 Satpy developers
#
# This file is part of satpy.
#
# satpy is free software: you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
#
# satpy is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along with
# satpy. If not, see <http://www.gnu.org/licenses/>.
"""Composite classes for the AGRI instrument."""

import logging

from satpy.composites import GenericCompositor

LOG = logging.getLogger(__name__)


class SimulatedRed(GenericCompositor):
"""A single-band dataset resembling a Red (0.64 µm) band.
This compositor creates a single band product by combining two
other bands by preset amounts. The general formula with
dependencies (d) and fractions (f) is::
result = (f1 * d1 - f2 * d2) / f3
See the `fractions` keyword argument for more information.
The default setup is to use:
- f1 = 1.0
- f2 = 0.13
- f3 = 0.87
"""

def __init__(self, name, fractions=(1.0, 0.13, 0.87), **kwargs):
"""Initialize fractions for input channels.
Args:
name (str): Name of this composite
fractions (iterable): Fractions of each input band to include in the result.
"""
self.fractions = fractions
super(SimulatedRed, self).__init__(name, **kwargs)

def __call__(self, projectables, optional_datasets=None, **attrs):
"""Generate the single band composite."""
c1, c2 = self.match_data_arrays(projectables)
res = (c1 * self.fractions[0] - c2 * self.fractions[1]) / self.fractions[2]
res.attrs = c1.attrs.copy()
return super(SimulatedRed, self).__call__((res,), **attrs)
110 changes: 110 additions & 0 deletions satpy/enhancements/atmosphere.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Copyright (c) 2022- Satpy developers
#
# This file is part of satpy.
#
# satpy is free software: you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
#
# satpy is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along with
# satpy. If not, see <http://www.gnu.org/licenses/>.
"""Enhancements related to visualising atmospheric phenomena."""

import datetime

import dask.array as da
import xarray as xr


def essl_moisture(img, low=1.1, high=1.6) -> None:
r"""Low level moisture by European Severe Storms Laboratory (ESSL).
Expects a mode L image with data corresponding to the ratio of the
calibrated reflectances for the 0.86 µm and 0.906 µm channel.
This composite and its colorisation were developed by ESSL.
Ratio values are scaled from the range ``[low, high]``, which is by default
between 1.1 and 1.6, but might be tuned based on region or sensor,
to ``[0, 1]``. Values outside this range are clipped. Color values
for red, green, and blue are calculated as follows, where ``x`` is the
ratio between the 0.86 µm and 0.905 µm channels:
.. math::
R = \max(1.375 - 2.67 x, -0.75 + x) \\
G = 1 - \frac{8x}{7} \\
B = \max(0.75 - 1.5 x, 0.25 - (x - 0.75)^2) \\
The value of ``img.data`` is modified in-place.
A color interpretation guide is pending further adjustments to the
parameters for current and future sensors.
Args:
img: XRImage containing the relevant composite
low: optional, low end for scaling, defaults to 1.1
high: optional, high end for scaling, defaults to 1.6
"""
ratio = img.data
if _is_fci_test_data(img.data):
# Due to a bug in the FCI pre-launch simulated test data,
# the 0.86 µm channel is too bright. To correct for this, its
# reflectances should be multiplied by 0.8.
ratio *= 0.8

with xr.set_options(keep_attrs=True):
ratio = _scale_and_clip(ratio, low, high)
red = _calc_essl_red(ratio)
green = _calc_essl_green(ratio)
blue = _calc_essl_blue(ratio)
data = xr.concat([red, green, blue], dim="bands")
data.attrs["mode"] = "RGB"
data["bands"] = ["R", "G", "B"]
img.data = data


def _scale_and_clip(ratio, low, high):
"""Scale ratio values to [0, 1] and clip values outside this range."""
scaled = (ratio - low) / (high - low)
scaled.data = da.clip(scaled.data, 0, 1)
return scaled


def _calc_essl_red(ratio):
"""Calculate values for red based on scaled and clipped ratio."""
red_a = 1.375 - 2.67 * ratio
red_b = -0.75 + ratio
red = xr.where(red_a > red_b, red_a, red_b)
red.data = da.clip(red.data, 0, 1)
return red


def _calc_essl_green(ratio):
"""Calculate values for green based on scaled and clipped ratio."""
green = 1 - (8/7) * ratio
green.data = da.clip(green.data, 0, 1)
return green


def _calc_essl_blue(ratio):
"""Calculate values for blue based on scaled and clipped ratio."""
blue_a = 0.75 - 1.5 * ratio
blue_b = 0.25 - (ratio - 0.75)**2
blue = xr.where(blue_a > blue_b, blue_a, blue_b)
blue.data = da.clip(blue.data, 0, 1)
return blue


def _is_fci_test_data(data):
"""Check if we are working with FCI test data."""
return ("sensor" in data.attrs and
"start_time" in data.attrs and
data.attrs["sensor"] == "fci" and
isinstance(data.attrs["start_time"], datetime.datetime) and
data.attrs["start_time"] < datetime.datetime(2022, 11, 30))
44 changes: 42 additions & 2 deletions satpy/etc/composites/agri.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,55 @@ composites:
modifiers: [sunz_corrected]
standard_name: toa_bidirectional_reflectance

true_color:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
green_nocorr:
compositor: !!python/name:satpy.composites.abi.SimulatedGreen
# FUTURE: Set a wavelength...see what happens. Dependency finding
# probably wouldn't work.
prerequisites:
# should we be using the most corrected or least corrected inputs?
- name: C01
- name: C02
- name: C03
standard_name: toa_bidirectional_reflectance

pseudored:
compositor: !!python/name:satpy.composites.agri.SimulatedRed
# FUTURE: Set a wavelength...see what happens. Dependency finding
# probably wouldn't work.
prerequisites:
# should we be using the most corrected or least corrected inputs?
- name: C02
modifiers: [sunz_corrected, rayleigh_corrected]
- name: C03
modifiers: [sunz_corrected]
standard_name: toa_bidirectional_reflectance

pseudored_nocorr:
compositor: !!python/name:satpy.composites.agri.SimulatedRed
# FUTURE: Set a wavelength...see what happens. Dependency finding
# probably wouldn't work.
prerequisites:
# should we be using the most corrected or least corrected inputs?
- name: C02
- name: C03
standard_name: toa_bidirectional_reflectance

true_color:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- name: pseudored
- name: green
- name: C01
modifiers: [sunz_corrected, rayleigh_corrected]
standard_name: true_color

true_color_nocorr:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- name: pseudored_nocorr
- name: green_nocorr
- name: C01
standard_name: true_color
#
cloud_phase_distinction:
description: >
Expand Down
21 changes: 21 additions & 0 deletions satpy/etc/composites/ghi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
sensor_name: visir/ghi

composites:
true_color:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- name: C04
modifiers: [sunz_corrected, rayleigh_corrected]
- name: C03
modifiers: [sunz_corrected, rayleigh_corrected]
- name: C02
modifiers: [sunz_corrected, rayleigh_corrected]
standard_name: true_color

true_color_nocorr:
compositor: !!python/name:satpy.composites.SelfSharpenedRGB
prerequisites:
- name: C04
- name: C03
- name: C02
standard_name: true_color
Loading

0 comments on commit 321c62f

Please sign in to comment.