Skip to content

Commit

Permalink
Merge branch 'pytroll:main' into add_features_to_background_compositor
Browse files Browse the repository at this point in the history
  • Loading branch information
yukaribbba committed Feb 23, 2024
2 parents a3fa965 + ef357a5 commit 4037c90
Show file tree
Hide file tree
Showing 41 changed files with 3,496 additions and 2,048 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: CI
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#concurrency
# https://docs.github.com/en/developers/webhooks-and-events/events/github-event-types#pullrequestevent
concurrency:
group: ${{ github.workflow }}-${{ github.event.number }}-${{ github.event.type }}
group: ${{ github.workflow }}-${{ github.event.number || github.event.ref }}-${{ github.event.type }}
cancel-in-progress: true

on: [push, pull_request]
Expand Down
76 changes: 76 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,79 @@
## Version 0.47.0 (2024/02/21)

### Issues Closed

* [Issue 2734](https://github.com/pytroll/satpy/issues/2734) - Using a static image alters time information ([PR 2737](https://github.com/pytroll/satpy/pull/2737) by [@pnuu](https://github.com/pnuu))
* [Issue 2723](https://github.com/pytroll/satpy/issues/2723) - MODIS Satpy scene Don't know how to open the following files: {'MOD021KM.A2017131.1325.061.2017314123114.hdf'}
* [Issue 2719](https://github.com/pytroll/satpy/issues/2719) - Add lat lon to Seviri plots
* [Issue 2718](https://github.com/pytroll/satpy/issues/2718) - Set invert as a modifier when do composite
* [Issue 2712](https://github.com/pytroll/satpy/issues/2712) - mitiff writer add config option to add or not to add the size of a pixel in the proj string ([PR 2713](https://github.com/pytroll/satpy/pull/2713) by [@TAlonglong](https://github.com/TAlonglong))
* [Issue 2710](https://github.com/pytroll/satpy/issues/2710) - scene.save_datasets() outputs different values for AHI_HSD reader with calibration="brightness_temperature"
* [Issue 2708](https://github.com/pytroll/satpy/issues/2708) - this is regarding slstr_l1b geometry
* [Issue 2703](https://github.com/pytroll/satpy/issues/2703) - read swath in loop
* [Issue 2680](https://github.com/pytroll/satpy/issues/2680) - satpy_cf_nc reader cannot read FCI file written with cf writer
* [Issue 2672](https://github.com/pytroll/satpy/issues/2672) - Changes in NWC SAF GEO v2021 data ([PR 2673](https://github.com/pytroll/satpy/pull/2673) by [@pnuu](https://github.com/pnuu))
* [Issue 2630](https://github.com/pytroll/satpy/issues/2630) - wrong start_time with BackgroundCompositor ([PR 2737](https://github.com/pytroll/satpy/pull/2737) by [@pnuu](https://github.com/pnuu))
* [Issue 2447](https://github.com/pytroll/satpy/issues/2447) - add more options to time handling in combine_metadata ([PR 2737](https://github.com/pytroll/satpy/pull/2737) by [@pnuu](https://github.com/pnuu))
* [Issue 2446](https://github.com/pytroll/satpy/issues/2446) - combine metadata in `MultiFiller` ([PR 2737](https://github.com/pytroll/satpy/pull/2737) by [@pnuu](https://github.com/pnuu))
* [Issue 2427](https://github.com/pytroll/satpy/issues/2427) - Wrong start_time, end_time attributes after MultiScene.blend(blend_function=timeseries) ([PR 2737](https://github.com/pytroll/satpy/pull/2737) by [@pnuu](https://github.com/pnuu))
* [Issue 2319](https://github.com/pytroll/satpy/issues/2319) - slstr_l2.yaml points to deleted slstr_l2.py ([PR 2731](https://github.com/pytroll/satpy/pull/2731) by [@djhoese](https://github.com/djhoese))
* [Issue 1921](https://github.com/pytroll/satpy/issues/1921) - Standardize dataset information for SEVIRI and FCI L2 products
* [Issue 1174](https://github.com/pytroll/satpy/issues/1174) - combine_metadata only supports the average of time attrs ([PR 2737](https://github.com/pytroll/satpy/pull/2737) by [@pnuu](https://github.com/pnuu))

In this release 17 issues were closed.

### Pull Requests Merged

#### Bugs fixed

* [PR 2743](https://github.com/pytroll/satpy/pull/2743) - Fix nominal time attributes in SEVIRI HRIT ([](https://github.com/`nominal_start/issues/))
* [PR 2742](https://github.com/pytroll/satpy/pull/2742) - Fix nominal end time in AHI HSD
* [PR 2737](https://github.com/pytroll/satpy/pull/2737) - Change `start_time` and `end_time` handling in `combine_metadata` ([2734](https://github.com/pytroll/satpy/issues/2734), [2630](https://github.com/pytroll/satpy/issues/2630), [2447](https://github.com/pytroll/satpy/issues/2447), [2446](https://github.com/pytroll/satpy/issues/2446), [2427](https://github.com/pytroll/satpy/issues/2427), [1174](https://github.com/pytroll/satpy/issues/1174))
* [PR 2731](https://github.com/pytroll/satpy/pull/2731) - Remove slstr_l2 reader in favor of ghrsst_l2 ([2319](https://github.com/pytroll/satpy/issues/2319))
* [PR 2730](https://github.com/pytroll/satpy/pull/2730) - Pin pytest to fix CI
* [PR 2726](https://github.com/pytroll/satpy/pull/2726) - Fix AGRI L1 C07 having a valid LUT value for its fill value ([565](https://github.com/ssec/polar2grid/issues/565))
* [PR 2713](https://github.com/pytroll/satpy/pull/2713) - Add kwargs config option to turn off mitiff corner correction ([2712](https://github.com/pytroll/satpy/issues/2712))
* [PR 2711](https://github.com/pytroll/satpy/pull/2711) - Add support for NOAA-21 in MiRS limb correction
* [PR 2707](https://github.com/pytroll/satpy/pull/2707) - Fix 'viirs_edr' renaming two sets of dimensions to the same names
* [PR 2700](https://github.com/pytroll/satpy/pull/2700) - Fix eps_l1b reader Delayed usage causing docs failures

#### Features added

* [PR 2746](https://github.com/pytroll/satpy/pull/2746) - Fix concurrency group in ci
* [PR 2745](https://github.com/pytroll/satpy/pull/2745) - Sort reader table by name + diverse fixes
* [PR 2744](https://github.com/pytroll/satpy/pull/2744) - Fix cutoffs for night_ir_alpha and bump up trollimage version
* [PR 2737](https://github.com/pytroll/satpy/pull/2737) - Change `start_time` and `end_time` handling in `combine_metadata` ([2734](https://github.com/pytroll/satpy/issues/2734), [2630](https://github.com/pytroll/satpy/issues/2630), [2447](https://github.com/pytroll/satpy/issues/2447), [2446](https://github.com/pytroll/satpy/issues/2446), [2427](https://github.com/pytroll/satpy/issues/2427), [1174](https://github.com/pytroll/satpy/issues/1174))
* [PR 2728](https://github.com/pytroll/satpy/pull/2728) - Update asv dependencies
* [PR 2720](https://github.com/pytroll/satpy/pull/2720) - Add support for the MERSI-RM instrument on FY-3G
* [PR 2714](https://github.com/pytroll/satpy/pull/2714) - Add QC-based filtering to the VIIRS EDR AOD550 product
* [PR 2675](https://github.com/pytroll/satpy/pull/2675) - Make CF encoding of dataset attributes public
* [PR 2673](https://github.com/pytroll/satpy/pull/2673) - Add NWC SAF GEO v2021 ASIIF-TF and ASII-GW dataset names ([2672](https://github.com/pytroll/satpy/issues/2672))
* [PR 2534](https://github.com/pytroll/satpy/pull/2534) - Add fsspec functionality to `viirs_sdr` reader
* [PR 2441](https://github.com/pytroll/satpy/pull/2441) - Add channel aliases to the CLAVRx reader to facilitate composites

#### Documentation changes

* [PR 2700](https://github.com/pytroll/satpy/pull/2700) - Fix eps_l1b reader Delayed usage causing docs failures

#### Backward incompatible changes

* [PR 2731](https://github.com/pytroll/satpy/pull/2731) - Remove slstr_l2 reader in favor of ghrsst_l2 ([2319](https://github.com/pytroll/satpy/issues/2319))

#### Refactoring

* [PR 2699](https://github.com/pytroll/satpy/pull/2699) - Move Scene.to_hvplot internals to _scene_converters

#### Clean ups

* [PR 2711](https://github.com/pytroll/satpy/pull/2711) - Add support for NOAA-21 in MiRS limb correction
* [PR 2700](https://github.com/pytroll/satpy/pull/2700) - Fix eps_l1b reader Delayed usage causing docs failures
* [PR 2689](https://github.com/pytroll/satpy/pull/2689) - Fix/supress warnings in reader tests ([](https://github.com/and/issues/))
* [PR 2665](https://github.com/pytroll/satpy/pull/2665) - FCI L2 CF harmonization
* [PR 2597](https://github.com/pytroll/satpy/pull/2597) - Update CI to test Python 3.12

In this release 29 pull requests were closed.


## Version 0.46.0 (2023/12/18)

### Issues Closed
Expand Down
2 changes: 1 addition & 1 deletion continuous_integration/environment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,6 @@ dependencies:
- bokeh
- pip:
- trollsift
- trollimage>=1.20
- trollimage>=1.23
- pyspectral
- pyorbital
7 changes: 6 additions & 1 deletion doc/source/_static/main.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
$(document).ready( function () {
$('table.datatable').DataTable( {
"paging": false,
"dom": 'lfitp'
"layout": {
'topStart': 'info',
'topEnd': 'search',
'bottomStart': null
},
"order": [[0, 'asc']]
} );
} );
4 changes: 2 additions & 2 deletions doc/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -194,11 +194,11 @@ def __getattr__(cls, name):

html_css_files = [
"theme_overrides.css", # override wide tables in RTD theme
"https://cdn.datatables.net/1.10.23/css/jquery.dataTables.min.css",
"https://cdn.datatables.net/v/dt/dt-2.0.0/datatables.min.css",
]

html_js_files = [
"https://cdn.datatables.net/1.10.23/js/jquery.dataTables.min.js",
"https://cdn.datatables.net/v/dt/dt-2.0.0/datatables.min.js",
"main.js",
]

Expand Down
3 changes: 2 additions & 1 deletion doc/source/dev_guide/custom_reader.rst
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,8 @@ The parameters to provide in this section are:
file format. This can be multiline if formatted properly in YAML (see
example below).
status
The status of the reader (one of: Nominal, Beta, Alpha)
The status of the reader (one of: Nominal, Beta, Alpha, Defunct; see :ref:`Status Description <Status Description>`
for more details).
supports_fsspec
If the reader supports reading data via fsspec (either true or false).
sensors
Expand Down
24 changes: 24 additions & 0 deletions doc/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,30 @@ Documentation

.. include:: reader_table.rst

.. _Status Description:
.. note::

Status description:

Defunct
Most likely the reader is not functional. If it is there is a good chance of
bugs and/or performance problems (e.g. not ported to dask/xarray yet). Future
development is unclear. Users are encouraged to contribute (see section
:doc:`dev_guide/CONTRIBUTING` and/or get help on Slack or by opening a Github issue).

Alpha
This denotes early development status. Reader is functional and implements some
or all of the nominal features. There might be bugs. Exactness of results is
not be guaranteed. Use at your own risk.

Beta
This denotes final developement status. Reader is functional and implements all
nominal features. Results should be dependable but there might be bugs. Users
are actively encouraged to test and report bugs.

Nominal
This denotes a finished status. Reader is functional and most likely no new
features will be introduced. It has been tested and there are no known bugs.

Indices and tables
==================
Expand Down
8 changes: 4 additions & 4 deletions satpy/cf/attrs.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,8 @@ def _encode_to_cf(obj):
return _encode_python_objects(obj)


def _encode_nc_attrs(attrs):
"""Encode dataset attributes in a netcdf compatible datatype.
def encode_attrs_to_cf(attrs):
"""Encode dataset attributes as a netcdf compatible datatype.
Args:
attrs (dict):
Expand Down Expand Up @@ -161,7 +161,7 @@ def preprocess_attrs(
if flatten_attrs:
data_arr.attrs = flatten_dict(data_arr.attrs)

data_arr.attrs = _encode_nc_attrs(data_arr.attrs)
data_arr.attrs = encode_attrs_to_cf(data_arr.attrs)

return data_arr

Expand Down Expand Up @@ -224,7 +224,7 @@ def preprocess_header_attrs(header_attrs, flatten_attrs=False):
if header_attrs is not None:
if flatten_attrs:
header_attrs = flatten_dict(header_attrs)
header_attrs = _encode_nc_attrs(header_attrs) # OrderedDict
header_attrs = encode_attrs_to_cf(header_attrs) # OrderedDict
else:
header_attrs = {}
header_attrs = _add_history(header_attrs)
Expand Down
74 changes: 74 additions & 0 deletions satpy/cf/decoding.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Copyright (c) 2023 Satpy developers
#
# This file is part of satpy.
#
# satpy is free software: you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
#
# satpy is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along with
# satpy. If not, see <http://www.gnu.org/licenses/>.
"""CF decoding."""
import copy
import json
from datetime import datetime


def decode_attrs(attrs):
"""Decode CF-encoded attributes to Python object.
Converts timestamps to datetime and strings starting with "{" to
dictionary.
Args:
attrs (dict): Attributes to be decoded
Returns (dict): Decoded attributes
"""
attrs = copy.deepcopy(attrs)
_decode_dict_type_attrs(attrs)
_decode_timestamps(attrs)
return attrs


def _decode_dict_type_attrs(attrs):
for key, val in attrs.items():
attrs[key] = _str2dict(val)


def _str2dict(val):
"""Convert string to dictionary."""
if isinstance(val, str) and val.startswith("{"):
val = json.loads(val, object_hook=_datetime_parser_json)
return val


def _decode_timestamps(attrs):
for key, value in attrs.items():
timestamp = _str2datetime(value)
if timestamp:
attrs[key] = timestamp


def _datetime_parser_json(json_dict):
"""Traverse JSON dictionary and parse timestamps."""
for key, value in json_dict.items():
timestamp = _str2datetime(value)
if timestamp:
json_dict[key] = timestamp
return json_dict


def _str2datetime(string):
"""Convert string to datetime object."""
try:
return datetime.fromisoformat(string)
except (TypeError, ValueError):
return None
89 changes: 89 additions & 0 deletions satpy/etc/composites/mersi-rm.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
sensor_name: visir/mersi-rm

modifiers:
rayleigh_corrected:
modifier: !!python/name:satpy.modifiers.PSPRayleighReflectance
atmosphere: us-standard
aerosol_type: rayleigh_only
prerequisites:
- name: '1'
modifiers: [sunz_corrected]
optional_prerequisites:
- name: satellite_azimuth_angle
- name: satellite_zenith_angle
- name: solar_azimuth_angle
- name: solar_zenith_angle
sunz_corrected:
modifier: !!python/name:satpy.modifiers.SunZenithCorrector
prerequisites:
- solar_zenith_angle

nir_reflectance:
modifier: !!python/name:satpy.modifiers.NIRReflectance
prerequisites:
- name: '7'
optional_prerequisites:
- solar_zenith_angle


composites:
natural_color:
compositor: !!python/name:satpy.composites.RatioSharpenedRGB
prerequisites:
- name: '5'
modifiers: [sunz_corrected]
- name: '3'
modifiers: [sunz_corrected]
- name: '1'
modifiers: [sunz_corrected]
standard_name: natural_color

overview_raw:
compositor: !!python/name:satpy.composites.GenericCompositor
prerequisites:
- name: '1'
- name: '2'
- name: '7'
standard_name: overview

overview:
compositor: !!python/name:satpy.composites.GenericCompositor
prerequisites:
- name: '1'
modifiers: [sunz_corrected]
- name: '2'
modifiers: [sunz_corrected]
- name: '7'
standard_name: overview

cloudtop:
compositor: !!python/name:satpy.composites.GenericCompositor
prerequisites:
- name: '7'
- name: '8'
- name: '9'
standard_name: cloudtop

day_microphysics:
compositor: !!python/name:satpy.composites.GenericCompositor
prerequisites:
- name: '2'
modifiers: [sunz_corrected]
- name: '7'
modifiers: [nir_reflectance]
- name: '8'
standard_name: day_microphysics

night_fog:
compositor: !!python/name:satpy.composites.GenericCompositor
prerequisites:
- compositor: !!python/name:satpy.composites.DifferenceCompositor
prerequisites:
- name: '8'
- name: '7'
- compositor: !!python/name:satpy.composites.DifferenceCompositor
prerequisites:
- name: '7'
- name: '6'
- name: '7'
standard_name: night_fog
1 change: 1 addition & 0 deletions satpy/etc/composites/seviri.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ composites:
- name: HRV
modifiers: [sunz_corrected]
- IR_108

hrv_fog:
compositor: !!python/name:satpy.composites.GenericCompositor
standard_name: hrv_fog
Expand Down

0 comments on commit 4037c90

Please sign in to comment.