Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'day_night' flag to DayNightCompositor for day-only or night-only results #1816

Merged
merged 35 commits into from Sep 17, 2021
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
9cce883
Update __init__.py
yukaribbba Sep 2, 2021
4a75db1
Update __init__.py
yukaribbba Sep 3, 2021
b3b7b35
Update __init__.py
yukaribbba Sep 3, 2021
2ceb218
Update __init__.py
yukaribbba Sep 9, 2021
80ed09a
Merge branch 'main' into feature-daynightcompositor
djhoese Sep 9, 2021
630baaf
tests
yukaribbba Sep 10, 2021
b294981
Revert "tests"
yukaribbba Sep 10, 2021
baf8501
Revert "Merge branch 'main' into feature-daynightcompositor"
yukaribbba Sep 10, 2021
6bcf7eb
Revert "Revert "Merge branch 'main' into feature-daynightcompositor""
yukaribbba Sep 10, 2021
82b84a6
update add_bands_test
yukaribbba Sep 10, 2021
a700799
update DayNightCompositor test
yukaribbba Sep 10, 2021
9bf4207
update DayNightCompositor test
yukaribbba Sep 10, 2021
ca5c3e9
new structure
yukaribbba Sep 11, 2021
f5c873b
Update __init__.py
yukaribbba Sep 11, 2021
73c2cdc
Update __init__.py
yukaribbba Sep 11, 2021
142e4a5
Update __init__.py
yukaribbba Sep 11, 2021
8d58657
Update __init__.py
yukaribbba Sep 11, 2021
926389e
Update __init__.py
yukaribbba Sep 11, 2021
70f43ee
Update __init__.py
yukaribbba Sep 11, 2021
360d876
Update test_composites.py
yukaribbba Sep 11, 2021
0f23b88
Update satpy/composites/__init__.py
yukaribbba Sep 13, 2021
1b14f7f
Update satpy/composites/__init__.py
djhoese Sep 13, 2021
a571014
Update __init__.py
yukaribbba Sep 13, 2021
4c00823
Update __init__.py
yukaribbba Sep 13, 2021
5c265b2
Update __init__.py
yukaribbba Sep 13, 2021
4f32c4c
Update __init__.py
yukaribbba Sep 13, 2021
8cdc9fa
Update __init__.py
yukaribbba Sep 13, 2021
cd55547
Update __init__.py
yukaribbba Sep 13, 2021
8d8377a
Update __init__.py
yukaribbba Sep 13, 2021
af2c6ee
Update __init__.py
yukaribbba Sep 14, 2021
787a6a9
Merge remote-tracking branch 'upstream/main' into feature-daynightcom…
yukaribbba Sep 16, 2021
0a75d4a
Update __init__.py
yukaribbba Sep 16, 2021
b59557f
Update composites.rst
yukaribbba Sep 17, 2021
9922f23
Update AUTHORS.md
yukaribbba Sep 17, 2021
872d95a
Update doc/source/composites.rst
djhoese Sep 17, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
109 changes: 81 additions & 28 deletions satpy/composites/__init__.py
Expand Up @@ -549,71 +549,104 @@ def _insert_palette_colors(channels, palette):


class DayNightCompositor(GenericCompositor):
"""A compositor that blends a day data with night data."""
"""A compositor that blends day data with night data.
Using the `day_night` flag it is also possible to provide only a day product
or only a night product and mask out (make transparent) the opposite portion
of the image (night or day). See the documentation below for more details.
"""

def __init__(self, name, lim_low=85., lim_high=88., **kwargs):
def __init__(self, name, lim_low=85., lim_high=88., day_night="day_night", **kwargs):
"""Collect custom configuration values.

Args:
lim_low (float): lower limit of Sun zenith angle for the
blending of the given channels
lim_high (float): upper limit of Sun zenith angle for the
blending of the given channels
day_night (string): "day_night" means both day and night portions will be kept
"day_only" means only day portion will be kept
"night_only" means only night portion will be kept

"""
self.lim_low = lim_low
self.lim_high = lim_high
self.day_night = day_night
super(DayNightCompositor, self).__init__(name, **kwargs)

def __call__(self, projectables, **kwargs):
"""Generate the composite."""
projectables = self.match_data_arrays(projectables)

day_data = projectables[0]
night_data = projectables[1]
# At least one composite is requested.
foreground_data = projectables[0]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

match_data_arrays should always be called. You can do projectables = self.match_data_arrays(projectables) just above this line. match_data_arrays checks a lot of various things and modifies the provided DataArrays as needed to make them compatible with the compositor or with the rest of Satpy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well when I tested true_color_with_night_ir_alpha I got errors saying Dimension 'bands' size mismatch: 3 != 4 with match_data_arrays. This even occurred under original version. I didn't know what happened...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something related with #1821 and #1813 ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is exactly #1821. I'm trying to finish #1815 and then fix #1821. Everything is just taking longer than I expected. One work around is to switch to an older version of xarray (<0.19.0 if I remember correctly).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok so I'll just put match_data_arrays back here.


lim_low = np.cos(np.deg2rad(self.lim_low))
lim_high = np.cos(np.deg2rad(self.lim_high))
try:
coszen = np.cos(np.deg2rad(projectables[2]))
coszen = np.cos(np.deg2rad(projectables[2 if self.day_night == "day_night" else 1]))
except IndexError:
from pyorbital.astronomy import cos_zen
LOG.debug("Computing sun zenith angles.")
# Get chunking that matches the data
try:
chunks = day_data.sel(bands=day_data['bands'][0]).chunks
chunks = foreground_data.sel(bands=foreground_data['bands'][0]).chunks
except KeyError:
chunks = day_data.chunks
lons, lats = day_data.attrs["area"].get_lonlats(chunks=chunks)
coszen = xr.DataArray(cos_zen(day_data.attrs["start_time"],
chunks = foreground_data.chunks
lons, lats = foreground_data.attrs["area"].get_lonlats(chunks=chunks)
coszen = xr.DataArray(cos_zen(foreground_data.attrs["start_time"],
lons, lats),
dims=['y', 'x'],
coords=[day_data['y'], day_data['x']])
coords=[foreground_data['y'], foreground_data['x']])
# Calculate blending weights
coszen -= np.min((lim_high, lim_low))
coszen /= np.abs(lim_low - lim_high)
coszen = coszen.clip(0, 1)

# Apply enhancements to get images
day_data = enhance2dataset(day_data)
night_data = enhance2dataset(night_data)
# Apply enhancements
foreground_data = enhance2dataset(foreground_data)

# Adjust bands so that they match
# L/RGB -> RGB/RGB
# LA/RGB -> RGBA/RGBA
# RGB/RGBA -> RGBA/RGBA
day_data = add_bands(day_data, night_data['bands'])
night_data = add_bands(night_data, day_data['bands'])
if "only" in self.day_night:
# Only one portion (day or night) is selected. One composite is requested.
# Add alpha band to single L/RGB composite to make the masked-out portion transparent
# L -> LA
# RGB -> RGBA
foreground_data = add_alpha_bands(foreground_data)

# Replace missing channel data with zeros
day_data = zero_missing_data(day_data, night_data)
night_data = zero_missing_data(night_data, day_data)
# No need to replace missing channel data with zeros
# Get metadata
attrs = foreground_data.attrs.copy()

# Get merged metadata
attrs = combine_metadata(day_data, night_data)
# Determine the composite position
day_data = foreground_data if "day" in self.day_night else 0
night_data = foreground_data if "night" in self.day_night else 0

else:
# Both day and night portions are selected. Two composites are requested. Get the second one merged.
background_data = projectables[1]

# Apply enhancements
background_data = enhance2dataset(background_data)

# Adjust bands so that they match
# L/RGB -> RGB/RGB
# LA/RGB -> RGBA/RGBA
# RGB/RGBA -> RGBA/RGBA
foreground_data = add_bands(foreground_data, background_data['bands'])
background_data = add_bands(background_data, foreground_data['bands'])

# Replace missing channel data with zeros
foreground_data = zero_missing_data(foreground_data, background_data)
background_data = zero_missing_data(background_data, foreground_data)

# Get merged metadata
attrs = combine_metadata(foreground_data, background_data)

# Determine the composite position
day_data = foreground_data
night_data = background_data

# Blend the two images together
data = (1 - coszen) * night_data + coszen * day_data
day_portion = coszen * day_data
night_portion = (1 - coszen) * night_data
data = night_portion + day_portion
data.attrs = attrs

# Split to separate bands so the mode is correct
Expand All @@ -622,6 +655,27 @@ def __call__(self, projectables, **kwargs):
return super(DayNightCompositor, self).__call__(data, **kwargs)


def add_alpha_bands(data):
"""Only used for DayNightCompositor.
Add an alpha band to L or RGB composite as prerequisites for the following band matching
to make the masked-out area transparent.
"""
if 'A' not in data['bands'].data:
new_data = [data.sel(bands=band) for band in data['bands'].data]
# Create alpha band based on a copy of the first "real" band
alpha = new_data[0].copy()
alpha.data = da.ones((data.sizes['y'],
data.sizes['x']),
chunks=new_data[0].chunks)
# Rename band to indicate it's alpha
alpha['bands'] = 'A'
new_data.append(alpha)
new_data = xr.concat(new_data, dim='bands')
new_data.attrs['mode'] = data.attrs['mode'] + 'A'
data = new_data
return data


def enhance2dataset(dset, convert_p=False):
"""Return the enhancement dataset *dset* as an array.

Expand Down Expand Up @@ -691,7 +745,6 @@ def add_bands(data, bands):
new_data = xr.concat(new_data, dim='bands')
new_data.attrs['mode'] = data.attrs['mode'] + 'A'
data = new_data

return data


Expand Down
50 changes: 42 additions & 8 deletions satpy/tests/test_composites.py
Expand Up @@ -323,24 +323,60 @@ def setUp(self):
# not used except to check that it matches the data arrays
self.sza.attrs['area'] = my_area

def test_basic_sza(self):
"""Test compositor when SZA data is included."""
def test_daynight_sza(self):
"""Test compositor with both day and night portions when SZA data is included."""
from satpy.composites import DayNightCompositor
comp = DayNightCompositor(name='dn_test')
comp = DayNightCompositor(name='dn_test', day_night="day_night")
res = comp((self.data_a, self.data_b, self.sza))
res = res.compute()
expected = np.array([[0., 0.22122352], [0.5, 1.]])
np.testing.assert_allclose(res.values[0], expected)

def test_basic_area(self):
"""Test compositor when SZA data is not provided."""
def test_daynight_area(self):
"""Test compositor both day and night portions when SZA data is not provided."""
from satpy.composites import DayNightCompositor
comp = DayNightCompositor(name='dn_test')
comp = DayNightCompositor(name='dn_test', day_night="day_night")
res = comp((self.data_a, self.data_b))
res = res.compute()
expected = np.array([[0., 0.33164983], [0.66835017, 1.]])
np.testing.assert_allclose(res.values[0], expected)

def test_night_only_sza(self):
"""Test compositor with night portion when SZA data is included."""
from satpy.composites import DayNightCompositor
comp = DayNightCompositor(name='dn_test', day_night="night_only")
res = comp((self.data_b, self.sza))
res = res.compute()
expected = np.array([[np.nan, 0.], [0.5, 1.]])
np.testing.assert_allclose(res.values[0], expected)

def test_night_only_area(self):
"""Test compositor with night portion when SZA data is not provided."""
from satpy.composites import DayNightCompositor
comp = DayNightCompositor(name='dn_test', day_night="night_only")
res = comp((self.data_b))
res = res.compute()
expected = np.array([[np.nan, 0.], [0., 0.]])
np.testing.assert_allclose(res.values[0], expected)

def test_day_only_sza(self):
"""Test compositor with day portion when SZA data is included."""
from satpy.composites import DayNightCompositor
comp = DayNightCompositor(name='dn_test', day_night="day_only")
res = comp((self.data_a, self.sza))
res = res.compute()
expected = np.array([[0., 0.22122352], [0., 0.]])
np.testing.assert_allclose(res.values[0], expected)

def test_day_only_area(self):
"""Test compositor with day portion when SZA data is not provided."""
from satpy.composites import DayNightCompositor
comp = DayNightCompositor(name='dn_test', day_night="day_only")
res = comp((self.data_a))
res = res.compute()
expected = np.array([[0., 0.33164983], [0.66835017, 1.]])
np.testing.assert_allclose(res.values[0], expected)


class TestFillingCompositor(unittest.TestCase):
"""Test case for the filling compositor."""
Expand Down Expand Up @@ -1049,7 +1085,6 @@ class TestBackgroundCompositor(unittest.TestCase):
def test_call(self):
"""Test the background compositing."""
from satpy.composites import BackgroundCompositor
import numpy as np
comp = BackgroundCompositor("name")

# L mode images
Expand Down Expand Up @@ -1131,7 +1166,6 @@ def test_call(self):
def test_multiple_sensors(self):
"""Test the background compositing from multiple sensor data."""
from satpy.composites import BackgroundCompositor
import numpy as np
comp = BackgroundCompositor("name")

# L mode images
Expand Down