Support CLASS packed viirs files in viirs_sdr reader #538

mraspaud · 2018-12-07T15:55:11Z

This PR adds support for viirs sdr files containing more than one band/dataset at the time.

Tests added
Tests passed
Passes git diff origin/master -- "*py" | flake8 --diff
Fully documented

satpy/readers/viirs_sdr.py

mraspaud · 2018-12-07T15:55:57Z

What is left is find a way to prioritize TC geo files.

djhoese · 2018-12-08T13:13:20Z

satpy/etc/readers/viirs_sdr.yaml

-    file_key: 'All_Data/{file_group}_All/SolarZenithAngle'
+    file_type: generic_file
+    dataset_groups: [GDNBO]
+    file_key: 'All_Data/{dataset_group}_All/SolarZenithAngle'
  DNB_LZA:
    name: dnb_lunar_zenith_angle
    standard_name: lunar_zenith_angle
    resolution: 743
    coordinates: [dnb_longitude, dnb_latitude]
    file_type: gdnbo


Does this need to be file_type: generic_file to?

satpy/etc/readers/viirs_sdr.yaml

djhoese

get_bounding_box needs to be handled too, right? And we'll have to figure out how to handle the custom Reader at the bottom of the module which let's the user force TC or not.

satpy/readers/viirs_sdr.py

Conflicts: satpy/etc/readers/viirs_sdr.yaml

satpy/readers/viirs_sdr.py

djhoese · 2019-03-11T12:47:46Z

Any idea if this PR works with old DNB files that didn't have TC data?

mraspaud · 2019-03-11T12:48:30Z

use_tc can be either True, False or None, with None being 'use TC if available, non-TC otherwise'.

mraspaud · 2019-03-11T12:49:07Z

This PR will work with old DNB file as long as use_tc isn't True

coveralls · 2019-03-11T22:46:11Z

Coverage decreased (-0.2%) to 78.124% when pulling 4bf7b8d on mraspaud:feature-viirs-class into 0cefc09 on pytroll:master.

coveralls · 2019-03-11T22:46:12Z

Coverage increased (+0.09%) to 78.466% when pulling fd77a96 on mraspaud:feature-viirs-class into fb39c41 on pytroll:master.

codecov · 2019-03-11T23:05:31Z

Codecov Report

Merging #538 into master will increase coverage by 0.07%.
The diff coverage is 86.64%.

@@            Coverage Diff             @@
##           master     #538      +/-   ##
==========================================
+ Coverage   78.39%   78.46%   +0.07%     
==========================================
  Files         138      138              
  Lines       20013    20252     +239     
==========================================
+ Hits        15689    15891     +202     
- Misses       4324     4361      +37

Impacted Files	Coverage Δ
satpy/tests/reader_tests/test_ahi_hsd.py	`96.51% <ø> (ø)`	⬆️
satpy/multiscene.py	`87.08% <ø> (+0.53%)`	⬆️
satpy/tests/reader_tests/test_goes_imager_nc.py	`98.13% <ø> (-0.01%)`	⬇️
satpy/writers/geotiff.py	`42.01% <0%> (ø)`	⬆️
satpy/composites/viirs.py	`83.71% <100%> (ø)`	⬆️
satpy/readers/yaml_reader.py	`92.3% <100%> (+0.04%)`	⬆️
satpy/tests/reader_tests/test_viirs_sdr.py	`92.28% <84.86%> (-6.43%)`	⬇️
satpy/readers/viirs_sdr.py	`84.75% <88.33%> (+3.82%)`	⬆️
satpy/readers/nwcsaf_nc.py	`35.18% <0%> (-0.67%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb39c41...fd77a96. Read the comment docs.

mraspaud · 2019-03-13T08:21:06Z

@djhoese any more change requests on this PR ?

djhoese

Nice job some how cleaning up the code while also adding more functionality. I had a couple small docstring things. Otherwise, you obviously tested this with files where multiple datasets were combined in to one file, but what about multiple times in one file (multiple granules)?

satpy/composites/viirs.py

satpy/writers/geotiff.py

satpy/readers/viirs_sdr.py

mraspaud · 2019-03-14T09:17:33Z

Regarding multiple times in one file (multiple granules), I test this with the bounding box test.

djhoese

LGTM

Support CLASS packed viirs files in viirs_sdr reader

dd84cc1

mraspaud added enhancement code enhancements, features, improvements component:readers work in progress labels Dec 7, 2018

mraspaud self-assigned this Dec 7, 2018

stickler-ci reviewed Dec 7, 2018

View reviewed changes

satpy/readers/viirs_sdr.py Show resolved Hide resolved

djhoese reviewed Dec 8, 2018

View reviewed changes

satpy/etc/readers/viirs_sdr.yaml Show resolved Hide resolved

djhoese requested changes Dec 8, 2018

View reviewed changes

mraspaud added 2 commits January 14, 2019 21:08

Fix moon fraction from viirs_sdr enhancements

6697f79

Fix filetype for lza in viirs_sdr

7c466b2

stickler-ci reviewed Mar 4, 2019

View reviewed changes

satpy/readers/viirs_sdr.py Show resolved Hide resolved

Merge branch 'master' into feature-viirs-class

d267817

Conflicts: satpy/etc/readers/viirs_sdr.yaml

stickler-ci reviewed Mar 5, 2019

View reviewed changes

satpy/readers/viirs_sdr.py Show resolved Hide resolved

Remove unneeded file

0bd23a1

stickler-ci reviewed Mar 5, 2019

View reviewed changes

satpy/readers/viirs_sdr.py Show resolved Hide resolved

mraspaud added 8 commits March 5, 2019 15:59

Add support for single-valued metadata items in aggregated viirs data

7ecf1ec

Update support for TC/nonTC files in viirs aggregated files

0095b99

Add support for DNB TC navigation for viirs SDR

b837e82

Cleanup viirs sdr config file

bff75bd

Remove obsolete function

110d8ac

Fix viirs_sdr tests

f36c48f

Add TC support for M and I bands

8b3d582

Fix viirs_sdr to pass unittests

ad71a88

mraspaud added 2 commits March 11, 2019 23:30

Restructure viirs_sdr for angles selection and bounding boxes

7a46afd

Provide example filenames for viirs_sdr file patterns

4bf7b8d

mraspaud added 2 commits March 11, 2019 23:58

Document viirs_sdr

18c2e2a

Document more

e0a51ea

mraspaud requested a review from djhoese March 11, 2019 23:18

mraspaud removed the work in progress label Mar 11, 2019

mraspaud added this to the v0.13 milestone Mar 12, 2019

djhoese requested changes Mar 13, 2019

View reviewed changes

satpy/composites/viirs.py Show resolved Hide resolved

satpy/writers/geotiff.py Show resolved Hide resolved

satpy/readers/viirs_sdr.py Show resolved Hide resolved

satpy/readers/viirs_sdr.py Outdated Show resolved Hide resolved

satpy/readers/viirs_sdr.py Outdated Show resolved Hide resolved

mraspaud added 3 commits March 14, 2019 10:20

Merge branch 'master' into feature-viirs-class

2fbd406

Fix viirs_sdr docstrings

45d0524

Allow extra kwargs to the viirs_sdr filehandler

fd77a96

djhoese approved these changes Mar 15, 2019

View reviewed changes

djhoese merged commit be6e5a0 into pytroll:master Mar 15, 2019

mraspaud deleted the feature-viirs-class branch March 15, 2019 13:10

mraspaud mentioned this pull request Apr 3, 2019

Fix VIIRS SDR reading of visible channels at nighttime #693

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support CLASS packed viirs files in viirs_sdr reader #538

Support CLASS packed viirs files in viirs_sdr reader #538

mraspaud commented Dec 7, 2018 •

edited

mraspaud commented Dec 7, 2018

djhoese Dec 8, 2018

djhoese left a comment

djhoese commented Mar 11, 2019

mraspaud commented Mar 11, 2019

mraspaud commented Mar 11, 2019

coveralls commented Mar 11, 2019

coveralls commented Mar 11, 2019 •

edited

codecov bot commented Mar 11, 2019 •

edited

mraspaud commented Mar 13, 2019

djhoese left a comment

mraspaud commented Mar 14, 2019

djhoese left a comment

Support CLASS packed viirs files in viirs_sdr reader #538

Support CLASS packed viirs files in viirs_sdr reader #538

Conversation

mraspaud commented Dec 7, 2018 • edited

mraspaud commented Dec 7, 2018

djhoese Dec 8, 2018

Choose a reason for hiding this comment

djhoese left a comment

Choose a reason for hiding this comment

djhoese commented Mar 11, 2019

mraspaud commented Mar 11, 2019

mraspaud commented Mar 11, 2019

coveralls commented Mar 11, 2019

coveralls commented Mar 11, 2019 • edited

codecov bot commented Mar 11, 2019 • edited

Codecov Report

mraspaud commented Mar 13, 2019

djhoese left a comment

Choose a reason for hiding this comment

mraspaud commented Mar 14, 2019

djhoese left a comment

Choose a reason for hiding this comment

mraspaud commented Dec 7, 2018 •

edited

coveralls commented Mar 11, 2019 •

edited

codecov bot commented Mar 11, 2019 •

edited