Mimic TPW Reader #858

joleenf · 2019-07-16T17:26:27Z

Mimic TPW reader for files from SSEC, reads both the coarse and fine scale equirectangular grids produced. Addresses #807

Attempts to close Add a MIMIC-TPW2 reader #807
Passes flake8 satpy No - Weird error that I cannot figure out... "flake8-docstrings" failed during execution due to "module 'pydocstyle' has no attribute 'tokenize_open'" Run flake8 with greater verbosity to see more details.

I ran flake8 with doctstrings extension and could not reproduce outside of pre-commit environment.

not good files to use as an example.

…t. Add tpwGrid to yaml because the automatic detection is not adding the tpwGrid variable to the handled_variables outside of debugging mode in pycharm. This is still not a model of code to follow.

"Fix" the area definition, because the data is in degrees and needs longlat projection from Proj4 but I am not sure if that was the only thing wrong.

…n the mimic yaml

…oarse and fine mimic files Since the mimic files for the coarse and fine resolution are named exactly the same, errors and failure result when data variables are described in yaml. Follow TommyJ approach to extracting any data variable that matches navigation shape. This then smoothly reads all the variables in the coarse resolution files, while still being able to read the single tpwGrid variable in the fine resolution files. The coordinates of 'latitude', 'longitude' were returing datasetID errors and were removed from the new_info update for the tpw variables

…in commit 91f1131

…olor change

…able

Fix informational text for reader

…olor scale

coveralls · 2019-07-16T17:45:14Z

Coverage decreased (-0.5%) to 83.561% when pulling 17473ab on joleenf:mimic into cc6ba42 on pytroll:master.

coveralls · 2019-07-16T17:45:14Z

Coverage increased (+0.004%) to 87.387% when pulling 4fd2c32 on joleenf:mimic into 73e081f on pytroll:master.

codecov · 2019-07-16T17:48:36Z

Codecov Report

Merging #858 into master will increase coverage by 0.02%.
The diff coverage is 88%.

@@            Coverage Diff             @@
##           master     #858      +/-   ##
==========================================
+ Coverage   87.36%   87.39%   +0.02%     
==========================================
  Files         185      188       +3     
  Lines       28446    28621     +175     
==========================================
+ Hits        24853    25012     +159     
- Misses       3593     3609      +16

Impacted Files	Coverage Δ
satpy/enhancements/mimic.py	`0% <0%> (ø)`
satpy/tests/reader_tests/__init__.py	`98.41% <100%> (+0.02%)`	⬆️
satpy/readers/mimic_TPW2_nc.py	`90% <90%> (ø)`
satpy/tests/reader_tests/test_mimic_TPW2_nc.py	`92.04% <92.04%> (ø)`
satpy/scene.py	`90.45% <0%> (+0.17%)`	⬆️
satpy/readers/__init__.py	`94.56% <0%> (+1.2%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73e081f...4fd2c32. Read the comment docs.

joleenf · 2019-07-24T16:28:58Z

I cannot resample using the built-in scn.resample() method. I think there must be something wrong with the way the data is being loaded. I need to look into this.

joleenf · 2019-07-24T17:42:28Z

Re-open and just try to fix the problem with the pull request open.

joleenf · 2019-07-31T16:18:19Z

That is an interesting question, it is an addition I made before finding out that the resampling issue was due to the area_of_influence setting in resample. I looked more closely at why I thought it was a keyword. It can be included in certain AreaDefinitions from certain methods (I think) but may not be appropriate here. When I first wrote the code in that section, I was thinking that the grid spacing of the longlat proj would be inferred... Here is the documentation that included resolution, but looking at it closer, it might be the wrong doc again: https://pyresample.readthedocs.io/en/stable/geometry_utils.html#areadefinition-creation In particular, it does not accept a tuple resolution as documented.

djhoese · 2019-07-31T16:20:21Z

That is for the create_area_def function, you are calling AreaDefinition directly, but more importantly you added the resolution to the proj_dict and not the AreaDefinition keyword arguments.

…word

djhoese · 2019-09-27T14:12:32Z

@joleenf Could you merge this with master again to see if the tests pass?

joleenf · 2019-10-01T22:11:12Z

@djhoese How would I merge this again? I have a message that "Merging is blocked." I am not authorized.

mraspaud · 2019-10-24T09:00:19Z

How does it look now ? Are we ready to merge @joleenf @djhoese ?

mraspaud

LGTM, thanks for adding this new reader for precip products !
Could you just add the test suite to the master suite for readers in satpy/tests/reader_tests/__init__.py ?

This reverts commit 999a347.

… tests to test suite" This reverts commit b74bcf4.

djhoese · 2019-10-25T16:24:13Z

Tests failing with:

======================================================================

ERROR: test_load_mimic (satpy.tests.reader_tests.test_mimic_TPW2_nc.TestMimicTPW2Reader)

Load Mimic data

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/pytroll/satpy/satpy/tests/reader_tests/test_mimic_TPW2_nc.py", line 137, in test_load_mimic

    ds = r.load(['tpwGrid'])

  File "/home/travis/build/pytroll/satpy/satpy/readers/yaml_reader.py", line 843, in load

    ds = self._load_dataset_with_area(dsid, coords)

  File "/home/travis/build/pytroll/satpy/satpy/readers/yaml_reader.py", line 749, in _load_dataset_with_area

    area = self._load_dataset_area(dsid, file_handlers, coords)

  File "/home/travis/build/pytroll/satpy/satpy/readers/yaml_reader.py", line 731, in _load_dataset_area

    return self._load_area_def(dsid, file_handlers)

  File "/home/travis/build/pytroll/satpy/satpy/readers/yaml_reader.py", line 655, in _load_area_def

    area_defs = [fh.get_area_def(dsid) for fh in file_handlers]

  File "/home/travis/build/pytroll/satpy/satpy/readers/yaml_reader.py", line 655, in <listcomp>

    area_defs = [fh.get_area_def(dsid) for fh in file_handlers]

  File "/home/travis/build/pytroll/satpy/satpy/readers/mimic_TPW2_nc.py", line 137, in get_area_def

    area_def = AreaDefinition(area_id, description, proj_id, proj_dict, width, height, area_extent, )

  File "/home/travis/miniconda/envs/test/lib/python3.7/site-packages/pyresample/geometry.py", line 1042, in __init__

    self.crs = CRS(projection)

  File "/home/travis/miniconda/envs/test/lib/python3.7/site-packages/pyproj/crs.py", line 303, in __init__

    super(CRS, self).__init__(projstring)

  File "pyproj/_crs.pyx", line 1335, in pyproj._crs._CRS.__init__

pyproj.exceptions.CRSError: Invalid projection: +proj=longlat +datum=WGS84 +ellps=WGS84 +units=degrees +type=crs: (Internal Proj Error: proj_create: Error -7: unknown unit conversion id)

I don't think +units=degrees is a thing in all versions of PROJ.

…ot always valid.

…2_nc in import block. Merge branch 'master' of https://github.com/pytroll/satpy into mimic

…module added Merge branch 'master' of https://github.com/pytroll/satpy into mimic

joleenf added 22 commits June 12, 2019 14:42

First attempt to create a reader and yaml for MIMIC TPW files. These are

b5903f7

not good files to use as an example.

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

ec5043b

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

4b06cbc

BUG FIX: add get_dataset and get_area_def missing from original commi…

41a3378

…t. Add tpwGrid to yaml because the automatic detection is not adding the tpwGrid variable to the handled_variables outside of debugging mode in pycharm. This is still not a model of code to follow.

Remove redundant code. rename the dims to x and y (based on docs)

9a7d52b

"Fix" the area definition, because the data is in degrees and needs longlat projection from Proj4 but I am not sure if that was the only thing wrong.

Fix names to match code

12a23ea

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

71589f0

Add a function to colorize the mimic data with the colors described i…

91f1131

…n the mimic yaml

Oversight correction: add mimic.yaml that matches enhancement reader …

96e8834

…in commit 91f1131

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

f13337e

Reduce the number of colors in the mimic yaml to new colors only

cd6d2ee

Reduce colors even more to about increments of 0.5 mm at most rapid c…

4270bc7

…olor change

Try to get ideal combination of color definition and speed for colort…

d7c3359

…able

Remove comment around add resolution for lat/lon arrays

a9f7bfe

Fix informational text for reader

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

f3b0ddd

Use an enhancement provided by Tony Wimmers for Mimic TPW

237c407

Fix type of name key to string

2a5d41b

readers/mimic_TPW2_nc.py

291c0de

Test reader code for the MIMIC TPW reader

e659a51

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

4ba6b22

Add enhancement for tpwGridPrior so that tpwGrid and Prior are same c…

17473ab

…olor scale

joleenf requested review from djhoese and mraspaud as code owners July 16, 2019 17:26

joleenf added 2 commits July 16, 2019 13:01

remove newline at end of file

b1a7d27

fix description of projection

8002bed

joleenf reopened this Jul 24, 2019

Add information to the area definition.

d2d4620

BUG FIX: remove resolution from AreaDefinition, it is not a valid key…

bfd0dbc

…word

joleenf added 2 commits October 3, 2019 05:40

Fixed lint error of missing space after comma

695191d

missed another space after comma...

999a347

mraspaud assigned joleenf Oct 24, 2019

mraspaud added component:readers enhancement code enhancements, features, improvements labels Oct 24, 2019

mraspaud requested changes Oct 24, 2019

View reviewed changes

joleenf added 6 commits October 25, 2019 09:47

added mimic tests to test suite

b74bcf4

BUG FIX: forgot to add import for mimic tests

5968fb5

Revert "missed another space after comma..."

5e6ca31

This reverts commit 999a347.

Go back before adding import and merge upstream...Revert "added mimic…

7380a73

… tests to test suite" This reverts commit b74bcf4.

Merge branch 'master' of https://github.com/pytroll/satpy into mimic

dc17b7e

fix missing whitespace error

dbbdbc9

joleenf and others added 5 commits October 28, 2019 09:33

BUG FIX: remove units=degrees from proj definition; unit=degrees is n…

4bd04a3

…ot always valid.

Resolved conflicts by keeping test_viirs_compact after test_mimic_TPW…

d396dca

…2_nc in import block. Merge branch 'master' of https://github.com/pytroll/satpy into mimic

removed trailing whitespace

49cc14d

Resolved conflicts in __init__.py with new modules added then, mimic …

874fde3

…module added Merge branch 'master' of https://github.com/pytroll/satpy into mimic

Merge branch 'master' into mimic

4fd2c32

mraspaud approved these changes Dec 23, 2019

View reviewed changes

mraspaud merged commit 5ce2122 into pytroll:master Dec 26, 2019

joleenf deleted the mimic branch May 19, 2021 12:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mimic TPW Reader #858

Mimic TPW Reader #858

joleenf commented Jul 16, 2019

coveralls commented Jul 16, 2019

coveralls commented Jul 16, 2019 •

edited

codecov bot commented Jul 16, 2019 •

edited

joleenf commented Jul 24, 2019

joleenf commented Jul 24, 2019

joleenf commented Jul 31, 2019

djhoese commented Jul 31, 2019

djhoese commented Sep 27, 2019

joleenf commented Oct 1, 2019

mraspaud commented Oct 24, 2019

mraspaud left a comment

djhoese commented Oct 25, 2019

Mimic TPW Reader #858

Mimic TPW Reader #858

Conversation

joleenf commented Jul 16, 2019

coveralls commented Jul 16, 2019

coveralls commented Jul 16, 2019 • edited

codecov bot commented Jul 16, 2019 • edited

Codecov Report

joleenf commented Jul 24, 2019

joleenf commented Jul 24, 2019

joleenf commented Jul 31, 2019

djhoese commented Jul 31, 2019

djhoese commented Sep 27, 2019

joleenf commented Oct 1, 2019

mraspaud commented Oct 24, 2019

mraspaud left a comment

Choose a reason for hiding this comment

djhoese commented Oct 25, 2019

coveralls commented Jul 16, 2019 •

edited

codecov bot commented Jul 16, 2019 •

edited