Fix GPM IMERG reader. #2137

simonrp84 · 2022-07-06T11:30:28Z

The IMERG reader was not correctly dealing with coordinates, and was using numpy instead of dask. This PR fixes these problems and also updates the tests.

simonrp84 · 2022-07-06T11:31:24Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

mraspaud · 2022-07-06T11:35:48Z

Asked the user to check if this fixed there problem.

codecov · 2022-07-06T11:45:54Z

Codecov Report

Merging #2137 (345336d) into main (bafe54e) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #2137   +/-   ##
=======================================
  Coverage   93.88%   93.88%           
=======================================
  Files         283      283           
  Lines       43093    43100    +7     
=======================================
+ Hits        40458    40466    +8     
+ Misses       2635     2634    -1

Flag	Coverage Δ
behaviourtests	`4.78% <0.00%> (-0.01%)`	⬇️
unittests	`94.54% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
satpy/readers/gpm_imerg.py	`100.00% <100.00%> (+2.08%)`	⬆️
satpy/tests/reader_tests/test_gpm_imerg.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bafe54e...345336d. Read the comment docs.

coveralls · 2022-07-06T12:03:15Z

Coverage increased (+0.003%) to 94.483% when pulling 345336d on simonrp84:imerg_fix into bafe54e on pytroll:main.

djhoese · 2022-07-06T14:27:14Z

User reports:

Traceback (most recent call last):
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
File "stringsource", line 2, in h5py.h5r.Reference.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 153, in deepcopy
y = copier(memo)
SystemError: <built-in method __deepcopy__ of numpy.ndarray object at 0x7fa8c97529d0> returned a result with an exception set
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 137, in deepcopy
d = id(x)
SystemError: <built-in function id> returned a result with an exception set
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 137, in deepcopy
d = id(x)
SystemError: <built-in function id> returned a result with an exception set
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/snap/pycharm-community/286/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/snap/pycharm-community/286/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/media/nturini/501E62DF1E62BE1A/DARWIN/WP2/scripts/Modeldevelopment/Dataprocessing/IMERG.py", line 33, in <module>
variable_GP=scn.resample(myarea, resampler='nearest')
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/satpy/scene.py", line 951, in resample
self._resampled_scene(new_scn, destination, resampler=resampler,
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/satpy/scene.py", line 867, in _resampled_scene
res = resample_dataset(dataset, destination_area, **kwargs)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/satpy/resample.py", line 1418, in resample_dataset
new_data = resample(source_area, dataset, destination_area, fill_value=fill_value, **kwargs)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/satpy/resample.py", line 1381, in resample
res = resampler_instance.resample(data, **kwargs)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/satpy/resample.py", line 439, in resample
return self.compute(data, cache_id=cache_id, **kwargs)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/satpy/resample.py", line 627, in compute
res = self.resampler.get_sample_from_neighbour_info(data, fill_value)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/pyresample/kd_tree.py", line 1144, in get_sample_from_neighbour_info
attrs=deepcopy(data.attrs))
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/copy.py", line 153, in deepcopy
y = copier(memo)
SystemError: <built-in method __deepcopy__ of numpy.ndarray object at 0x7fa8c97528b0> returned a result with an exception set
/home/nturini/anaconda3/envs/Dataprocessing/lib/python3.10/site-packages/pyproj/crs/crs.py:1282: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
proj = self._crs.to_proj4(version=version)

My guess is the NetCDF4 dimension attributes in the .attrs are the problem. I hate that people call files like these .HDF5 when they are actually NetCDF4 files.

# Conflicts: # satpy/readers/gpm_imerg.py

satpy/tests/reader_tests/test_gpm_imerg.py

simonrp84 · 2022-07-06T14:52:45Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

simonrp84 · 2022-07-06T14:57:40Z

@djhoese Should be fixed now.

satpy/readers/gpm_imerg.py

simonrp84 · 2022-07-06T16:15:54Z

Not sure why Coveralls is saying that coverage is down, it's in something untouched by this PR.

djhoese · 2022-07-06T17:41:42Z

It looks like there are two coveralls results listed in the checks in this PR. One has negative change, one has positive change. I think the one showing decrease in coverage is a specific environment that changed. This is likely due (from what I can tell) to a new version of fsspec or something else that is no longer triggering an exception and running the couple lines of code that would be expected. Bottom line, I wouldn't worry about it.

simonrp84 · 2022-07-07T08:59:48Z

The user reports that this solves their problem, so I think we're good to go now.

mraspaud

LGTM

Fix GPM IMERG reader.

bd26450

simonrp84 requested review from djhoese and mraspaud as code owners July 6, 2022 11:30

[pre-commit.ci] auto fixes from pre-commit.com hooks

d0d1b90

for more information, see https://pre-commit.ci

simonrp84 added 2 commits July 6, 2022 15:49

Further fixes for GPM IMERG reader.

887174e

Merge remote-tracking branch 'origin/imerg_fix' into imerg_fix

fd7cfa6

# Conflicts: # satpy/readers/gpm_imerg.py

stickler-ci reviewed Jul 6, 2022

View reviewed changes

satpy/tests/reader_tests/test_gpm_imerg.py Outdated Show resolved Hide resolved

[pre-commit.ci] auto fixes from pre-commit.com hooks

9c8185c

for more information, see https://pre-commit.ci

djhoese reviewed Jul 6, 2022

View reviewed changes

satpy/readers/gpm_imerg.py Show resolved Hide resolved

simonrp84 added 2 commits July 6, 2022 16:39

Update GPM tests.

b5f9a1e

Merge remote-tracking branch 'origin/imerg_fix' into imerg_fix

345336d

mraspaud approved these changes Jul 7, 2022

View reviewed changes

mraspaud merged commit 229ed1b into pytroll:main Jul 7, 2022

mraspaud assigned simonrp84 Jul 7, 2022

mraspaud added bug component:readers labels Jul 7, 2022

simonrp84 deleted the imerg_fix branch July 7, 2022 10:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix GPM IMERG reader. #2137

Fix GPM IMERG reader. #2137

simonrp84 commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

mraspaud commented Jul 6, 2022

codecov bot commented Jul 6, 2022 •

edited

coveralls commented Jul 6, 2022 •

edited

djhoese commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

djhoese commented Jul 6, 2022

simonrp84 commented Jul 7, 2022

mraspaud left a comment

Fix GPM IMERG reader. #2137

Fix GPM IMERG reader. #2137

Conversation

simonrp84 commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

mraspaud commented Jul 6, 2022

codecov bot commented Jul 6, 2022 • edited

Codecov Report

coveralls commented Jul 6, 2022 • edited

djhoese commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

simonrp84 commented Jul 6, 2022

djhoese commented Jul 6, 2022

simonrp84 commented Jul 7, 2022

mraspaud left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 6, 2022 •

edited

coveralls commented Jul 6, 2022 •

edited