Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix HDF5 utility file handler not decoding byte arrays consistently #1422

Merged
merged 5 commits into from Nov 16, 2020

Conversation

djhoese
Copy link
Member

@djhoese djhoese commented Nov 4, 2020

A couple users noticed that some readers are now causing errors when multiple granule's metadata dictionaries are combined. You'll see errors like:

  File "/home/cpeters/anaconda2/envs/satpy/lib/python3.7/site-packages/numpy/core/numeric.py", line 2287, in isclose
    xfin = isfinite(x)
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

I've tracked this down to at least one specific issue: some readers are putting byte arrays (ex. b''"') in numpy arrays in their DataArray.attrs metadata. When attempting to combine metadata satpy sees these as arrays and attempts to combine them. This fails with the above error.

Both readers that have encountered this are using the base HDF5 utility file handler. I discovered that although the file handler is converting attributes to regular strings when possible, it is only doing this for attributes accessed through the string form my_handler['/my_data/attr/some_attr'] but NOT for the DataArray that is produced so doing my_handler['my_data'].attrs['some_attr] will have the unconverted data.

Additionally this fixes the unstable travis environment failures in the HY2 reader.

  • Closes #xxxx
  • Tests added
  • Passes flake8 satpy

@ghost
Copy link

ghost commented Nov 4, 2020

Congratulations 🎉. DeepCode analyzed your code in 2.871 seconds and we found no issues. Enjoy a moment of no bugs ☀️.

👉 View analysis in DeepCode’s Dashboard | Configure the bot

@codecov
Copy link

codecov bot commented Nov 4, 2020

Codecov Report

Merging #1422 into master will increase coverage by 0.00%.
The diff coverage is 92.85%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1422   +/-   ##
=======================================
  Coverage   90.58%   90.58%           
=======================================
  Files         236      236           
  Lines       33797    33820   +23     
=======================================
+ Hits        30615    30637   +22     
- Misses       3182     3183    +1     
Impacted Files Coverage Δ
satpy/readers/hdf5_utils.py 91.42% <77.77%> (-1.00%) ⬇️
satpy/readers/mersi2_l1b.py 97.70% <85.71%> (+1.18%) ⬆️
satpy/readers/hy2_scat_l2b_h5.py 98.00% <100.00%> (ø)
satpy/tests/reader_tests/test_hdf5_utils.py 94.59% <100.00%> (+1.04%) ⬆️
satpy/tests/reader_tests/test_mersi2_l1b.py 99.63% <100.00%> (ø)
satpy/readers/nwcsaf_msg2013_hdf5.py 94.87% <0.00%> (-1.29%) ⬇️
satpy/tests/reader_tests/test_nucaps.py 100.00% <0.00%> (ø)
satpy/readers/nucaps.py 89.18% <0.00%> (+0.04%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ccddbf...3765b95. Read the comment docs.

@coveralls
Copy link

coveralls commented Nov 4, 2020

Coverage Status

Coverage increased (+0.003%) to 90.588% when pulling 3765b95 on djhoese:bugfix-hy2-reader into 9ccddbf on pytroll:master.

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.03%) to 90.551% when pulling fa0f17b on djhoese:bugfix-hy2-reader into 9ccddbf on pytroll:master.

Copy link
Collaborator

@TAlonglong TAlonglong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this, LGTM

@mraspaud mraspaud merged commit 1922d97 into pytroll:master Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants