Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'crop' and 'save_datasets' to MultiScene #613

Merged
merged 14 commits into from
Feb 13, 2019

Conversation

djhoese
Copy link
Member

@djhoese djhoese commented Feb 11, 2019

This PR adds the 'crop' and 'save_datasets' method to the MultiScene object. It also cleans up some of the dask distributed usage from a previous PR. This adds the functionality to use dask distributed in save_datasets for the MultiScene as well which can improve processing time.

The biggest possible cause of issues from this PR is the work I had to do is to fix the serialization of multiple readers and other parts of satpy. Dask distributed does not like map_blocks or delayed functions to be internal functions because they can't be serialized. This should be fixed in all of satpy now.

I still need to write tests.

  • Tests added
  • Tests passed
  • Passes git diff origin/master -- "*py" | flake8 --diff

@djhoese djhoese added the enhancement code enhancements, features, improvements label Feb 11, 2019
@djhoese djhoese added this to the v0.12 milestone Feb 11, 2019
@djhoese djhoese self-assigned this Feb 11, 2019
@codecov
Copy link

codecov bot commented Feb 11, 2019

Codecov Report

Merging #613 into master will increase coverage by 0.19%.
The diff coverage is 81.28%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #613      +/-   ##
==========================================
+ Coverage   78.13%   78.33%   +0.19%     
==========================================
  Files         136      136              
  Lines       19615    19714      +99     
==========================================
+ Hits        15327    15442     +115     
+ Misses       4288     4272      -16
Impacted Files Coverage Δ
satpy/readers/goes_imager_hrit.py 51.65% <0%> (ø) ⬆️
satpy/readers/msi_safe.py 2.52% <0%> (ø) ⬆️
satpy/readers/hrit_jma.py 98.43% <100%> (ø) ⬆️
satpy/tests/test_multiscene.py 97.47% <100%> (+0.98%) ⬆️
satpy/readers/electrol_hrit.py 34.4% <33.33%> (+0.25%) ⬆️
satpy/readers/olci_nc.py 43.42% <33.33%> (+0.57%) ⬆️
satpy/resample.py 88.95% <33.33%> (ø) ⬆️
satpy/readers/slstr_l1b.py 60.99% <40%> (+0.99%) ⬆️
satpy/readers/sar_c_safe.py 20.65% <50%> (+0.54%) ⬆️
satpy/composites/crefl_utils.py 84.52% <50%> (+1.19%) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71d215c...68b8f70. Read the comment docs.

@coveralls
Copy link

coveralls commented Feb 11, 2019

Coverage Status

Coverage increased (+0.2%) to 78.33% when pulling 68b8f70 on djhoese:feature-mscn-crop-save-datasets into 71d215c on pytroll:master.

satpy/tests/test_multiscene.py Outdated Show resolved Hide resolved
satpy/tests/test_multiscene.py Outdated Show resolved Hide resolved
@djhoese djhoese merged commit 57c7d21 into pytroll:master Feb 13, 2019
@djhoese djhoese deleted the feature-mscn-crop-save-datasets branch February 13, 2019 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants