Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up cviirs tiepoint interpolation #941

Merged
merged 11 commits into from Oct 27, 2019

Conversation

mraspaud
Copy link
Member

@mraspaud mraspaud commented Oct 16, 2019

This PR by factorizes expand_array (renamed to expand_arrays) to handle similar arrays at the same time.

@mraspaud mraspaud added enhancement code enhancements, features, improvements component:readers labels Oct 16, 2019
@mraspaud mraspaud requested a review from pnuu October 16, 2019 12:48
@mraspaud mraspaud self-assigned this Oct 16, 2019
@coveralls
Copy link

coveralls commented Oct 16, 2019

Coverage Status

Coverage increased (+0.4%) to 86.614% when pulling a861637 on mraspaud:feature-faster-cviirs-dnb into 2442ed2 on pytroll:master.

@codecov
Copy link

codecov bot commented Oct 16, 2019

Codecov Report

Merging #941 into master will increase coverage by 0.54%.
The diff coverage is 67.3%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #941      +/-   ##
==========================================
+ Coverage   86.19%   86.73%   +0.54%     
==========================================
  Files         174      175       +1     
  Lines       26714    27178     +464     
==========================================
+ Hits        23026    23574     +548     
+ Misses       3688     3604      -84
Impacted Files Coverage Δ
satpy/tests/reader_tests/test_viirs_compact.py 94.33% <ø> (ø)
satpy/tests/reader_tests/__init__.py 98.24% <100%> (+0.03%) ⬆️
satpy/readers/utils.py 75.23% <100%> (ø) ⬆️
satpy/readers/viirs_compact.py 64.56% <66.66%> (+49.52%) ⬆️
satpy/config.py 82.38% <0%> (+0.4%) ⬆️
satpy/tests/test_writers.py 98.33% <0%> (+0.52%) ⬆️
satpy/readers/file_handlers.py 97.01% <0%> (+2.98%) ⬆️
satpy/writers/__init__.py 88.28% <0%> (+3.07%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2442ed2...a861637. Read the comment docs.

@pnuu
Copy link
Member

pnuu commented Oct 16, 2019

Seems to make loading significantly faster. Timings for single DNB granule in Compact SDR format, master branch vs. this PR:

  • scn.load(['DNB']): 7.1 s -> 4.5 s
  • scn.load(['hncc_dnb']): 22.6 s -> 14.6 s

@mraspaud
Copy link
Member Author

I think it could go even faster if the code between navigation and angles was also factorized

@pnuu
Copy link
Member

pnuu commented Oct 17, 2019

With the latest updates the performance is looking pretty spectacular vs. master branch. Timings for one DNB granule:

  • scn.load(['DNB']): 7.1 s -> 0.6 s
  • scn.load(['hncc_dnb']): 22.6 s -> 0.8 s

For hncc_dnb that's a speedup of 26-27x.

@mraspaud
Copy link
Member Author

Just to clarify, it is just creating the datasets that we discuss here, not any computation of the data

@mraspaud mraspaud marked this pull request as ready for review October 25, 2019 11:51
@mraspaud mraspaud merged commit 283ca90 into pytroll:master Oct 27, 2019
@mraspaud mraspaud deleted the feature-faster-cviirs-dnb branch October 27, 2019 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements help wanted
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Loading of DNB data from VIIRS compact SDR is slow
4 participants