Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for MERSI-1 on FY-3A/B/C #2795

Merged
merged 44 commits into from
Jun 4, 2024

Conversation

yukaribbba
Copy link
Contributor

@yukaribbba yukaribbba commented Apr 28, 2024

  • Tests added
  • Fully documented
  • RSR for pyspectral

Datasets for test:
https://drive.google.com/file/d/1cCoC14AioC2l34jFfxa19F88k5NXuqpi/view?usp=sharing

  1. FY-3A:
    image

  2. FY_3B:
    image

  3. FY_3C:
    image

@yukaribbba
Copy link
Contributor Author

pre-commit.ci autofix

@yukaribbba
Copy link
Contributor Author

@djhoese One of the CI test keeps failing on macos-latest, 3.9, false. I don't know what's going on.

=========================== short test summary info ============================
FAILED satpy/tests/reader_tests/test_sar_c_safe.py::TestSAFEGRD::test_read_lon_lats - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 3 / 256 (1.17%)
Max absolute difference: 4.98866104e-09
Max relative difference: 2.33333333
 x: array([[ 2.081090e-16,  5.91[666](https://github.com/pytroll/satpy/actions/runs/8872941895/job/24358006466?pr=2795#step:9:667)7e-01,  9.097222e-01,  1.000000e+00,
         9.083333e-01,  6.805556e-01,  3.625000e-01, -1.110223e-16,
        -3.611111e-01, -6.750000e-01, -8.958333e-01, -9.777778e-01,...
 y: array([[ 3.794929e-16,  5.916[667](https://github.com/pytroll/satpy/actions/runs/8872941895/job/24358006466?pr=2795#step:9:668)e-01,  9.097222e-01,  1.000000e+00,
         9.083333e-01,  6.805556e-01,  3.625000e-01,  8.326673e-17,
        -3.611111e-01, -6.750000e-01, -8.958333e-01, -9.777778e-01,...
= 1 failed, 2541 passed, 4 skipped, 5 xfailed, 1 xpassed, 443 warnings in 352.85s (0:05:52) =

@djhoese
Copy link
Member

djhoese commented Apr 29, 2024

Yes. This is a known problem with a scipy update causing a test failure. It is being looked at.

Copy link

codecov bot commented Apr 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.94%. Comparing base (3b9c04e) to head (f48d740).
Report is 53 commits behind head on main.

Current head f48d740 differs from pull request most recent head d04bd36

Please upload reports for the commit d04bd36 to get more accurate results.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2795      +/-   ##
==========================================
- Coverage   95.95%   95.94%   -0.01%     
==========================================
  Files         379      379              
  Lines       53888    53764     -124     
==========================================
- Hits        51708    51585     -123     
+ Misses       2180     2179       -1     
Flag Coverage Δ
behaviourtests 4.10% <0.00%> (+<0.01%) ⬆️
unittests 96.04% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@djhoese djhoese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is still a draft, but I'm curious if there are actual differences between the datasets for the 3 satellite instruments. If they're the same channels then maybe the 3 readers could be combined into a single "mersi_l1b" reader?

@yukaribbba
Copy link
Contributor Author

Well it's kinda annoying here that there are no genuine differences inside the datasets but they are still organized in different forms. So I think merging them into one could be a little bit messy for the users...

@djhoese
Copy link
Member

djhoese commented Apr 29, 2024

Yeah MERSI and AGRI were the two instruments I think of with designs like that (new version of the instrument, but completely different numbering/ordering of channels).

@yukaribbba
Copy link
Contributor Author

pre-commit.ci autofix

@yukaribbba
Copy link
Contributor Author

pre-commit.ci autofix

@yukaribbba
Copy link
Contributor Author

@djhoese Some of the CI failed again. But at least the tests passed.

@yukaribbba yukaribbba marked this pull request as ready for review May 1, 2024 15:17
@yukaribbba yukaribbba requested a review from mraspaud as a code owner May 1, 2024 15:17
@yukaribbba yukaribbba requested a review from djhoese May 1, 2024 15:18
@djhoese
Copy link
Member

djhoese commented May 2, 2024

I restarted the CI. Very odd error from the test coverage reporting tool so let's see if it fixes itself...

@yukaribbba
Copy link
Contributor Author

Failed again....

@djhoese
Copy link
Member

djhoese commented May 2, 2024

Looks like a major release of the coveralls python package causing issues: https://pypi.org/project/coveralls/

I have appointments half of today, but I'll try debugging later.

@coveralls
Copy link

coveralls commented May 10, 2024

Pull Request Test Coverage Report for Build 9369822041

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 258 of 258 (100.0%) changed or added relevant lines in 2 files are covered.
  • 570 unchanged lines in 59 files lost coverage.
  • Overall coverage decreased (-0.009%) to 96.023%

Files with Coverage Reduction New Missed Lines %
satpy/readers/glm_l2.py 1 98.55%
satpy/readers/ascat_l2_soilmoisture_bufr.py 1 97.44%
satpy/tests/multiscene_tests/test_save_animation.py 1 99.1%
satpy/tests/reader_tests/_li_test_utils.py 1 99.26%
satpy/dataset/metadata.py 1 99.26%
satpy/tests/reader_tests/test_li_l2_nc.py 1 99.76%
satpy/tests/reader_tests/test_ascat_l2_soilmoisture_bufr.py 1 97.59%
satpy/readers/mws_l1b.py 1 98.52%
satpy/tests/reader_tests/test_avhrr_l1b_gaclac.py 1 99.66%
satpy/tests/reader_tests/test_ahi_hsd.py 1 99.67%
Totals Coverage Status
Change from base Build 9157044280: -0.009%
Covered Lines: 51524
Relevant Lines: 53658

💛 - Coveralls

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks a lot for refactoring the tests! Just one comment inline, but after that I can merge

satpy/tests/reader_tests/test_mersi_l1b.py Outdated Show resolved Hide resolved
@mraspaud mraspaud added the enhancement code enhancements, features, improvements label Jun 4, 2024
@mraspaud mraspaud added this to the v0.49.0 milestone Jun 4, 2024
@yukaribbba yukaribbba requested a review from mraspaud June 4, 2024 15:31
Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mraspaud mraspaud merged commit 7a7740a into pytroll:main Jun 4, 2024
15 of 17 checks passed
@yukaribbba yukaribbba deleted the add_support_mersi1 branch June 5, 2024 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants