Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add satpy cf-reader and eumetsat gac reader #1394

Merged

Conversation

ninahakansson
Copy link
Contributor

@ninahakansson ninahakansson commented Oct 8, 2020

The PR adds a reader for the EUMETSAT AVHRR GAC L1C FDR (yaml) with uses the satpy cf-reader (py).
It is a continuation of PR #1205 which included a reader for data written with
the satpy cf-writer.

Closes #1205

  • Tests added
  • Passes flake8 satpy
  • Fully documented

mraspaud and others added 15 commits May 13, 2020 22:33
Signed-off-by: Martin Raspaud <martin.raspaud@smhi.se>
For files from pygac wavelenghts are a list with four items with the
last one 'µm' which makes all elements strings.
For example if they where created by the area2lonlat in cf_writer
# Conflicts:
#	satpy/readers/yaml_reader.py
Do not assume resolution to be 742.
This is already done in available_datasets()
Note that latitude can't be loaded as dataset (yet?).
@codecov
Copy link

codecov bot commented Oct 15, 2020

Codecov Report

Merging #1394 (6216f4a) into master (2a1aa95) will increase coverage by 0.03%.
The diff coverage is 99.15%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1394      +/-   ##
==========================================
+ Coverage   90.60%   90.64%   +0.03%     
==========================================
  Files         236      238       +2     
  Lines       33814    33932     +118     
==========================================
+ Hits        30638    30756     +118     
  Misses       3176     3176              
Impacted Files Coverage Δ
satpy/readers/yaml_reader.py 95.21% <ø> (+0.13%) ⬆️
satpy/readers/satpy_cf_nc.py 98.55% <98.55%> (ø)
satpy/tests/reader_tests/test_satpy_cf_nc.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a1aa95...6216f4a. Read the comment docs.

@coveralls
Copy link

coveralls commented Oct 15, 2020

Coverage Status

Coverage increased (+0.03%) to 90.637% when pulling 6216f4a on ninahakansson:feature-satpy-cf-reader-eumetsat-gac into 2a1aa95 on pytroll:master.

@djhoese
Copy link
Member

djhoese commented Oct 15, 2020

I know this is still a draft, but I'm curious, could this be made a generic "cf" reader? It could include extra code where it looks for special attributes that the satpy cf writer creates, but otherwise it should probably be fine with a "regular" CF standard NetCDF file, right?

@mraspaud
Copy link
Member

We discussed this and decided that the first step was to read the pygac fdr datasets, as its what @ninahakansson needs. After that we make it generic, but by then we will have a set of test that ensures we don't break the fdr reading.

@ghost
Copy link

ghost commented Oct 21, 2020

Congratulations 🎉. DeepCode analyzed your code in 4.608 seconds and we found no issues. Enjoy a moment of no bugs ☀️.

👉 View analysis in DeepCode’s Dashboard | Configure the bot

@ninahakansson ninahakansson marked this pull request as ready for review October 26, 2020 09:20
Nina.Hakansson added 6 commits October 26, 2020 10:22
Trying to fix problem "Assignment Branch Condition too high",
by creating a function for handeleing the modifier attribute.
@pnuu
Copy link
Member

pnuu commented Nov 4, 2020

The Travis job fails because of a error in Windows build: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'testingcfwriter2020309101408-viirs-mband-20201007075915-20201007080744.nc'. Maybe there needs to be a locking mechanism used?

@mraspaud
Copy link
Member

mraspaud commented Nov 4, 2020

That usually happens because of a namedtemporaryfile in the tests. I think we have a workaround for this in some of the reader tests.

@djhoese
Copy link
Member

djhoese commented Nov 4, 2020

I think we normally just skip the test on Windows...sadly.

@mraspaud mraspaud added component:readers enhancement code enhancements, features, improvements labels Nov 9, 2020
self.assertTrue(np.all(scn_['image0'].data == self.scene['image0'].data))
self.assertTrue(np.all(scn_['lat'].data == self.scene['lat'].data)) # lat loaded as dataset
self.assertTrue(np.all(scn_['image0'].coords['lon'] == self.scene['lon'].data)) # lon loded as coord
os.remove(filename)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generating a Permission error on windows. I suspect the file isn't closed before removal, either because of the writting or the reading, or both... Could you try except the remove for now?

@mraspaud mraspaud merged commit 76e7d18 into pytroll:master Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants