Improve MiRS reader handling of missing metadata #1671
Conversation
joleenf
commented
May 13, 2021
joleenf
commented
- adds metadata to the yaml file for files which do not have internal metadata
- combines metadata that may be in file to yaml attributes if they are different but file takes precedence. This is meant to address the loss of the units from the final metadata in the previous version
- applies valid_range, fill_data and scale before limb correction
- all mirs data files will get more information about lat/lon coordinates from the yaml
- adjusts "Kelvin" to "K" for consistency within the project
- adds tests for units, and fill_value
- add coordinates to fake test data set to match how actual files look to open_dataset.
Merge branch 'master' of https://github.com/pytroll/satpy
Remove platform name from parameterize in last test for kwargs because they are not used in that test. Add a test for the NOAA-20 platform when the limb_correction assertions are tested and platform name is tested.
Merge remote-tracking branch 'upstream/master'
Fixes a bug in which valid range was ignored.
Change get_valid_range=>apply_valid_range and check for case of valid_range not None
only read fake_data one time, not every time check_valid_range is called.
…including _FillValue scale. - Fill value needs to be scaled in these files. - Try to combine yaml and file attributes with file attributes taking precedence over yaml. - Make sure units is saved in data array attributes
Fix name of units
- Need to make sure that yaml metadata when present is used and supplements attributes from data_arr. - Apply scaling to valid_range and fill_value - move location of apply_attributes - change name of get_metadata
- apply the attributes before the limb correction so that scaling, valid_range, fill are applied correctly - the yaml had some copy and pasted errors that would make it useless and unreadable - remove BT from yaml and handle attributes in reader - use the yaml for lat/lon attributes from every file
…eded for mirs_gpm Added because I did not consider that the limb correction is applied based on sensor which is assigned by platform. The mirs_gpm addition was not needed so I removed it.
satpy/readers/mirs.py
Outdated
| @@ -433,6 +465,7 @@ def _available_btemp_datasets(self): | |||
| 'name': new_name, | |||
| 'description': desc_bt, | |||
| 'units': 'K', | |||
| 'scale_factor': self.nc['BT'].attrs['scale_factor'], | |||
There was a problem hiding this comment.
Why only scale_factor and not add_offset?
satpy/etc/readers/mirs.yaml
Outdated
| scale_factor: 0.1 | ||
| _FillValue: -99.9 |
There was a problem hiding this comment.
Both scale_factor and _FillValue aren't provided in the files?
djhoese
changed the title
djhoese
changed the title
Codecov Report
@@ Coverage Diff @@
## main #1671 +/- ##
==========================================
- Coverage 92.66% 92.32% -0.34%
==========================================
Files 258 258
Lines 37988 38123 +135
==========================================
- Hits 35200 35196 -4
- Misses 2788 2927 +139
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
…gh to btemp dataset information
If the information is in the metadata of the file, use that, otherwise define in the yaml. Fix getting keys from ds_info in apply_attributes function so it does not needlessly break data that has no units.
…cients is consistent Old coefficients have a blank line at end, new coefficients do not. Previously, n_chn and n_fov were fixed in the coefficient reading. So, return to that so it is easier to swap new and old coefficients if necessary for testing.
|
👉 View analysis in DeepCode’s Dashboard | Configure the bot |
|
FYI, I squashed all your commits into a single commit and merged it. Looking at your fork, I think your main has commits that it shouldn't so your pull requests are getting out of sync with what github expects for going in the |
|
What do you want me to do with my main branch? |
|
I should start with what I think I understand. I have too many commits that are more intermediate/confused and are not needed. This is causing problems with the main branch of this repo when trying to merge my fork into pytroll/satpy main. How do I know that my main branch is at the proper commit? What do you mean by git magic? |
|
Let's talk in the CSPP slack. |
