Support multiple readers in group_files and MultiScene.from_files #1269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added a test for using multiple readers in a MultiScene. Note that the corresponding functionality is not yet implemented and actually not even yet properly defined (see pytroll#1268). To implement this in `MultiScene.from_files` requires updating `satpy.readers.group_files`. This commit does change the implementation of the keyword-only argument in `satpy.readers.group_files` by using Pythons own keyword-only argument functionality (available since Python 3.0). This has the effect that failing to pass the `reader` to that function now raises a TypeError rather than previously a ValueError.
Refactor readers.group_files as a preparation for allowing multiple readers. Some functionality is now relegated to dedicated functions. Single-reader now works again (at least tests pass!), multiple readers are raising a proper NotImplementedError rather than silently doing weird stuff; but in upcoming commits I intend to make multiple readers work, matching non-primary sensors to the primary using the provided tolerance, dropping the other non-primary sensors with a warning or error (configurable).
Add support for multiple readers in group_files. It appears to work, except that I have broken some other unit tests that I will investigate in the next commits.
|
👉 View analysis in DeepCode’s Dashboard | Configure the bot |
Fix the usage of custom grouping keys.
For the new multi-sensor behaviour in group_files, improve test coverage by adding more cases. Also point out that when passing multiple readers, it is strongly recommended to pass group_keys.
gerritholl
changed the title
gerritholl
changed the title
In MultiScene.from_files, for the use case when passing multiple readers, add a flag to return only those groups where all readers have at least one file found. The default remains that it returns those groups where at least one reader has at least one file found.
When no reader is passed to group_files and some readers can't be imported, skip those while logging the exception communicating the unimportability.
Under protest, but aligning strings to the beginning of the first line's string prefix rather than with the opening quote thereof.
|
(and again, codebeat thinks I've been deleting several files) |
Add an example to the MultiScene documentation on how to combine multiple readers into a single MultiScene.
There was a missing continue statement to accompany the message that there was a YAMLConstructorError.
Codecov Report
@@ Coverage Diff @@
## master #1269 +/- ##
==========================================
+ Coverage 90.06% 90.07% +0.01%
==========================================
Files 218 218
Lines 31530 31580 +50
==========================================
+ Hits 28397 28447 +50
Misses 3133 3133
Continue to review full report at Codecov.
|
Documentation clarification on the MultiScene.from_files construction. It was of course already possible to have multiple readers in a MultiScene, just it couldn't be automatically constructed with from_files.
In the new part of the MultiScene documentation, fix sphinx syntax as well as a typo in a method name.
In the new MultiScene documentation, fix the cross-reference type to the satpy.Scene class and clarify a bit more on the example with the superimposed lightning.
mraspaud
approved these changes
Jul 23, 2020
djhoese
requested changes
Jul 28, 2020
For the multiscene-multifile routine, improve the documentation, change a variable name, and raise a warning if no matching keys are found for grouping, which may happen if the user passes a string rather than a collection of strings.
Added a real non-breaking space in the file source via the Github editor, avoiding the use of a sphinx declaration.
djhoese
approved these changes
Jul 29, 2020
Fixed a merge conflict in satpy/tests/test_multiscene.py.
…into multiscene-multireader
djhoese
added
component:multiscene
component:readers
enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support multiple readers in group_files and
MultiScene.from_filesAdapt
group_filesso thatgroup_filesandMultiScene.from_filescan support multiple readers.flake8 satpy