Fixes and features needed for SELENE Kaguya TC data import #63
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With a slight reordering of the built-in level 1 tiles so that we can recurse simply.
This will allow us to be a lot more efficient when doing chunked TOAST samplings.
This is for the Kaguya lunar dataset. We're a bit sloppy here by using 0 as the "mask" value, but it's a reasonable approach. This patch also includes a few fixups to handle the F64 mode in the same way as F32, where it wasn't added to all of the logic.
Fortunately, all we need to do is spin things by 180 degrees in longitude.
This can be used with a chunked image to do a TOAST tiling from something that's too large to fit in memory.
I need to propagate this fix to other multiprocessing tasks too.
For some reason I can't get a reference to `toasty.image.SUPPORTED_FORMATS` to work as a Sphinx `:data:...` reference?
... which actually introduces some dead code here. Oh well.
This turned out to be a bit of a silly idea, since the main thread can just do the dispatch work. Also make sure that we handle the done_event without race conditions.
Codecov Report
@@ Coverage Diff @@
## master #63 +/- ##
==========================================
+ Coverage 73.11% 74.77% +1.65%
==========================================
Files 21 22 +1
Lines 2738 3037 +299
==========================================
+ Hits 2002 2271 +269
- Misses 736 766 +30
Continue to review full report at Codecov.
|
We weren't correctly updating Images when they were originally read in by PIL, because we'd write out the old PIL object and not the updated array. I thought that the fact that we would get a read-only Numpy array would prevent such issues, but apparently not? We may need a more generic API to ensure that such bugs don't reoccur, but there are some specific cases where it will always be correct to clear the PIL data.
My initial code was hardcoded for the U8 image format, but we should be better than that. Unfortunately I haven't tested whether it still works for U8 images. This is a signpost that the TOAST sampling code should be migrated to use an Image buffer in general, but I won't do that here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These are a bunch of improvements needed to handle the Kaguya dataset.
U8image modeglymur)toasty transform u8-to-rgb