Masking by bit values #415

vpipkt · 2019-11-11T21:14:15Z

function reference docs for new functions
unit test for SQL snip in docs
expose extract and mask functions in python api
ensure python and scala api's are equivalent
consider extract bits input and output cell types, unit tests
update release notes
Fix mask by value function incorrectly interprets 0 #416
Reorganize masking stuff in RasterFucntionsSpec
update the masking page to show landsat mask using mask_by_bits

Signed-off-by: Jason T. Brown <jason@astraea.earth>

Mask by value of 0 fixed Masking cell type mutations in rf_mask_by_values is resolved Masking by extraction of bits implemented Signed-off-by: Jason T. Brown <jason@astraea.earth>

…ests Signed-off-by: Jason T. Brown <jason@astraea.earth>

Signed-off-by: Jason T. Brown <jason@astraea.earth>

courtney-layman · 2019-11-15T20:56:48Z

I read through the masking doc. I think it looks great!

metasim

There are a few minor change requests. But don't delay because of that; it's so amazing I want to get it merged ASAP.

core/src/main/scala/org/locationtech/rasterframes/expressions/package.scala

core/src/main/scala/org/locationtech/rasterframes/expressions/transformers/ExtractBits.scala

core/src/main/scala/org/locationtech/rasterframes/expressions/transformers/Mask.scala

metasim · 2019-11-18T14:16:18Z

docs/src/main/paradox/reference.md

+
+Applies a mask from blacklisted bit values in the `mask_tile`. Working from the right, the bits from `start_bit` to `start_bit + num_bits` are @ref:[extracted](reference.md#rf_local_extract_bits) from cell values of the `mask_tile`. In all locations where these are in the `mask_values`, the returned tile is set to NoData; otherwise the original `tile` cell value is returned.
+
+This function is not available in the SQL API. The below is equivalent:


You'll have to explain to me why... If it's array creation, it should be available with the array(x, y, z, ...) function.

Basically I threw in the towel on trying to create an Expression with 5 arguments (tile, tile, int, int, array) ... If you have any intiution on this it's welcome! Maybe we can file another issue to bring that along.

docs/src/main/paradox/reference.md

pyrasterframes/src/main/python/docs/masking.pymd

metasim · 2019-11-18T14:19:14Z

pyrasterframes/src/main/python/docs/masking.pymd

+l8_df.select('data', 'mask', 'data_masked', 'data2', rf_extent('data_masked')) \
+     .filter(rf_data_cells('data_masked') > 32000)
+```
+


Wow. That's all I can say. This is absolutely brilliant.

metasim · 2019-11-18T14:20:07Z

pyrasterframes/src/main/python/tests/PyRasterFramesTests.py

            long_trip = df.first()["tile"]
            self.assertEqual(long_trip, a_tile)

+    def test_masked_deser(self):


Not sure I understand what this is testing...

basically just testing that a Tile that gets deserialized has the expected masking on it.

Basically is the same thing as my commented out block at RasterFunctionsTest.py lines 282 on.... paranoia

metasim · 2019-11-18T14:22:25Z

pyrasterframes/src/main/python/tests/RasterFunctionsTests.py

+        # self.assertEqual(mask_fill_tile.cells.mask.sum(), 4,
+        #                  f'Expected {16 - 4} data values but got the masked tile:'
+        #                  f'{mask_fill_tile}'
+        #                  )


Are you expecting mask_fill_tile.cells.mask.sum() to return 42*4*4, or the cells.sum() after the mask has been applied?

Or something else?

so the mask is binary where 1 indicates that location is nodata.

So by doing .mask.sum() we expect the same as rf_no_data_cells. But i kept getting this assertion failing locally.

so the mask is binary where 1 indicates that location is nodata.

So by doing .mask.sum() we expect the same as rf_no_data_cells. But i kept getting this assertion failing locally.

Signed-off-by: Jason T. Brown <jason@astraea.earth>

…deserialization Signed-off-by: Jason T. Brown <jason@astraea.earth>

vpipkt added 3 commits November 6, 2019 13:41

Initial implementation of extracting bit values, e.g. for a quality band

a5c651c

Signed-off-by: Jason T. Brown <jason@astraea.earth>

WIP: mask bits

bebf00f

Signed-off-by: Jason T. Brown <jason@astraea.earth>

Merge remote-tracking branch 'lt/develop' into feature/mask_by_bits

1c1c632

vpipkt mentioned this pull request Nov 11, 2019

Extract and mask on bit values, e.g. for a quality band s22s/rasterframes#56

Closed

vpipkt added 6 commits November 11, 2019 16:57

Update docs and scala function api

b45543b

Signed-off-by: Jason T. Brown <jason@astraea.earth>

Add failing unit test for mask by value on 0

b2f8d3c

Signed-off-by: Jason T. Brown <jason@astraea.earth>

Masking improvements and unit tests.

f4f4e9a

Mask by value of 0 fixed Masking cell type mutations in rf_mask_by_values is resolved Masking by extraction of bits implemented Signed-off-by: Jason T. Brown <jason@astraea.earth>

Fix for both masking by def and value; expand code comments; update t…

abb7add

…ests Signed-off-by: Jason T. Brown <jason@astraea.earth>

Extract bits should throw on non-integral cell types

b67049a

Signed-off-by: Jason T. Brown <jason@astraea.earth>

Add mask bits python api and unit test

5529d48

Signed-off-by: Jason T. Brown <jason@astraea.earth>

vpipkt marked this pull request as ready for review November 15, 2019 18:30

vpipkt requested a review from metasim November 15, 2019 18:30

vpipkt added 2 commits November 15, 2019 14:53

Add landsat masking section to masking docs page

ea8974c

Signed-off-by: Jason T. Brown <jason@astraea.earth>

Merge remote-tracking branch 'lt/develop' into feature/mask_by_bits

a46b3a5

metasim approved these changes Nov 18, 2019

View reviewed changes

vpipkt added 2 commits November 18, 2019 12:37

register rf_local_extract_bit with SQL functions

a71505d

Signed-off-by: Jason T. Brown <jason@astraea.earth>

break out commented assert into skipped unit test around masking and …

419a920

…deserialization Signed-off-by: Jason T. Brown <jason@astraea.earth>

vpipkt merged commit b396400 into locationtech:develop Nov 18, 2019

vpipkt deleted the feature/mask_by_bits branch November 18, 2019 19:00


		Applies a mask from blacklisted bit values in the `mask_tile`. Working from the right, the bits from `start_bit` to `start_bit + num_bits` are @ref:[extracted](reference.md#rf_local_extract_bits) from cell values of the `mask_tile`. In all locations where these are in the `mask_values`, the returned tile is set to NoData; otherwise the original `tile` cell value is returned.

		This function is not available in the SQL API. The below is equivalent:

Masking by bit values #415

Masking by bit values #415

Uh oh!

Conversation

vpipkt commented Nov 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

courtney-layman commented Nov 15, 2019

Uh oh!

metasim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vpipkt commented Nov 11, 2019 •

edited

Loading