Focal Operations and Buffer Tile support #563

pomadchin · 2021-09-22T00:22:59Z

It is a port of #551

TODO:

Focal Expressions should be of a correct arity
Python Focal API ~~, blocked by Python binding updates for PySpark 3.1 #562~~
- Docker image for tests: quay.io/daunnc/rasterframes-notebook:0.9.2-SNAPSHOT
Python STAC API
After reading a buffered tile and performing a local operation the buffer is lost: I propose that to be not a part of this PR, but to move it into a separate one as an enchancement.

Focal Operations notebook screenshot

STAC API Example notebook screenshot

Closes #441
Closes #551
Closes #487

Blocked by locationtech/geotrellis#3419

core/src/main/scala/org/locationtech/rasterframes/expressions/focalops/SurfaceOp.scala

datasource/src/test/scala/examples/BufferTiles.scala

pomadchin · 2021-09-22T23:18:41Z

core/src/main/scala/geotrellis/raster/BufferTile.scala

+ * - mutable version makes sense
+ * - toBytes needs to encode padding size?
+ */
+case class BufferTile(


TODO: merge into GeoTrellis

locationtech/geotrellis#3419

core/src/main/scala/org/apache/spark/sql/rf/TileUDT.scala

…unction signature change

metasim

🤘 🧙 💪

Can you update the release-notes.md file under a 0.10 heading? I forgot to ask for that in the last PR too.

metasim · 2021-10-01T18:50:20Z

core/src/main/scala/geotrellis/raster/mapalgebra/focal/BufferedFocalMethods.scala

+   */
+  def aspect(cs: CellSize, bounds: Option[GridBounds[Int]] = None, target: TargetCell = TargetCell.All): BufferTile =
+    self.mapTile(_.aspect(cs, bounds, target))
+}


Should this go into GT?

locationtech/geotrellis#3419

metasim · 2021-10-01T18:52:58Z

core/src/main/scala/org/apache/spark/sql/rf/TileUDT.scala

@@ -49,6 +51,7 @@ class TileUDT extends UserDefinedType[Tile] {
    StructField("cols", IntegerType, false),
    StructField("rows", IntegerType, false),
    StructField("cells", BinaryType, true),
+    StructField("gridBounds", gridBoundsEncoder[Int].schema, true),


Yes! This is what I meant by making "everything a buffered tile".... it's effectively enabled now (with perhaps some further propogation)

metasim · 2021-10-01T18:53:55Z

core/src/main/scala/org/apache/spark/sql/rf/TileUDT.scala

+        val ct = CellType.fromName(row.getString(0))
+        val cols = row.getInt(1)
+        val rows = row.getInt(2)
+        val bytes = row.getBinary(3)
+        val gridBounds = row.getStruct(4, 5).as[GridBounds[Int]]
+        BufferTile(ArrayTile.fromBytes(bytes, ct, cols, rows), gridBounds)


One thing I miss about the CatalystSerializers... all this magic index number stuff was in one place.

metasim · 2021-10-01T19:03:13Z

core/src/main/scala/org/locationtech/rasterframes/expressions/UnaryRasterFunction.scala

-    val (childTile, childCtx) = tileExtractor(child.dataType)(row(input))
-    val result = op(childTile)
-    toInternalRow(result, childCtx)
+    // TODO: Ensure InternalRowTile is preserved


This reminds me... if we want to start improving performance (mainly reducing copies), we need to figure out how ensure InternalRowTile <op> R => InternalRowTile. Either requires changes to how GeoTrellis instantiates destination tiles, or reimplementation of some fundamental operations.

metasim · 2021-10-01T19:04:48Z

core/src/main/scala/org/locationtech/rasterframes/functions/FocalFunctions.scala

+    rf_hillshade(tileCol, lit(azimuth), lit(altitude), lit(zFactor))
+
+  def rf_hillshade(tileCol: Column, azimuth: Column, altitude: Column, zFactor: Column): Column =
+    Hillshade(tileCol, azimuth, altitude, zFactor)


metasim · 2021-10-01T19:12:50Z

pyrasterframes/src/main/python/pyrasterframes/__init__.py

+    """
+    uri - STAC API uri
+    filters - a STAC API Search filters dict (bbox, datetime, intersects, collections, items, limit, query, next)
+    search_limit - search results convenient limit method


This might be better as named parameters, or as **kwargs.

Discoverability when working in a notebook or something would benefit from these parameters being fully spelled out with defaults.

metasim · 2021-10-01T19:14:14Z

pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py

+        altitude = lit(altitude)
+    if isinstance(z_factor, (int, float)):
+        z_factor = lit(z_factor)
+    return _apply_column_function('rf_hillshade', tile_col, azimuth, altitude, z_factor)


Nicely done!!

Sorry there's documentation duplicated. In another universe I would have figured out how to have the Python API generated from the annotations on the Expressions.

metasim · 2021-10-01T19:15:45Z

pyrasterframes/src/main/python/pyrasterframes/rf_types.py

@@ -390,6 +390,11 @@ def __init__(self, cells, cell_type=None):
                # if the value in the array is `nd_value`, it is masked as nodata
                self.cells = np.ma.masked_equal(self.cells, nd_value)

+        # is it a buffer tile? crop it on extraction to preserve the tile behavior


Not sure this is the right thing to do, but don't want to hold up the PR for this. Back to the "everything's a buffered tile" perspective, I'd want to explore supporting buffered tile semantics on the Python mirror type.

@metasim yes, for now it is done to preserve the old behavior and not to break anything.

metasim · 2021-10-01T19:16:55Z

pyrasterframes/src/main/python/pyrasterframes/rf_types.py

        except ValueError as e:
            raise ValueError({
                "cell_type": cell_type,
                "cols": cols,
                "rows": rows,
                "cell_data.length": len(cell_data_bytes),
                "cell_data.type": type(cell_data_bytes),
-                "cell_data.values": repr(cell_data_bytes)
+                "cell_data.values": repr(cell_data_bytes),
+                "grid_bounds": datum.gridBounds


metasim · 2021-10-01T19:17:14Z

rf-notebook/src/main/docker/Dockerfile

@@ -1,21 +1,21 @@
 # jupyter/scipy-notebook isn't semantically versioned.
 # We pick this arbitrary one from Sept 2019 because it's what latest was on Oct 17 2019.
-FROM jupyter/scipy-notebook:7a0c7325e470
+FROM jupyter/scipy-notebook:python-3.8.8


pomadchin force-pushed the feature/buffer-tiles branch 15 times, most recently from ed2c406 to 29dc373 Compare September 22, 2021 19:17

metasim reviewed Sep 22, 2021

View reviewed changes

core/src/main/scala/org/locationtech/rasterframes/expressions/focalops/SurfaceOp.scala Outdated Show resolved Hide resolved

metasim reviewed Sep 22, 2021

View reviewed changes

datasource/src/test/scala/examples/BufferTiles.scala Show resolved Hide resolved

pomadchin force-pushed the feature/buffer-tiles branch from 11dd3d9 to 9f0e79d Compare September 22, 2021 21:54

pomadchin commented Sep 22, 2021

View reviewed changes

core/src/main/scala/org/apache/spark/sql/rf/TileUDT.scala Outdated Show resolved Hide resolved

pomadchin force-pushed the feature/buffer-tiles branch 10 times, most recently from 2134586 to 602d818 Compare September 23, 2021 01:29

Move to the scipy notebook version

9cb9c7f

pomadchin force-pushed the feature/buffer-tiles branch 2 times, most recently from db29294 to 9cb9c7f Compare September 28, 2021 20:34

pomadchin added 5 commits September 28, 2021 17:20

Add FocalFunctions specs

005492f

Add a comment about how to achieve BufferTile preservation

de57a3b

Make FocalOps valid expressions

d569195

Update FocalOps Python bindings

07723da

Add a missing copyright header

b127329

pomadchin force-pushed the feature/buffer-tiles branch from 7684fd8 to b127329 Compare September 29, 2021 21:58

pomadchin added 2 commits September 29, 2021 22:51

Add integers support in py focal functions

a092d49

Update FocalOperations.ipynb results

c406536

pomadchin force-pushed the feature/buffer-tiles branch from 6637698 to c406536 Compare September 30, 2021 04:02

pomadchin mentioned this pull request Sep 30, 2021

BufferTile buffer preservation #565

Open

pomadchin requested a review from metasim September 30, 2021 12:41

metasim changed the title ~~[WIP] Focal Operations and Buffer Tile support~~ Focal Operations and Buffer Tile support Sep 30, 2021

pomadchin marked this pull request as ready for review September 30, 2021 13:39

make extractBufferTile function total, FocalNeighborhood.fromString f…

d79a6ed

…unction signature change

pomadchin force-pushed the feature/buffer-tiles branch from 5c48898 to d79a6ed Compare September 30, 2021 17:01

pomadchin added 3 commits September 30, 2021 17:48

Add STAC API Python proxy

d24fd3a

Change the STAC API search limit handling

934fa0b

Add STAC API Example jupyther notebook

fdd46cb

metasim reviewed Oct 1, 2021

View reviewed changes

pomadchin added 2 commits October 1, 2021 15:29

Remove STACDataFrame searchLimit parameter

1a74338

Update release notes and fix pydocs

c7d8f48

pomadchin force-pushed the feature/buffer-tiles branch from f803707 to c7d8f48 Compare October 1, 2021 19:50

metasim merged commit dc72ece into locationtech:develop Oct 1, 2021

pomadchin deleted the feature/buffer-tiles branch October 2, 2021 03:29

pomadchin restored the feature/buffer-tiles branch October 4, 2021 21:18

jpolchlo mentioned this pull request Aug 4, 2022

Unstable shapely dependency locationtech-labs/geopyspark#712

Closed

soxofaan mentioned this pull request Aug 10, 2022

GeoPySpark in the long term? Open-EO/openeo-geopyspark-driver#193

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Focal Operations and Buffer Tile support #563

Focal Operations and Buffer Tile support #563

pomadchin commented Sep 22, 2021 •

edited

Loading

pomadchin Sep 22, 2021

pomadchin Sep 23, 2021

metasim left a comment

metasim Oct 1, 2021

pomadchin Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

pomadchin Oct 1, 2021

metasim Oct 1, 2021

metasim Oct 1, 2021

Focal Operations and Buffer Tile support #563

Focal Operations and Buffer Tile support #563

Conversation

pomadchin commented Sep 22, 2021 • edited Loading

Focal Operations notebook screenshot

STAC API Example notebook screenshot

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metasim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pomadchin commented Sep 22, 2021 •

edited

Loading