Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geotrellis 3.x #398

Merged
merged 32 commits into from Feb 20, 2020
Merged

Geotrellis 3.x #398

merged 32 commits into from Feb 20, 2020

Conversation

metasim
Copy link
Member

@metasim metasim commented Oct 21, 2019

Geotrellis 3.x Upgrade Log

  • Add Int type parameter to Grid
  • Add Int type parameter to CellGrid
  • Add Int type parameter to GridBounds... or TileBounds
  • Use GridBounds.toGridType to coerce from Int to Long type parameter
  • Update imports for layers, particularly geotrellis.spark.tiling to geotrellis.layer
  • Update imports for geotrellis.spark.io to geotrellis.spark.store...
  • Removed FixedRasterExtent
  • Removed org.locationtech.rasterframes.util.Shims
  • Change Extent.jtsGeom to Extent.toPolygon
  • Change TileLayerMetadata.gridBounds to TileLayerMetadata.tileBounds
  • Add geotrellis-gdal dependency
  • Remove any conversions between JTS geometry and old geotrellis.vector geometry
  • Changed org.locationtech.rasterframes.encoders.StandardEncoders.crsEncoder to crsSparkEncoder
  • Change (cols, rows) dimension destructuring to Dimensions(cols, rows)
  • Revisit use of Tile equality since it's more strict
  • Update reference.conf to use geotrellis.raster.gdal namespace.
  • Replace all uses of TileDimensions with geotrellis.raster.Dimensions[Int].

TODO

  • PR to GT to make FixedDelegatingTile obsolete. (cf combine when implementing a DelegatingTile incorrectly handles NoData geotrellis#3153)
  • Replace TileDimension with geotrellis.raster.Dimensions
  • Local RasterSource updated to use CellGrid[Long]
  • Replace SimpleRasterInfo with GeoTiffInfo (where possible). Revisit RasterExtent vs GridExtent[Long]
  • Replace all Fixed* classes.
  • Remove RasterSource altogether.
  • Replace spray-json with circe.

Remaining tasks carried over to #476.

@metasim metasim changed the title Initial port to Geotrellis 3.0 Geotrellis 3.0 Oct 21, 2019
@metasim
Copy link
Member Author

metasim commented Oct 22, 2019

Depends on locationtech/geotrellis#3134

@pomadchin
Copy link
Member

pomadchin commented Nov 5, 2019

@metasim you can depend on GeoTrellis 3.1.0 and use GeoTrellis projected like now.

@metasim
Copy link
Member Author

metasim commented Nov 5, 2019

@pomadchin Got pulled onto other work.... definitely near the top of my RF todo list.

metasim added 2 commits Nov 8, 2019
* develop:
  PR feedback.
  PR feedback. Regression fix. Release notes update.
  Added forced truncation of WKT types in Markdown/HTML rendering.
  Ensure default tile size is applied to `raster` reader.
  Fix nodata doc
  Doc supervised, set tile size to 256 for visual
  Update doc to use rf_local_is_in when masking; fix locationtech#351
  Close locationtech#310 move reference to static
  rf_local_is_in python implementation
  Updated intro section. Added additional raster-read section.
  Applying pre-partitioning to DataSources.
  Expanded RasterRefSpec to ensure lazy tiles provide metadata without I/O.
  Fix unit tests for rf_local_is_in
  Attempting to keep TravisCI from timing out by using jobs.
  Add rf_local_is_in function
@metasim
Copy link
Member Author

metasim commented Nov 14, 2019

@pomadchin GT 3.1.0 working great!

metasim added 3 commits Nov 14, 2019
* develop:
  Masking docs: show preview of clipped df
  Mark final code chunk as executable in masking page
  Masking docs: show preview of clipped df
  Attempt to fix "ImportError: libpoppler.so.76: cannot open shared object file: No such file or directory"
  Attempt to fix "ImportError: libpoppler.so.76: cannot open shared object file: No such file or directory"
  Mark final code chunk as executable in masking page
  Bumped to next dev version.
  0.8.4 release prep.
  Misc documentation fixes.
  Fixes to rf_mask_by_values to compose over IfIn and have SQL docs. Removed inverse flag from rf_mask_by_values before committing to an approach with it.
  Attempt to create Expression for MaskByValues to enable sql api
  Fix test for rf_mask_by_values and add SQL test
  Expand rf_mask_by_value[s] signatures to match python api, expand unit tests
  Update documentation on masking
  Update mask functions with inverse arg, add rf_mask_by_values function
  Add Scala overload taking a literal Array in rf_local_is_in

# Conflicts:
#	pyrasterframes/src/main/python/pyrasterframes/version.py
#	version.sbt
@metasim metasim changed the title Geotrellis 3.0 Geotrellis 3.1 Nov 18, 2019
metasim and others added 9 commits Nov 22, 2019
* develop: (24 commits)
  remove unused imports
  Python unit tests check read pipeline stages
  Cruft removal.
  Fix for ML transformer read/write
  Add failing unit tests for issue 425 ML custom transformer loading is broken
  PR feedback.
  Fixed `ReprojectToLayer` to work with `proj_raster`. Closes locationtech#357 Depends on locationtech#420.
  break out commented assert into skipped unit test around masking and deserialization
  register rf_local_extract_bit with SQL functions
  Added the ability to do a raster_join on proj_raster types. Fixes locationtech#419.
  Add landsat masking section to masking docs page
  Add mask bits  python api and unit test
  Extract bits should throw on non-integral cell types
  Fix for both masking by def and value; expand code comments; update tests
  Python regression.
  Masking improvements and unit tests.
  Regression
  Added (disabled) integration test for profiling spatial index effects. Reorganized non-evaluated documentationm files to `docs`.
  Add failing unit test for mask by value on 0
  Updated python tests against spatial index functions.
  ...

# Conflicts:
#	core/src/main/scala/org/locationtech/rasterframes/expressions/DynamicExtractors.scala
#	core/src/main/scala/org/locationtech/rasterframes/expressions/transformers/XZ2Indexer.scala
#	core/src/main/scala/org/locationtech/rasterframes/extensions/Implicits.scala
#	core/src/test/scala/examples/CreatingRasterFrames.scala
#	core/src/test/scala/org/locationtech/rasterframes/RasterLayerSpec.scala
#	core/src/test/scala/org/locationtech/rasterframes/expressions/XZ2IndexerSpec.scala
#	datasource/src/main/scala/org/locationtech/rasterframes/datasource/raster/RasterSourceRelation.scala
#	datasource/src/test/scala/org/locationtech/rasterframes/datasource/raster/RasterSourceDataSourceSpec.scala
Signed-off-by: Jason T. Brown <jason@astraea.earth>
Formally adopting Python 3 / abandon Python 2
* develop: (41 commits)
  Bumped dev version.
  Release prep.
  Added `fix/docs.*` to docs build triggers.
  Atttempted fix for failing pip install in docs CI job.
  Fix mistakes in merge with develop
  Fixes to scala classification model example
  Reconstituted two forms of the mini classification example in Scala.
  PR feedback
  Close locationtech#228 and locationtech#331 with discussion on writing page; clean up unsupervised learning output.
  Various PR improvements
  Update Python bindings and test
  Remove incorrect Expression and SQL registry
  Add documentation, python bindings for rf_agg_approx_quantiles
  Fix column selection
  Refine CRS WKT parsing error message and test
  Update tests to prefer use of rf_agg_approx_quantiles implementation
  Revert "Add DataFrame tileStats extension"
  Remove incorrect whitespace before triple backtick
  Filter sample raster join DF to show non-empty tiles
  Initial raster join page; expand spatial index discussion
  ...

# Conflicts:
#	core/src/main/scala/org/locationtech/rasterframes/RasterFunctions.scala
#	core/src/main/scala/org/locationtech/rasterframes/encoders/StandardSerializers.scala
#	core/src/main/scala/org/locationtech/rasterframes/expressions/aggregates/TileRasterizerAggregate.scala
#	core/src/main/scala/org/locationtech/rasterframes/extensions/Implicits.scala
#	core/src/main/scala/org/locationtech/rasterframes/model/LazyCRS.scala
#	core/src/main/scala/org/locationtech/rasterframes/util/package.scala
#	core/src/test/scala/org/locationtech/rasterframes/RasterFunctionsSpec.scala
#	core/src/test/scala/org/locationtech/rasterframes/RasterJoinSpec.scala
#	core/src/test/scala/org/locationtech/rasterframes/RasterLayerSpec.scala
#	core/src/test/scala/org/locationtech/rasterframes/TestData.scala
#	project/RFDependenciesPlugin.scala
#	pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py
#	pyrasterframes/src/main/python/pyrasterframes/rf_context.py
#	pyrasterframes/src/main/python/pyrasterframes/rf_ipython.py
#	pyrasterframes/src/main/python/pyrasterframes/rf_types.py
#	pyrasterframes/src/main/python/pyrasterframes/version.py
#	pyrasterframes/src/main/python/tests/RasterFunctionsTests.py
#	version.sbt
Signed-off-by: Jason T. Brown <jason@astraea.earth>
* develop:
  Fix locationtech#452. Upgrade rasterframes-notebook container version of rtree.
  Add map display to raster-read docs with tile and extent
@metasim metasim marked this pull request as ready for review Jan 29, 2020
@metasim metasim requested a review from vpipkt Jan 29, 2020
metasim and others added 8 commits Feb 10, 2020
* develop:
  Refactor clip to clamp
  Use rf_local_clip and other viz functions in supervised learning doc page
  rf_rescale tests and refinements
  Add Rescale function to Scala, Python and SQL APIs
  Add rf_where and rf_standardize functions
  Add rf_local_min, rf_local_max, and rf_local_clip functions

# Conflicts:
#	pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py
* develop:
  Cherry picked: de8464b
  Expand raster_join test case with null value in left hand tile column
  Updated raster_join test to filter NA rows.
  Typo.
  reproject_and_merge returns null if LHS dimension, extent or crs are null.
  Additional fixes to type handling of CRS and Extent from Python.
  Added ability to specify Extent with arbitrarily ordered fields in Python.
  Fixed misnamed field in Envelope serializer.
  Add failing unit test for issue 462
  Added support for Extents defined with bigint in Python.
  Failing unit test for st_geometry extents bug
  Skip failing test for 458
  Add failing test for issue 458; failing to render null values in tile column

# Conflicts:
#	build.sbt
#	core/src/main/scala/org/locationtech/rasterframes/expressions/DynamicExtractors.scala
#	core/src/main/scala/org/locationtech/rasterframes/expressions/accessors/GetCRS.scala
#	core/src/main/scala/org/locationtech/rasterframes/expressions/transformers/ExtentToGeometry.scala
#	pyrasterframes/src/main/python/pyrasterframes/rasterfunctions.py
#	pyrasterframes/src/main/python/pyrasterframes/rf_types.py
Signed-off-by: Jean-Denis Giguère <jean-denis@anagraph.io>
(cherry picked from commit 380e40b)
@metasim metasim changed the title Geotrellis 3.1 Geotrellis 3.x Feb 20, 2020
metasim added 2 commits Feb 20, 2020
* develop:
  rf_tile_to_array_int returns Int, not double. Fix locationtech#473
  Disabled column type checking in RasterJoin.
  Fix for propogating TileLayerMetadata tile dimensions to layer creation.
  Fix for pyspark AttributeError on sparksession _conf member
  Regressions.

# Conflicts:
#	core/src/main/scala/org/locationtech/rasterframes/extensions/RasterJoin.scala
#	core/src/main/scala/org/locationtech/rasterframes/extensions/ReprojectToLayer.scala
@metasim metasim merged commit f6c7c67 into locationtech:develop Feb 20, 2020
@metasim metasim deleted the feature/gt-3.0 branch Feb 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants