Provide granular control to SpatialRDD sampling utils #116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kontinuation
added a commit
to Kontinuation/sedona
that referenced
this pull request
Mar 21, 2024
Kontinuation
added a commit
to Kontinuation/sedona
that referenced
this pull request
Mar 21, 2024
Kontinuation
added a commit
to Kontinuation/sedona
that referenced
this pull request
Mar 22, 2024
jiayuasu
pushed a commit
that referenced
this pull request
Mar 22, 2024
* [SEDONA-406] Raster deserializer for PySpark (#116) * Update documentation * Add documentation for writing Python UDF to work with raster data
Kontinuation
added a commit
to Kontinuation/sedona
that referenced
this pull request
Oct 11, 2024
* Refactor the serializer of RasterUDT to use a language and library neutral serialization format * Changed band data serializer (DataBufferSerializer) to a language neutral format. * Add tests for serializers for raster components * out-db raster now uses a language neutral serialization format * Fixed serializing JAI images with non-zero offset (minX != 0 or minY != 0) * Refactored constructor of DeepCopiedRenderedImage, removed imageBounds from the serialization format since it can be faithfully constructed from minX, minY, width and height of the image * A draft design of the PySpark RasterType deserializer * Implemented affine transformation translation for pixel anchors and image with non-zero origins. * Support all band data types * Add a test constructor for testing python raster deserialization * Implemented as_numpy for ComponentSampleModel and PixelInterleavedSampleModel * Implemented deserializer for SinglePixelPackedSampleModel * Implemented deserializer for MultiPixelPackedSampleModel * Added RS_MakeRasterForTesting to Sedona Spark SQL. This will be used by python unit test to test the correctness of raster deserializer * Switch struct.unpack in a loop with numpy.frombuffer. This is way faster. * Move the prototype code to sedona package * Parsed byte sequence of serialized out-db raster. Now we can start implementing a raster data reader using rasterio WarpedVRT. * Support reading band data as numpy array using rasterio's WarpVRT * out-db raster deserializer is fully functioning, though as_rasterio is still slow. * Added unittests for PySpark raster integration * Added unit test for raster deserialization * Fixed CI errors * Don't shade JAI jars into the sedona-spark shaded jar, otherwise JAI imageio won't work correctly. * Added a test case for Pandas UDF with raster as param * Skip the PandasUDF test if spark version is lower than 3.4 * Minor fixes for formats * Fix a bug of as_rasterio() when retrieving the number of bands of an in-db sedona raster * Added toPandas() to raster serde test cases
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.