-
Notifications
You must be signed in to change notification settings - Fork 693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WKT parsing problem in 0.6.1 version of geospark #83
Comments
Hi @geoHeil , I have investigated the problem. This is not caused by GeoSpark. There is a typo in your code. It guides PolygonRDD to take a rectangle csv as inputLocation. Any Polygon CSV/TSV input require that, in each tuple, the last coordinate must be same with the first coordinate. In VisualizationGeosparkLocalRaster.scala, L50 - L53
|
Sorry for that one. Indeed, you are right. I wonder what else could have changed with my real dataset. |
* Start to implement self-driving optimized spatial join * Making the self-driving optimized spatial join not interfering with the original RDD API * Added some tests for AdvancedStatCollector and fixed several bugs * Added tests for advanced stat collector and the size estimator. * Add tests for advanced spatial join * Run advancedAnalyze on left and right datasets in parallel * Use number of geometries inside the join extent to create spatial partitions. Added test for cases where the extent of joined datasets are completely disjoint. * Use a combination of reservoir sampling and Bernoulli sampling in AdvancedStatCollector * Implemented adaptive index lookup judgement, without deduplication yet. * Fixed parallel advanced analyze code * Use both LongAccumulator and SQLMetric in advanced spatial join * Add deduplication to AdaptiveIndexLookupJudgement. Now advanced spatial join could generate correct results. * Added an initial implementation of local spatial join planner; Added a PREPARE_NONE execution mode when prepared geometry won't help accelerating evaluating the predicate. * Added comprehensive tests for AdaptiveIndexLookupJudgement * Fix quadtree performance degrade for original spatial join implementations * Fix compatibility with Spark 3.0 and 3.1 * Fixed configuration rollback in TestBaseScala.withConf * Don't show SQL metrics when running old spatial join * Make more conservative estimation on String sizes * Fixed several edge case bugs in advanced spatial join * Add metrics for inspecting the selection of per-partition local join parameters. * Optimize broadcast index join to make it aware of the type of spatial predicate, and choose the prepared side accordingly. * Make spatial partitioning strategy involving 2 datasets tunable using spark config. * Revert a bad optimization I made to skip creating prepared geometry * Fixed quadtree performance degrade for other join types as well, especially for broadcast indexed join.
With the latest updates from 0.6.1-snapshot to 0.6.1 release a bug was introduced regarding line string parsing. The error is:
LinearRing do not form a closed linestring
To reproduce you can use the data from the Babylon showcase https://github.com/geoHeil/geoSparkScalaSample/tree/master/src/test/resources and run
The text was updated successfully, but these errors were encountered: