Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WKT parsing problem in 0.6.1 version of geospark #83

Closed
geoHeil opened this issue Apr 27, 2017 · 2 comments
Closed

WKT parsing problem in 0.6.1 version of geospark #83

geoHeil opened this issue Apr 27, 2017 · 2 comments

Comments

@geoHeil
Copy link

geoHeil commented Apr 27, 2017

With the latest updates from 0.6.1-snapshot to 0.6.1 release a bug was introduced regarding line string parsing. The error is: LinearRing do not form a closed linestring

To reproduce you can use the data from the Babylon showcase https://github.com/geoHeil/geoSparkScalaSample/tree/master/src/test/resources and run

git clone https://github.com/geoHeil/geoSparkScalaSample.git
cd geoSparkScalaSample
sbt run
# when propmpted for multiple main classes select 2 
@jiayuasu
Copy link
Member

Hi @geoHeil , I have investigated the problem. This is not caused by GeoSpark. There is a typo in your code. It guides PolygonRDD to take a rectangle csv as inputLocation. Any Polygon CSV/TSV input require that, in each tuple, the last coordinate must be same with the first coordinate.

In VisualizationGeosparkLocalRaster.scala, L50 - L53

  val RectangleOffset = prop.getProperty("offset").toInt
  val RectangleSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"))
  val RectangleNumPartitions = prop.getProperty("numPartitions").toInt
  val PolygonInputLocation = "file://" + System.getProperty("user.dir") + "/" + resourcePath + prop.getProperty("inputLocation")

@geoHeil
Copy link
Author

geoHeil commented Apr 27, 2017

Sorry for that one. Indeed, you are right. I wonder what else could have changed with my real dataset.

@geoHeil geoHeil closed this as completed Apr 27, 2017
Kontinuation added a commit to Kontinuation/sedona that referenced this issue Oct 11, 2024
* Start to implement self-driving optimized spatial join

* Making the self-driving optimized spatial join not interfering with the original RDD API

* Added some tests for AdvancedStatCollector and fixed several bugs

* Added tests for advanced stat collector and the size estimator.

* Add tests for advanced spatial join

* Run advancedAnalyze on left and right datasets in parallel

* Use number of geometries inside the join extent to create spatial partitions. Added test for cases where the extent of joined datasets are completely disjoint.

* Use a combination of reservoir sampling and Bernoulli sampling in AdvancedStatCollector

* Implemented adaptive index lookup judgement, without deduplication yet.

* Fixed parallel advanced analyze code

* Use both LongAccumulator and SQLMetric in advanced spatial join

* Add deduplication to AdaptiveIndexLookupJudgement. Now advanced spatial join could generate correct results.

* Added an initial implementation of local spatial join planner;
Added a PREPARE_NONE execution mode when prepared geometry won't help accelerating evaluating the predicate.

* Added comprehensive tests for AdaptiveIndexLookupJudgement

* Fix quadtree performance degrade for original spatial join implementations

* Fix compatibility with Spark 3.0 and 3.1

* Fixed configuration rollback in TestBaseScala.withConf

* Don't show SQL metrics when running old spatial join

* Make more conservative estimation on String sizes

* Fixed several edge case bugs in advanced spatial join

* Add metrics for inspecting the selection of per-partition local join parameters.

* Optimize broadcast index join to make it aware of the type of spatial predicate, and choose the prepared side accordingly.

* Make spatial partitioning strategy involving 2 datasets tunable using spark config.

* Revert a bad optimization I made to skip creating prepared geometry

* Fixed quadtree performance degrade for other join types as well, especially for broadcast indexed join.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants