GeoSpark Core vs GeoSpark SQL Performance in 1.2

[jhamiltonpro]

I have read the release notes but I still have a couple questions regarding the latest version of GeoSpark:

1. Any performance improvements with Spatial Join (ST_CONTAINS) in either core or sql?
2. As a general question, should we still always try to use GeoSpark Core and SpatialRDD instead of dataframes and SparkSQL for optimal performance? Especially for spatial joins, there have been issues like this one: Use Spatial Indexes in Geospark SQL? #217 (comment)
   which suggest using GeoSpark Core is preferable. This page http://datasystemslab.github.io/GeoSpark/tutorial/benchmark/ also suggests using core over sql.

I've encountered issues with slow GeoSpark SQL performance due to uneven distribution of work across executors as outlined here: #249 (comment)
and have followed a lot of the steps that you've outlined in previous posts to help with this issue. Unfortunately, the skew still persists and I'm wondering if there's simply a difference between using core and sql and if 1.2 has anything which helps with performance (compared to 1.1.3).

[jiayuasu]

@jhamiltonpro Hi, there is no join performance improvement in GeoSpark 1.2. The reason why I recommend GeoSpark-core because you have more granular control over the join query, such as num partitions used in the join, index type, grid type and caching.

If your polygon data has lots of overlaps and you use line string data, the skewness is very hard to eliminate.

[jhamiltonpro]

@jiayuasu thanks for the response. So regarding the skew, if I were joining a big table (of points) with a small table (of potentially complex or overlapping polygons), would implementing some tricks as outlined here: https://stackoverflow.com/questions/40373577/skewed-dataset-join-in-spark be worth it in your opinion or does it not really apply in the case of spatial joins?

And do you think splitting the actual polygons in the polygon table would help? Say instead of 1,000 rows of big complex polygons, split each polygon into 4 partitions and have 4,000 rows of smaller polygons? (I'm not sure how I would split the polygons yet, but if it's worth doing maybe I''ll try.)