Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to pass in Spatial Index precision as a hint during the join #144

Closed

Conversation

halfabrane
Copy link
Contributor

There are use cases where within the same Spark run (ie with the same spark context) we want to join multiple different spatial datasets at different resolutions.
This PR allows you to pass in the resolution as a hint as follows:

import org.apache.spark.sql.magellan.dsl.expressions._
point.join(polygons index 15).where($"point" within $"polygon")

allows you to pass a hint to the optimizer that the index resolution should be set to 15.

Future enhancements:

  • Ability to pass in a min/ max precision to greedily cover the geometries?
  • Ability to index geometries at the right precision on the fly?

@codecov-io
Copy link

codecov-io commented Aug 11, 2017

Codecov Report

Merging #144 into master will decrease coverage by 0.02%.
The diff coverage is 84.61%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #144      +/-   ##
==========================================
- Coverage   85.47%   85.44%   -0.03%     
==========================================
  Files          47       48       +1     
  Lines        1363     1374      +11     
  Branches       96       95       -1     
==========================================
+ Hits         1165     1174       +9     
- Misses        198      200       +2
Impacted Files Coverage Δ
src/main/scala/magellan/Utils.scala 100% <100%> (ø) ⬆️
src/main/scala/magellan/catalyst/SpatialJoin.scala 100% <100%> (ø) ⬆️
src/main/scala/magellan/dsl/package.scala 84.61% <100%> (+4.61%) ⬆️
...main/scala/magellan/catalyst/SpatialJoinHint.scala 50% <50%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b8c595...938ace8. Read the comment docs.

@harsha2010 harsha2010 closed this Aug 11, 2017
@halfabrane halfabrane deleted the DYNAMICALLY-CONFIGURE-PRECISION branch August 11, 2017 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants