Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix OOM for high precision exotic shapes #10652

Merged
merged 1 commit into from Apr 21, 2015

Conversation

Projects
None yet
4 participants
@nknize
Copy link
Member

commented Apr 17, 2015

This is currently submitted as a patch in LUCENE-6422 (placed in our lucene package until its committed to lucene 5.x). It removes unnecessary transient memory usage for QuadPrefixTree and, for 1.6.0+ shape indexes adds a new compact bit encoded representation for each quadcell. This is the heart of numerous false positive matches, OOM exceptions, and all around poor shape indexing performance. The compact bit representation will also allows for encoding 3D shapes in future enhancements.

closes #2361
closes #9860
closes #10583

@rjernst

This comment has been minimized.

Copy link
Member

commented Apr 17, 2015

LGTM, but I would rename the lucene classes XBlah to make it clear we will remove once these are in lucene?

@nknize

This comment has been minimized.

Copy link
Member Author

commented Apr 20, 2015

Thanks @ryan! I had given that a shot and it turned into a mass copy/refactor. Because of the abstraction layers in lucene-spatial refactoring the abstract base class names to X* requires copying almost the entire lucene-spatial module.

I can put a comment above the classes indicating they will be removed once LUCENE-6422 is committed?

@rjernst

This comment has been minimized.

Copy link
Member

commented Apr 20, 2015

@nknize Ok, please add the comment.

@nknize nknize added v1.6.0 and removed v1.5.2 labels Apr 21, 2015

@nknize nknize force-pushed the nknize:enhancement/LUCENE_6422 branch from e631888 to ec11003 Apr 21, 2015

[GEO] Fix OOM for high precision exotic shapes
This is currently submitted as a patch in LUCENE-6422. It removes unnecessary transient memory usage for QuadPrefixTree and, for 1.6.0+ shape indexes adds a new compact bit encoded representation for each quadcell. This is the heart of numerous false positive matches, OOM exceptions, and all around poor shape indexing performance. The compact bit representation will also allows for encoding 3D shapes in future enhancements.

@nknize nknize force-pushed the nknize:enhancement/LUCENE_6422 branch from ec11003 to 0205fc7 Apr 21, 2015

@nknize nknize merged commit 0205fc7 into elastic:master Apr 21, 2015

1 check passed

CLA Commit author is a member of Elasticsearch
Details

@clintongormley clintongormley changed the title [GEO] Fix OOM for high precision exotic shapes Fix OOM for high precision exotic shapes May 29, 2015

@clintongormley clintongormley added the >bug label May 29, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.