Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move IntBlockPool's slice allocator and SliceReader/Writer out to MemoryIndex [LUCENE-10211] #11248

Closed
asfimport opened this issue Oct 28, 2021 · 0 comments · Fixed by #12409
Closed

Comments

@asfimport
Copy link

asfimport commented Oct 28, 2021

Spinoff from exciting issue #10712.

Lucene's ByteBlockPool and IntBlockPool are sort of like a little malloc implementation, embedding many logical growing int[] into a series of block buffers.  This is needed to compactly/efficiently store postings data, since there can be many unique terms, each with their own growing byte[] holding postings.

IntBlockPool also has these slices, however, nowhere in core do we use those.  Rather, the allocation needs for int[] is simpler: just allocated a fixed length 1, 2 or 3 int[] per unique term.  We can greatly simplify IntBlockPool.

However, MemoryIndex does use these slices from IntBlockPool.  I think we should move the complex slice logic in IntBlockPool out to MemoryIndex, simplifying core?


Migrated from LUCENE-10211 by Michael McCandless (@mikemccand)

mikemccand pushed a commit that referenced this issue Jul 10, 2023
* [WIP] Move IntBlockPool slices to MemoryIndex

* [WIP] Working TestMemoryIndex

* [WIP} Working TestSlicedIntBlockPool

* Working many allocations tests

* Add basic IntBlockPool test

* SlicedIntBlockPool inherits from IntBlockPool

* Tidy
mikemccand pushed a commit that referenced this issue Jul 10, 2023
* [WIP] Move IntBlockPool slices to MemoryIndex

* [WIP] Working TestMemoryIndex

* [WIP} Working TestSlicedIntBlockPool

* Working many allocations tests

* Add basic IntBlockPool test

* SlicedIntBlockPool inherits from IntBlockPool

* Tidy
@zhaih zhaih added this to the 9.8.0 milestone Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants