Improve performance changes (draft) #18
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marcus-pousette
changed the title
marcus-pousette
changed the title
marcus-pousette
force-pushed
the
add-document-cow
branch
from
3aac183
to
81fc550
Compare
tmpfs
reviewed
Aug 21, 2022
| fmt::{Debug, Formatter}, | ||
| hash::Hash, | ||
| usize, | ||
| }; | ||
|
|
||
| use crate::{FieldAccessor, Filter, Tokenizer}; | ||
| use crate::{FieldAccessor, Tokenizer}; | ||
| use hashbrown::{HashMap, HashSet}; |
There was a problem hiding this comment.
That's quite a significant change in performance, I guess most of it is the switch to hashbrown?
I can imagine a small gain from removing the Filter calls but not that significant.
There was a problem hiding this comment.
Yes. Most is hashbrown . A small portion seem to be from the Cow change in the add_document also, but much less significant
tmpfs
approved these changes
Aug 21, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was playing our with the library so I created a few changes I could possibly split into multiple PRs.
Would like to hear you thoughts about this
Changes for improved performance
add_documentuseCowfor hash keys.hashbrownHashmap instead ofstd::collections. Faster for small hash keys. Though does not provide the same level of HashDoS resistanceTermDataquery_termsproperty is replaced withquery_terms_lento prevent unnecessary copies of all query terms during query (not benchmarked yet) (Breaking change for theScoreCalculatortrait)Bench results on my computer
add_100k_docsMaster
301.19 ms
This branch
221.24 ms
API change ideas
Filteris removed, since theTokenizercan just be seen as a preprocessing step to indexation and query and can do anything that theFilterdoes. One argument less to worry about. (Breaking change)