Skip to content

Shared trie#24

Merged
maxitg merged 9 commits intomasterfrom
feature/sharedTrie
Mar 2, 2021
Merged

Shared trie#24
maxitg merged 9 commits intomasterfrom
feature/sharedTrie

Conversation

@maxitg
Copy link
Copy Markdown
Owner

@maxitg maxitg commented Mar 2, 2021

Changes

  • Use shared trie for a range evaluation.
  • This uses more memory but saves a lot of time because the same evaluation paths ("highways") don't need to be evaluated multiple times.
  • To accommodate this case, MergedWithAnotherInit was added as a ConclusionReason.

Comments

  • There is currently no way to find out which other init this one was merged into. This potentially can be added, but it will require a change in the file format, so I did not add it now.
  • A better approach might be to make this optimization invisible by automatically figuring out the values of all evaluation result components based on the information about the merge. I can work on this approach now, but I'm not sure I can finish it today (so it might have to wait until next week).

Examples

  • The difference in performance can be seen with a disabled test:
TEST(PostTagSearcher, DISABLED_rangePerformance) {
  // With separate tries: 0.36 GB, 1133 seconds
  // With shared trie: 1.1 GB of RAM, 93 seconds, 12x speedup, 3x more memory use
  PostTagSearcher().evaluateRange(30, 0, 1000000, PostTagSearcher::EvaluationParameters());
}

This change is Reviewable

@maxitg maxitg added the enhancement New feature or request label Mar 2, 2021
@maxitg maxitg requested review from okofish and sskini2 March 2, 2021 01:14
@maxitg maxitg merged commit 135f11e into master Mar 2, 2021
@maxitg maxitg deleted the feature/sharedTrie branch March 2, 2021 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants