Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate prefix file, fixing #141 #143

Merged
merged 3 commits into from Nov 15, 2018
Merged

Separate prefix file, fixing #141 #143

merged 3 commits into from Nov 15, 2018

Conversation

niklas88
Copy link
Member

This PR currently has two major commits.

  • The first one simplifies the old code without (hopefully) changing functionality, except for ignoring additional prefixes in the configuration file
  • The second one moves the prefixes from the configuration file into a separate .prefixes file leaving only a boolean flag in the configuration file.

Sadly the MetaDataConverter seems to break on current indices instead of recognizing them as fine. I don't think that has anything to do with my new changes though. Until I figure that out it remains untested. Still I wanted to get this PR here so we don't get a duplication of work.

@niklas88 niklas88 requested a review from joka921 October 17, 2018 17:04
* Remove the unused getJsonForPrefixes() method
* Merge initializeNewPrefixes and initializeRestartPrefixes which only
  differed in if they accepted too many/too few prefixes and clearing
  the _prefixVec
* Also remove TODOs that were cleared up
The MetaDataConverter seems to break on trying to convert (i.e. detect
as fine) a current index. I don't think that has anything todo with the
new changes but these are untested until I figure that out.
@niklas88 niklas88 requested review from floriankramer and removed request for joka921 November 14, 2018 13:17
Copy link
Member

@floriankramer floriankramer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't worked with the index much, so I'm not certain that I haven't missed some important impact of the change, but it mostly looks good to me.
One change I'm not sure about is, that having less than NUM_COMPRESSION_PREFIXES prefixes only throws a warning instead of thrwoing an error. As far as I can tell that should not be an issue, as the value only seems to be important when decompressing strings, where a std::array instead of a vector is used for prefix lookup, preventing access of uninitialized memory, but I'm not quite certain.

src/index/MetaDataConverter.cpp Show resolved Hide resolved
src/index/VocabularyImpl.h Outdated Show resolved Hide resolved
@niklas88 niklas88 merged commit 1de2748 into ad-freiburg:master Nov 15, 2018
@niklas88 niklas88 deleted the separate_prefix_file branch July 19, 2019 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants