Zero downtime upgrades, merge-based index construction #42
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Have a single class responsible for encoding and decoding URL ids, as it's a bit finicky and used all over.
Deprecate the LoadUrl instruction entirely. We no longer need to be told upfront about which URLs to expect, as IDs are generated from the domain id and document ordinal. For now, we no longer store new URLs in different domains. We need to re-implement this somehow, probably in a different job or a as a different output.
Also refactor along the way. Really needs an additional pass, these tests are very hairy.
They seemed like a good idea at the time, but in practice they're wasting resources and not really providing the clarity I had hoped.
SWAP_LEXICON doesn't instruct the index service to do anything. It just moves the file.
It's not necessary anymore with the new linkdb.
This provides a much cleaner separation of concerns, and makes it possible to get rid of a lot of the gunkier parts of the index service. It will also permit lowering the Xmx on the index service a fair bit, so we can get CompressedOOps again :D
This is a system-wide change. The index used to have a lexicon, mapping words to wordIds using a large in-memory hash table. This made index-construction easier, but it also added a fairly significant RAM penalty to both the index service and the loader. The new design moves to 64 bit word identifiers calculated using the murmur hash of the keyword, and an index construction based on merging smaller indices. It also became necessary half-way through to upgrade guice as its error reporting wasn't *quite* compatible with JDK20.
vlofgren
changed the title
This enables documents to be ranked properly.
…ats for reverse index construction.
* Reduce memory churn in LoaderIndexJournalWriter, fix bug with keyword mappings as well * Remove remains of OldDomains * Ensure LOADER_PROCESS_OPTS gets fed to the processes * LinkdbStatusWriter won't execute batch after each added item post 100 items
vlofgren
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes modifications to the loader so that the live production environment doesn't need to be taken offline to prepare a new index.
A big change is keeping the URL database in a separate sqlite db instead of mariadb. This removes the need to take the system offline during loading.
It also moves the index construction bits out of the index-server and into a process, to make it possible to process with a different version of the logic from the index. A very neat side-effect of this is that you get a sort of dehydrated index you can back-up and restore to roll a problematic release with minimal downtime.
The pull request also deprecates the lexicon service altogether, and almost completely rewrites index-construction to use an index merging based approach that does not require as much RAM.
It also reduces the RAM requirements for the index service by a lot, since it no longer needs a lexicon. This makes the index faster because it can use a sub-32 Gb heap and CompressedOOPs. The index service also no longer needs to load the lexicon on start-up, enabling it to restart instantaneously.