feat(sqlite): Gracefully handle issues creating sqlite indices #458
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The creation of the `spr_obsolete` index is problematic for the PIP service and any of the admin lookup workers running in the importers. Because there are always multiple processes doing admin lookup, and each one tries to create the index if it doesn't exist, most end up failing. SQLite only allows a single write lock on the DB, so all but one will fail to achieve it. However, in my testing, all it takes to solve this gracefully is to wrap the index creation in a try/catch block. The index will be created by one of the worker processes, and all subsequent queries appear to have the performance improvements of the index (about 30-50% faster time to load admin data). Supersedes #431 Fixes #454 Connects #453 (we should not actually remove the index creation quite yet)
orangejulius
force-pushed
the
gracefully-handle-index-creation-error
branch
from
f00bf30
to
8b0c2ae
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The creation of the
spr_obsoleteindex is problematic for the PIP service and any of the admin lookup workers running in the importers.Because there are always multiple processes doing admin lookup, and each one tries to create the index if it doesn't exist, most end up failing. SQLite only allows a single write lock on the DB, so all but one will fail to achieve it.
However, in my testing, all it takes to solve this gracefully is to wrap the index creation in a try/catch block. The index will be created by one of the worker processes, and all subsequent queries appear to have the performance improvements of the index (about 30-50% faster time to
load admin data).
Supersedes #431
Fixes #454
Connects #453 (we should not actually remove the index creation quite yet, the
whosonfirst-data-latestfile does not have the index)