feat: refresh packages #819
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Haroenv
reviewed
Dec 13, 2021
src/watch.ts
Outdated
| } | ||
| pushed.push(pkg.objectID); | ||
|
|
||
| // If an update come at the same time, this could override the update with old rev. |
There was a problem hiding this comment.
do you mean while this function is called, or right after this function is called? Could we use a semaphore, and delay adding items to changes consumer until this process is finished, to make sure it's first? Or maybe read the changes that happen between start of this process and end of this process, and skip adding if it's already refreshed for another reason?
|
I'll merge, I need those update to speedup watch.ts |
algolia-crawler-bot
pushed a commit
that referenced
this pull request
Apr 16, 2022
# [1.5.0](v1.4.32...v1.5.0) (2022-04-16) ### Features * refresh packages ([#819](#819)) ([8379ee1](8379ee1))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce the long-awaited live package refresh that will avoid the need for full bootstrap (most of the time).
The strategy is to store the
rev(the internal version of the package) in the record.The
expiresAtfacets is queried, sorted and picked by oldest.We query some records (batch of 20 every 2 minutes, to avoid filling the queue) from the oldest facet value and update them like regular job.
To completely enable this feature we need the
revso for the moment it's not possible and will require one more bootstrap.I don't know yet if it's enough to process everything in a timely fashion.