-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
POC: Feat/make indexing more resiliant #16546
base: v13/dev
Are you sure you want to change the base?
Conversation
Hi there @bielu, thank you for this contribution! 👍 While we wait for one of the Core Collaborators team to have a look at your work, we wanted to let you know about that we have a checklist for some of the things we will consider during review:
Don't worry if you got something wrong. We like to think of a pull request as the start of a conversation, we're happy to provide guidance on improving your contribution. If you realize that you might want to make some changes then you can do that by adding new commits to the branch you created for this work and pushing new commits. They should then automatically show up as updates to this pull request. Thanks, from your friendly Umbraco GitHub bot 🤖 🙂 |
Thanks @bielu - I have noted this PR and asked for the team to have a look and see if it's the right direction. Of course we have a "little" conference coming up next week so it's taking a bit longer to get to. |
/// <summary> | ||
/// | ||
/// </summary> | ||
public class IndexRebuildStatusManager : IIndexRebuildStatusManager |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My recommended approach would be to do this: Shazwazza/Examine#372 (comment)
The only real way to know if the indexing is done in a resilient way would be to have an actual document in the index certifying that rebuilding is successful instead of relying on in-memory cache which is problematic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Shazwazza I am not 100% convinced about usage of additional index, as we both know less indexes is actually better with lucene. I am thinking maybe we should use additonal sql table, as it will be eqally resiliant as using index, but it will not require us to create index, what you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bielu Sorry, I probably wasn't clear in my suggestion. We don't want to use an extra index to store any data, we can just use a marker document within the index. For example:
- Rebuilding an index deletes all data
- The index is populated with the normal data
- When the IndexPopulator is done populating the index, it then writes a special marker document signaling that the populator is done. Perhaps this document has a field like
__Populated: y
Then the rebuild checker, just checks if the document count for __Populated: y
== 1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Shazwazza that's make sense now! We can also extend it to check what populator are registered to show how many of them is done! I will make update to this pr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Shazwazza i started changing implementation of this service to use examine underhood, can you have quick look and check if that is what you had in mind?
as in this way now we can also repeat failed batches (but i think I will need play around little more)
Prerequisites
If there's an existing issue for this PR then this fixes
Description
This Pr is POC for solving issue with rebuilding of indexes on startup. Also adding small flexibility around page size when reindexing as current we pulled 10k nodes, which if they have more than 100 properties might cause really slow indexing when we pull more than 1k. I am making it as POC as won't spend more time on this pr unless HQ confirm this is the way and I should improve code in this pr.