Error locating index files - System does not self heal and rebuild indexes.

[ChrisMIntuitiv]

Which Umbraco version are you using?

16.1.0

Bug summary

We are seeing sporadic but frequent errors relating to lucene indexes, both custom indexes and umbraco built in ones.

There are a varienty of error messages being served but most seem to relate to not being able to find a file for that index for a particular action. Here are some examples:

• Error deleting Lucene index
• Error indexing queue items
• Failed to create report of type Event
• An error occured trying to get determine index shouldRebuild status for index MembersIndex. The index will NOT be considered for rebuilding
• An error occurred during the index commit operation, if this error is persistent then index rebuilding is necessary

Our current line of thinking is that if a system action is taking place on the index, perhaps commiting new records or updating for eaxmple when the app restarts this causes the issue. This would account for the issue not occuring on every app restart.

We also have never seen this in our development environment, we tend to use IISExpress locally rather than IIS.

During day to day running of the application in production the indexes dont seem to have any issues, however when the issue occurs the application does not recover and the index is never rebuilt until we manually intervene.

Specifics

In nearly all cases we are able to rebuild the indexes via the 'settings' tab in the back office or by deleting the current index files from disk and then restarting the app causing Umbraco to see the folder is empty and rebuild the index from the ground up. (The second route is typically only required in our load balanced environments where the admin is separate as the rebuild from the backoffice only triggers the action on itself and not on subcriber nodes).

We have come up against this issue in a few different applications, differing umbraco versions from 13.X -> 16.X, some load balanced, some not, some with custom indexes, some just using the built in ones. We have also tried storing the index files outside of the site folder in some cases and the issue repeats using either the default storage location or in the system TEMP directory. We have also had the same issue with the Umbraco Forms index.

Below is the stack trace of one example from a 16.1.0 application, but they appear to all follow the same pattern. I can provide these if useful but they are just different index names and files names.

System.IO.FileNotFoundException: Could not find file '...\TEMP\ExamineIndexes\cde747b94d888d940774d00d09a39ebb\ExternalIndex\_70.si'.

File name: '...\TEMP\ExamineIndexes\cde747b94d888d940774d00d09a39ebb\ExternalIndex\_70.si'

   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)

   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)

   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)

   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)

   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)

   at Lucene.Net.Store.MMapDirectory.OpenInput(String name, IOContext context)

   at Lucene.Net.Store.NRTCachingDirectory.OpenInput(String name, IOContext context)

   at Lucene.Net.Store.Directory.OpenChecksumInput(String name, IOContext context)

   at Lucene.Net.Codecs.Lucene46.Lucene46SegmentInfoReader.Read(Directory dir, String segment, IOContext context)

   at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)

   at Lucene.Net.Index.SegmentInfos.FindSegmentsFileAnonymousClass.DoBody(String segmentFileName)

   at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)

--- End of stack trace from previous location ---

   at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)

   at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run()

   at Lucene.Net.Index.SegmentInfos.Read(Directory directory)

   at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)

   at Examine.Lucene.Providers.LuceneIndex.CreateIndexWriterWithOpenMode(Directory d, OpenMode openMode)

   at Examine.Lucene.Providers.LuceneIndex.CreateIndexWriter(Directory d)

   at Examine.Lucene.Providers.LuceneIndex.CreateIndexWriterWithLockCheck()

   at Examine.Lucene.Providers.LuceneIndex.get_IndexWriter()

   at Examine.Lucene.Providers.LuceneIndex.GetDocumentCount()

   at Umbraco.Cms.Infrastructure.Examine.ExamineIndexRebuilder.ShouldRebuild(IIndex index)

Steps to reproduce

Whilist we cannot reliably reproduce this currently, our current findings suggest that this usually happens when the IIS app pool is recycled, either naturally when IIS decides to recycle the app pool at the time configured or when we run a deployment of new code and stop and start the app pool ourselves as part of this process.

Expected result / actual result

It would appear that the software is checking the indexes and then choosing not to rebuild the ones its recognised as being incomplete/corrupt/broken.

I would have assumed the opposite would be the default action, if we know its broken why not rebuild to fix it? This of course may be a gap in my understanding of the behaviour, any info you can give on this would be helpful.

My preferred outcome:
When the application restarts and the indexes are checked, if there is any corruption or files the system is expecting are not present the index should rebuild or heal its self to guarantee a complete and functional index. This could also be done via a helthcheck so that the application could self heal during normal activity or if the actual cause of the files being missing happens after the app has launched.

[github-actions]

Hi there @ChrisMIntuitiv!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[ChrisMIntuitiv]

This appears to be related to the issue that has been raised on the Umbraco Forms issue tracker. umbraco/Umbraco.Forms.Issues#1444