Large search indexes can cause Lunr to freeze the page while it creates the index.

[sashadt]

When using readthedocs theme, search shows "no results" for a few seconds while search is being performed, and then results are loaded. However, few seconds are enough to convince the users nothing is found, and they move on.

For example see: http://docs.datatorrent.com Try searching for "test" and you will see the following for a while:

Perhaps an indicator of search in progress can be added?

[waylan]

Perhaps an indicator of search in progress can be added?

That seems reasonable to me. Presumably the problem occurs due to the large number of pages. On a smaller site, this would not be a problem as the wait time would be much less.

[sashadt]

We have 50 files with 18,936 lines of Markdown. I don't know if that's considered large, but even for smaller projects running on slower servers it would provide a better user experience. Thanks for looking into it!

[waylan]

It appears that the RTD theme includes the default content "Sorry, page not found." on the search results page (see search.html#L18). However, the search tool will also replace any content with <p>No results found</p> (see search.js#L63) which makes the initial filler redundant. It appears that RTD displays nothing. Should we do the same or display a message to wait for results?

[d0ugal]

This is quite interesting. I assumed the delay was the length of time it takes to download the search index (about 0.5mb). However, there is a long delay after it is downloaded. So is the delay while lunr.js is processing the index? Meaning the slowness is probably more dependant users machines than the server. Looking a the Lunr change log there are a few performance improvements in 0.5.8 and later (we ship 0.5.7). So I think there are two things we should do:

• Add a loading indicator (this could just be static text saying "Searching...") and remove the incorrect text in ReadTheDocs's search.html
• Update Lunr.js to 0.6.0

[d0ugal]

I just cloned and tested with the DataTorrent docs locally, which confirms the slowness is with the JavaScript as downloading is almost instantaneous.

Even worse, my browser (Chrome) is locked up by the JavaScript for a good few seconds, so I can't actually click anything. I done some very rough measurements to try and find what was making it so slow. My timings below are all rounded because I don't have confidence in my measuring.

The loading of the JSON into Lunr is the culprit. This for loop takes around 8(!!) seconds. More specifically it is the call to index.add(doc), so I timed that for each document (the project has 1022, as we add individual markdown documents and page headers individually to improve the search results). Most of them are very fast (less than 50ms) but two are really slow. The Application Developer Guide takes around 1 second consistently and the Release Notes take around 4.5 seconds consistently.

I don't have time to keep looking now, but we need try and determine what makes these so slow. Is it just because these are long pages? Is there something that causes an issue with Lunr? I tested with both Lunr 0.5.7 and 0.6.0 and got similar results. 0.6.0 might actually be a little slower, but I didn't do enough tests to be confident about that.

[waylan]

Even worse, my browser (Chrome) is locked up by the JavaScript for a good few seconds, so I can't actually click anything.

I noticed the same behavior when looking into this as well. Although I didn't big any deeper. The point is that I can confirm this behavior is not isolated.

Add a loading indicator (this could just be static text saying "Searching...") and remove the incorrect text in ReadTheDocs's search.html

I don't see any reason why we can't do this now. I'll push a PR when I get a few minutes unless someone beats me to it.

[d0ugal]

I investigated further. FireFox is much faster than Chrome. I managed to improve the performance under Chrome a bit. See: olivernn/lunr.js#208

[olivernn]

I've pushed @d0ugal's changes in 0.7.0 of lunr, so there should hopefully be some improvements from that change.

Without knowing anything of how mkdocs makes use of lunr, I have a few suggestions that could improve the performance and user experience further:

• Pre-build the index and then load this pre-built index
• Move searching/building over to a webworker

As an example, the example lunr app makes use loading a pre-built index. I think a member of the Angular.js team wrote up how they improved performance using web workers too.