[local search] indexing too slow to be usable for large sites

[zhangyx1998]

Describe the bug

I am trying to convert an ancient documentation site into vitepress. The volume is very large (a lot of of pages, and many of them has very long content).

I waited for more than 4 hours to build my site with local search enabled. I would have to wait for the same amount of time to launch a dev server.

Unfortunately, although the build finished without any error, the rendered search index was empty. This is caused by another problem: the local search plugin use one regex to find headers, and another regex to remove tags. Which works only if the html is directly parsed from markdown, and is very likely to fail upon embedded html (or vue template).

In addition, if you've looked at the outputs from PR #3374, you might have already noticed that each file was indexed twice for each build. That doubles the wait time.

Reproduction

  vitepress v1.0.0-rc.32

⠹ building client + server bundles...
(!) Some chunks are larger than 500 kB after minification. Consider:
 - Using dynamic import() to code-split the application
 - Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/configuration-options/#output-manualchunks
 - Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.
/Users/Yuxuan/Lab/xorg-doc/docs/index.md
✓ building client + server bundles...
✓ rendering pages...
- build complete in 16008.00s.

16,008 seconds = 4.45 hours

Expected behavior

1. Index faster.

2. No double indexing for same source file.

3. When splitting page into sections, match headers more robustly.

System Info

Not relevant

Additional context

Solution to improve performance provided here: #3374.

I want to discuss with a maintainer before proceeding to port the improved section-splitter implementation.

Also, I can provide a fix to the following issue after some discussion with maintainers:

1. On dev server, only provide the search-index when the search box is focused (expanded), and show a loading indicator in the search dropdown before indexing completes. In this way the indexing process will not block dev preview.

2. Fix the double indexing problem so large sites will build faster.

Validations

• Check if you're on the latest VitePress version.
• Follow our Code of Conduct
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.