Search: Research Preview 2 🧪

[squidfunk]

Important

This PR presents an early preview of our new search feature for gathering user feedback. It is not yet ready for production and is subject to modifications. Currently, it lacks support for search suggestions and does not cater to Japanese, due to the absence of segmentation functionality.

Overview of this PR

This PR offers the second iteration of our revamped search functionality. While still in development and devoid of UI changes (for now), it marks a departure from the current implementation. It builds upon the previous research preview, forming an intersection instead of a union, and employs a new, experimental ranking method. We are eager to know whether you find this change enhances or diminishes the search results.

The ranking function is a variant of BM-25 that takes the proximity of matched terms into account. Our testing shows, that typeahead results should be much more stable, and results should be much more sensical. For example:

Lunr.js (current master)

This PR:

As noted, the new ranking function is still experimental, and we have several ideas to make it even more concise. It should, however, be already better than what we currently have, as it can now find consecutive words more easily.

Current limitations

• No Chinese and Japanese segmentation
• No stemming
• No special syntax, e.g. "foo bar" or +foo -bar
• No boosting

All of those will be resolved before we release the new search.

Your feedback is essential 🙋‍♀️

We want to co-create the search feature with our community. Our goal is to deliver an unparalleled search experience to all Material for MkDocs users, not just our sponsors. This journey begins with a revamped interface and, more critically, heightened accuracy to help users find what they need quickly and effortlessly. After establishing a solid foundation, we will explore many exciting extensions, which, as always, will first be released exclusively to our sponsors. But first, we aim to elevate the experience for everyone.

To test this version featured in the PR, you can install it using pip:

pip install git+https://github.com/squidfunk/mkdocs-material.git@spike/search-preview-2

We warmly invite you to share your honest feedback on this PR! Please note that this PR is specifically designed for feedback collection and will not be merged. Your insights are invaluable to us! ☺️

[strausmann]

I just wanted to test Preview 2 on our project. Unfortunately, I get this error message.

ERROR - Config value 'plugins': The "meta" plugin is not installed

The branch will support all other features or not?

[squidfunk]

It appears you're trying to install it on an Insiders project. Since this is the community edition, it is currently not compatible with Insiders projects. Thus, you'd have to disable the plugins that are only available in Insiders. Once we have a stable version (and are beyond mere research previews), we'll release an alpha version on both editions.

[MaximilianKohler]

Looks great! Thank you!

[pawamoy]

I did not test extensively, but it looks like the results are better with the new search.

• Some entries that were previously displayed too low (8th position) are now appearing higher (3rd position). It seems results for searches with multiple words are better than before.
• Most results are similar to the previous version, with the ordering just a bit different (first and second results switched).
• Some searches would previously yield lots of pages with "Missing: words" before a good result. Now those results are not shown and only the matching page is shown 👍

Sorry if my phrasing does not make sense, let me know if you'd like some screenshots or anything else 🙂

[karengermond]

I tested this and am wondering if there will be a mechanism for boosting topics/sections or some other method like that for controlling search results? I'm finding that the new search isn't performing better than lunr because I made extensive use of boosting which worked well in many cases.

I do like that multi-term search strings are not showing up as "Missing". That's a great improvement.

[squidfunk]

Thanks for the great feedback @karengermond. We're working on an entirely new way of interacting with search. The current research previews are only meant to test if we broke some case that was previously possible but doesn't work anymore. This doesn't seem to be the case until now, except for that we still need to add support for segmentation and query syntax. I'd be curious to learn how exactly you're imagining this to work, i.e., if you could spare some time and enumerate some use cases that you feel don't work well right now, but are important for your company.

Regarding boosting: this will be possible of course, same mechanism as currently, but boosting rarely leads to good results and it's too hard to find a good balance, especially when you're looking at a large corpus. Therefore, we're trying to improve the situation in a way that none of this is necessary anymore.

[karengermond]

pip install git+https://github.com/squidfunk/mkdocs-material.git@spike/search-preview-2

We just have nonstandard data (proprietary so I can't make it available). Here's an example: we have a language construct "on" which is used in a heading H2. But the term "on" is used in a whole bunch of other headings as H1s. All of those hits come up first in the new search and the old search. That's a really tough one so I'd need to do something to boost the section.

I have a question -- how do I go back to the old version of mkdocs with lunr?

[squidfunk]

I have a question -- how do I go back to the old version of mkdocs with lunr?

You can just reinstall the latest version with:

pip install mkdocs-material

[karengermond]

Thanks!

…

On Mon, Dec 4, 2023 at 9:32 AM Martin Donath ***@***.***> wrote: I have a question -- how do I go back to the old version of mkdocs with lunr? You can just reinstall the latest version with: pip install mkdocs-material — Reply to this email directly, view it on GitHub <#6372 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2OIWP57H5C6EXIBWFAKG2TYHXNITAVCNFSM6AAAAAA7S6XZPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZYG42TSMZQGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[day-jeff]

Search is dramatically improved! Nice work.

I want to deploy to our internal production site ASAP. Any idea when you might integrate into the main version? (Since search is important for us, and most of the recent updates are not, I'm tempted to deploy the test version.)

[squidfunk]

I would not encourage you to deploy this version. This is only a research preview and we don't offer support for bugs that you discover when deploying it into production. Please see #6307 (comment) for when search will be available. You can probably imagine that I'm being asked that every other day 😅

[squidfunk]

Closing. Next search preview will be ready soon ☺️