Search: whole phrase like "a-b.log"

[digihernandez]

i want to search the whole phrase or word in the search bar like "a-b.log". I have added in the plugins:

plugins:

• search:
  separator: '[\s-,:!=[]()"/]+|(?!\b)(?=[A-Z][a-z])|.(?!\d)|&[lg]t;'

but unfortunately, the search bar didn't give me the result. Please help me. Thanks.

[kamilkrzyskow]

Hello @digihernandez,
please be aware that lunr.js has issues with that sort of thing and therefore the new search implementation is actively being developed.
For further updates follow: #6307

As for the issue at hand, I'm no expert in lunr.js, but will explain it how I understand it.
The separator is used to split up words in the tokenizer:
https://lunrjs.com/docs/tokenizer.js.html

The separator proposed on the documentation is [\s\-,:!=\[\]()"/]+|(?!\b)(?=[A-Z][a-z])|\.(?!\d)|&[lg]t;:
https://squidfunk.github.io/mkdocs-material/plugins/search/#config.separator

You can check any regex compiler site to see how it will behave for your content. I used https://regexr.com/7t62l for example.

Each blue selection is a regex match, and in this context means that it will split the text in this point excluding the matched character.

As you can see on the image, "a-b.log" will be split up into a, b, log.
Modifying the regex to this [\s,:!=\[\]()"/]+|(?!\b)(?=[A-Z][a-z])|\.(?!\d|\w)|&[lg]t;, should result in a-b.log not being split, but the " are still removed. However, as seen on the image also gskinner.com won't be split, and before it was split into gskinner, com

Note I didn't test it out with MkDocs, just displayed my understanding, trial and error is still up to you ✌️

[kamilkrzyskow]

I had some time and tested it manually, and the provided regex did split the words accordingly. However, searching for them doesn't work, because of the - inside. Looking at the code, I can't pinpoint the issue.
https://github.com/squidfunk/mkdocs-material/blob/master/src/templates/assets/javascripts/integrations/search/query/transform/index.ts

9.5.13-separator.zip
Here is a minimal reproduction with the modified separator, I also noticed that a configured empty pipeline does still result with "stopWordFilter" inside. I temporarily modified source code of the plugin to leave it empty, and tested that no matter the setting of the pipeline the search still doesn't find the file name.

---

Any comments @squidfunk? The behaviour of firstpart- is rather odd as it suddenly matches every word (except for the stopWordFilter words), and firstpart-firstpart proves that - kind of removes the previous word from being a part of the string hmm. When checking the code I found a wildcard mention, and when I tried it, it sort of works with a*b.log, as it shows the result, but also shows missing.

[squidfunk]

1. Correct, the pipeline setting had a bug. When it was falsey, the default was used, which led to the stopWordFilter being added. I just fixed this in f325238.

2. I debugged this for 20 minutes: when searching for firstpart-, a wildcard is being added at the end. This is one of the hacks (or "transforms") that we employ to try to implement typeahead with Lunr.js. Lunr.js parses firstpart- with the given separator into two tokens, namely "firstpart*" and "*", which explains why you get everything back. I also just discovered, that the QueryLexer uses a separate tokenizer, i.e., Lunr's default tokenizer. We override the tokenizer when initializing the index, but until now we did not do that for the QueryLexer, as I was not aware that it's a different one. Since the QueryLexer is initialized before we override the separator, it keeps the default one, explaining the behavior. Lunr's tokenizer has serious shortcomings (i.e., does only allow for single-character splitting, albeit you can provide any RegExp), and I'm not sure what would happen if we would patch the QueryLexer to use our tokenizer as well.

3. a*b.log triggers an infix wildcard search, which explains why you can find it. However, when we try to analyze the query and associate the terms, we can't find it due to a*b not matching with a-b.

I think it's getting pretty clear what the motivation for #6307 is – we've reached the end of monkey-patching economics. Lunr.js and basically no other existing JavaScript search engine is meant for client-side typeahead on structured data (HTML + metadata).

Thus, I currently deem the non-pipeline issues as not feasible to fix. Please wait for #6307.

[squidfunk]

... additionally, sadly, we're still waiting for more feedback for #6372, which might already fix the issue. If not, that's valuable feedback for us to move problems on our radar. So many people are complaining about the search, and for good reason as we confirm, but almost nobody tested the research preview.

[kamilkrzyskow]

Thanks for your time, I had a hunch that the separator somehow breaks the input on - and I thought the default one is being used. However looking at your explanation, the issue grew quite complex, so I doubt I would figure it out. I tried to comment out some of the transoform logic bit by bit, but it either didn't find anything or kept breaking on the -.

From the looks of it the search plugin in #6372 is still the same?
So perhaps instead of a preview that requires to be installed in a special way, perhaps ship it as preview-search.23af12.min.js and add a toggle in the search plugin option preview_version: True that you will later deprecate haha this would make it way easier for people to try it out. EDIT: Compare it to the AI-bot in search, people could use it without opt-in, and that lead to a lot of feedback (and questions for making it a feature 😆 )

I'll admit, I didn't try the new search implementation too, as the Gothic Modding Community is quite small and is used to much worse setups, so no one complained yet 😅Basically the current abilities of Lunr.js (modded by your theme), where one can search for a function name with _ in text or code blocks is enough. Some people suggested to display the description of the page if set in the front-matter to be shown in the result window, but I believe this could be achieved with hooks / JavaScript, but haven't gotten around to it.

The new search engine helps mainly with searching for substrings within bigger words, and better ranking of results.
This will of course improve the search experience, I just wanted to say that our community isn't in desperate need for better serach 😅

[squidfunk]

From the looks of it the search plugin in #6372 is still the same?

I checked it – the preview cuts hard after two paragraphs (so it doesn't print the entire section), because there's no summary logic right now. If you move "a-b.log" in the Markdown document to the top, matching at least works. Summarization is another topic.

So perhaps instead of a preview that requires to be installed in a special way, perhaps ship it as preview-search.23af12.min.js and add a toggle in the search plugin option preview_version: True that you will later deprecate haha this would make it way easier for people to try it out.

It boils down to one CLI command (pip install ...) – I'm not sure how I can make it any easier to try it out without actually shipping it to production. It is very much not intended for production, as some parts are still missing as can be read in the OP of the preview. The whole point of the preview is to not make users to include it into their sites to deploy it to production.

Compare it to the AI-bot in search, people could use it, and that lead to a lot of feedback (and questions for making it a feature 😆 )

Yes, we had dozens of requests that users want to install it on their site, albeit it being a feature that we only test drove on our very own site with mixed success. You could not try out the bot on your own docs, but you can with the search preview. I'm not sure I understand why that doesn't spark interest, given the many complaints about search and open issues, but I guess we have to force users into using it by shipping it to production. At least then, we'll get feedback 😅

Some people suggested to display the description of the page if set in the front-matter to be shown in the result window, but I believe this could be achieved with hooks / JavaScript, but haven't gotten around to it.

Yes, you should be able to replace text with the meta.description of a page in search_index.json.

The new search engine helps mainly with searching for substrings within bigger words, and better ranking of results.
This will of course improve the search experience, I just wanted to say that our community isn't in desperate need for better serach 😅

Good to hear that even the current implementation works for you! However, I think it's quite easy to understand that we'll gravitate towards much better solutions with more feedback – see the latest developments in #6704 ☺️ That's all of a concern I want to raise. Test it, break it, report it, so we can iterate 💪

[digihernandez]

no. it's not URL. it's a file.

…

On Sat, Mar 9, 2024 at 9:15 AM Martin Donath ***@***.***> wrote: Possibly related (although not sure if a-b.log is an URL or not, but it's similar): - #5936 <#5936> — Reply to this email directly, view it on GitHub <#6885 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A662QRVEHDMXHGHP5DP3WTTYXJPCXAVCNFSM6AAAAABEMSJ36GVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DOMRWGEZTC> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[squidfunk]

It is yet related to the reported issue ☺️ You can follow #6307, where we're improving the search experience.