Upgrade to lunr 2.x

[waylan]

At some point recently, lunr 2.0 was released. A summary of changes are here: https://lunrjs.com/guides/upgrading.html

More importantly, there is a page on Pre-Building Indexes which includes an example script. Unfortunately, I still don't see a spec (of the index format) so that a Python lib could reliably build the index.

In any event, when search is refactored, we should use lunr 2.x.

[olivernn]

I still don't see a spec (of the index format) so that a Python lib could reliably build the index.

There is a JSON schema describing the format of a serialised index. The repository is marked as WIP since I really want to get some feedback from implementors before committing to it. That said it is pretty stable, the only open questions I have are on how to version the schema.

I've also started working on a Rust library for generating an index, it might prove useful in getting started with a Python implementation. Of course, if anyone is interested on taking on that work (lunr.py) then I'm more than happy to answer questions and offer guidance, I'm not a Python programmer though so its unlikely I'll be able to work on an implementation.

[squidfunk]

When you implement the pre-building - please, please, please make it optional. Material builds the index in the browser in a very specific way that is incompatible with a pre-built index.

[waylan]

@squidfunk when I played with this some time ago, my proof of concept pre-built the index but also provided the current JSON file. Your theme could ignore the pre-build index although given the huge improvement it makes, I can't imagine why you would want to. Seems like it would be worth it to make you theme work with the pre-built index. Of course, any such change will come with a complete refactor of the JavaScript provided by the plugin, so perhaps you will no longer even feel the need to provide your own at that point (that is one of the goals I have with the refactor).

[squidfunk]

Well, I do some post-processing on the search_index.json, cleaning some stuff up. Maybe this could be done afterward with a pre-built index, but one would have to check.

[waylan]

I'm not so sure we want to upgrade to Lunr 2.x. Being able to "boost" a search term is nice, but we need to always have the title field boosted, regardless of what search terms are used. As described in olivernn/lunr.js#312, because we cannot "boost" the title field in 2.x, the search results are quite different (less relevant) than with 1.x.

[squidfunk]

Material already uses lunr 2.x with very good results. You can take a look at the setup here: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/components/Material/Search/Result.jsx#L172

[waylan]

@squidfunk my point is that version 2 ignores the { boost: 10} on L189. In any event, if you aren't noticing a problem with the results returned, that's good to hear. Thanks.

Btw, how are you passing user settings from Python to Lunr (such as lang, ect)? Are you rendering then as JS variables in the template or some other way? I was thinking of including plugin specific settings in the JSON file. By way of explanation, the JSON currently is structured like this:

{ 'docs': [...] }

So I was thinking of adding a second key, config, like this:

{ 'docs': [...], 'config': { 'key1': 'value1', 'key2': 'value2'} }

Of course, you could ignore config and continue to use your current method. Any thoughts.

[squidfunk]

I'm passing them as a configuration in the base.html, see this code section

EDIT: ... and for everything locale/translation related, it's mostly meta tags, see, which are then read like this

EDIT2: the reason why I'm doing this is that the translations for Material are done within the templates, so it's the only way to go. Some values (like extra.search.tokenizer) can be set within mkdocs.yml, but the translations provide default values which can be overridden, see language.html

[yeraydiazdiaz]

@waylan lunr.js 2.x moved the boost to the search according to upgrading docs

so theoretically in the search call we could do something like idx.search('title:' + query + '^10') to match the original search

[waylan]

so theoretically in the search call we could do something like idx.search('title:' + query + '^10') to match the original search

AFAICT, that makes things worse. If query = 'foo', then only results with "foo" in the title are returned. Any items with "foo" only in a second field (body in this case) are not included in the results at all. But if query = 'foo bar', then the same behavior happens for "bar" (results with "bar" only in the body are excluded) but "foo" seems to get normal treatment in that I get results containing "foo" in the title and/or the body with results containing "bar" in the title being boosted (listed first). At least that's my impression based on my observation of a few unscientific tests.