Upgrade to support lunr 2.x #30
Conversation
Lunr 2.x has changed the interface for pipeline functions, instead of just passing around strings, tokens are now represented as an instance of lunr.Token. Additionally, Lunr now supports per index tokenisers, this allows the JP plugin to not have to clobber the global lunr.tokenizer function. This is fine for when building the index, but queries are parsed by lunr.QueryParser which may need some help when it comes to parsing Japanese searches. Currently there is no extension point for the query parser so we may need to do some work here. This commit also includes updates for the demos.
|
I've tried it with the french language and it works good. |
|
Thanks @olivernn for this PR. It helped me understand the upcoming changes in Lunr 2. So, I took the insight from here and overhauled Lunr Languages to be compatible with all Lunr versions ( This will help users, since no matter what Lunr version they will use, they'll just to have to make sure they use the latest Lunr Languages version. In order to enforce this, I added integration tests that test the combination between Lunr versions In this way, we'll achieve two things:
I will close this MR now. Here's the commit in which these changes were made: 4c64ac6 . The key changes were made in:
Should you have any questions, please comment on the commit linked above, or let's talk in Gitter. @iDams , you can now use it :). |
|
💯 Nice work @MihaiValentin! |
This commit adds support for the upcoming release of lunr 2.x. This has not been released yet so its probably best off waiting to merge this until I do that release. Ideally I'd like to be able to have the same great language support in the new Lunr from day one though, so getting this out here now to get feedback.
Lunr has changed the interface for pipeline functions. Before, tokens were strings passed to pipeline functions, Lunr 2.x changes this, wrapping them in a
lunr.Tokenobject. This means that all pipeline functions that expect to be working with a string need to be updated to work with alunr.Token.This change covers most of the language plugins in this repository. The Japanese plugin required a few more changes, specifically to make use of the new, per index, tokeniser. This should allow a Japanese and non-Japanese indexes to coexist.
There is one potential issue though, searches are now parsed by
lunr.QueryParserwhich expects terms to be whitespace separated. I don't know enough (none) Japanese to get from the demos if this is an issue or not, perhaps someone can lend a hand here.I have not changed any of the versions etc, I don't know what you want to do here. I'd like to suggest still keeping support for the 0.x and 1.x branches of Lunr, as well as the new interfaces in Lunr 2.x. Perhaps lune-languages could have a similar versioning scheme, to indicate which major version of Lunr is supported. I'm open to ideas here though.