Tokenizer not unserialized properly?

[alexturpin]

Hello,

I'm trying to use a serialized index with a different tokenizer and it's not working for me.

The tokenizer is first created and registered. It gets set on the index and then I add my documents. Once that's done, the index is serialized to JSON. I can confirm by looking at the JSON that my tokenizer's label was serialized.

The problem comes when loading the index subsequently. I ensure that the tokenizer is first registered before loading the serialized index. The code in lunr.Index.load then uses lunr.tokenizer.load to load the serialized tokenizer. I've confirmed that this works with logging. The problem comes when it assigns the newly loaded tokenizer to idx.tokenizer.

idx.tokenizer = lunr.tokenizer.load(serialisedData.tokenizer)

That's the function used to set the tokenizer for an index. The actual tokenizer eventually resides in tokenizerFn. I believe the code to unseralize a tokenizer when loading an index should instead be:

idx.tokenizer = lunr.tokenizer.load(serialisedData.tokenizer)

That works on my end, and I will submit a PR shortly with that fix.

Cheers

[olivernn]

Fixed in 0.7.2