fix: create custom tokenizer

[stearm]

Fixes #492.

Correctly create the tokenizer using the custom one passed via parameters.

Speaking about the linked issue, the provided example seems to be wrong:

const orama = await create({
    schema: {
      foo: "string",
    },
    components: {
      tokenizer: {
        normalizationCache: new Map([["english:foo:Bar", "bar"]]),
        stemming: true,
      },
    },
  });

According to the documentation, a custom tokenizer is made up of at least language, normalizationCache, and the tokenize function (see Tokenizer interface). For the time being, passing a custom normalizationCache is only possible if a custom tokenizer is used.

Passing a custom normalisation cache with default values without using a custom tokenize function could be harmful (or at the very least ineffective), because it would be supplied with specific keys determined by the language inside the native tokenizer.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
orama-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 16, 2023 0:06am

[micheleriva]

LGTM