Bug: tolerance option not behaving as hoped

[ttillberg]

Thanks for the amazing lib and clear documentation! I'm looking at using Orama to search local chat messages (typically involving a few words up to several sentences).

Using @orama/orama ^1.2.3 I'm getting fast a correct results for exact and prefixed matching however however typos don't seem to work the way I was hoping. I'm probably missing the obvious but testing the tolerance parameter against an example in the docs returns poor results. So I'm wondering what could be wrong.

Looking at the following example.
https://docs.oramasearch.com/usage/search/introduction#typo-tolerance

If I grab a slightly bigger database:
https://github.com/erik-sytnyk/movies-list/blob/master/db.json

{ 
  term: "Christopher Nolan", 
  properties: ["director"] 
}

// result: OK: matches 1 exact result like expected

{
  term: "Cris",
  properties: ["director"],
}

// result: OK: matches 1 document "Michael Cristofer" (no tolerance was set, so this is kind of expected)

{
  term: 'Cris',
  properties: ['director'],
  tolerance: 1,
}

// result: "fails": matches 0 documents, in the documentation this query would return all "Chris's" - not this would still fail bumping the tolerance level
// one example in the DB: "director": "Pierre Coffin, Chris Renaud",

here's my playground (all output is in the console):
https://codesandbox.io/p/sandbox/keen-knuth-9wql22?file=/src/main.ts:65,28

I've played with other options, such as the tokenizer, stemming, relevance, threshold but without luck. What am I missing?

[micheleriva]

I fear that's a known issue. We're performing the Levenshtein edit distance on words living in the same prefix bucket, rather than performing the edit distance calculation on trees. For instance, searching for Chris and hris will give you totally different results, as they don't share a common prefix.

I'll be putting a bounty on this bug, thanks for opening it!

/bounty 500

[algora-pbc]

💎 $500 bounty created by micheleriva
🙋 If you start working on this, comment /attempt #480 to notify everyone
👉 To claim this bounty, submit a pull request that includes the text /claim #480 somewhere in its body
📝 Before proceeding, please make sure you can receive payouts in your country
💵 Payment arrives in your account 2-5 days after the bounty is rewarded
💯 You keep 100% of the bounty award
🙏 Thank you for contributing to oramasearch/orama!

Attempt	Started (GMT+0)	Solution
🟢 @mnmt7	Sep 15, 2023, 5:50:54 AM	WIP
🟢 @bicky21	Sep 18, 2023, 2:53:27 PM	WIP
🟢 @melsonic	Oct 2, 2023, 4:51:02 PM	WIP
🟢 @SP321	Oct 10, 2023, 12:23:49 PM	#516

[mnmt7]

Hey @micheleriva, I would like to work on this issue. Can you please assign this issue to me?
/attempt #480

Options

• Cancel my attempt

[bicky21]

Hey, I have a solution
/attempt #480

Options

• Cancel my attempt

[algora-pbc]

Note: The user @mnmt7 is already attempting to complete issue #480 and claim the bounty. If you attempt to complete the same issue, there is a chance that @mnmt7 will complete the issue first, and be awarded the bounty. We recommend discussing with @mnmt7 and potentially collaborating on the same solution versus creating an alternate solution.

[ogil7190]

@ttillberg is this open to work?

[micheleriva]

@ogil7190 yes

[melsonic]

/attempt #480

Options

• Cancel my attempt

[SP321]

/attempt #480

Options

• Cancel my attempt

[algora-pbc]

💡 @SP321 submitted a pull request that claims the bounty. You can visit your org dashboard to reward.

[algora-pbc]

🎉🎈 @SP321 has been awarded $500! 🎈🎊

[micheleriva]

Fixed with v1.2.11