Question: EnContactMatcher performance

[acoroleu-tempus]

Good morning,

While trying to use EnContactMatcher.ts with 10k contacts the constructor takes up to 1 min. By changing inner-matchers to non accelerated, it improves considerably (it goes down to 'only' 30 seconds).

I have a few questions:

1. Are those times reasonable/expected with 10k contacts?
2. Would it be ok to open a PR adding a new configuration flag for using or not accelerated matchers in EnContactMatcher? Another option would be passing in the EnContactMatcher constructor the fuzzy matcher class to use? Which one would you prefer?
3. With 10k contacts, there's a very small difference on the performance for find method call (0.1 seconds with accelerated vs 0.3 seconds without). Given the find time is almost negligible compared to the constructor time for accelerated vs non accelerated , we chose to use non accelerated fuzzy matchers. Does this make sense?

Thanks and congratulations for this amazing library :)

[Mmdixon]

Hey @acoroleu-tempus, thanks for trying out the library. I noticed some similar and some different behavior to what you described.

1. I've used it for typically ~200 contacts (constructor takes <1s), maybe <1000 contacts. I tried it with 10k and it took 55s, so similar to you
2. The EnContactMatcher is a fairly opinionated matcher. It has an AcceleratedFuzzyMatcher that uses the EnHybridDistance for both the contact's names and aliases that is processed with the WhitespaceTokenizer and EnPreProcessor that does a kind of a windowing on the name to add variations. So if you'd like to deviate from it you should have all the components to build a matcher to do what you want to do.
3. constructor time I agree with, switching from the internal AcceleratedFuzzyMatcher to FuzzyMatcher (55s->25s similar to you). But find time was fairly drastic (10ms to 192ms), a ~2x start cost for a ~19x query reduction gain. the "0.1 seconds with accelerated" seems a bit high in your case? If we were talking about a smaller number of contacts, like 200 instead of 10k, the comparison is a lot closer (425ms build 2ms find vs 301ms build 4ms find, respectively)

Because of a noticeable difference in (3), and how the performance gap gets bigger with more contacts and how EnContactMatcher may expand the contact list from the original size with the windowing it does (thus a bigger contact list), I do not recommend (2).

[acoroleu-tempus]

Hi @Mmdixon, the current preprocessing of EnContactMatcher looks what we need in order to increase accuracy results.

Would it be possible to make FuzzyMatcher serializable? Given the constructor takes some time, when being passed a large number of contacts, we'd like to serialize and store in Redis so it can be re-used (deserialized).

Thanks for your support!