[FEATURE REQUEST] Adding simhash to --filter-similar-to

[Luoooio]

I wondered why --filter-similar-to didn't work well until I read the article and realized that it uses SSDeep, which does a terrible job of handling text with little content.

SSDeep was selected as it does a good job of identifying near-duplicate pages once content-length reaches a certain size, while remaining performant. Other algorithms were tested but resulted in huge performance hits (orders of magnitude slower on requests/second).

But this is a necessary test, especially for some API endpoint that returns a small amount of information. so why not add simhash for it additionally ? I find it work well when dealing with short texts. I'm expecting such an update.

[epi052]

Good morning, and thanks for the suggestion!

I haven't heard of, nor tried simhash. When I get some time, I'll experiment with it and see how it goes. I'm definitely not opposed to a better algo, if it works out that way.

Thanks again!

[epi052]

@all-contributors add @Luoooio for ideas

[allcontributors]

@epi052

I've put up a pull request to add @Luoooio! 🎉