New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Exact Substring Deduplication] Analysis #8

Open

ChenghaoMou opened this issue Oct 1, 2022 · 1 comment

Collaborator

ChenghaoMou commented Oct 1, 2022

Near deduplication #7 only operates on file level. It is also possible for a file to be

a substring of another file, while the minhash/simhash fingerprints being wildly different
composed of multiple snippets from different sources

Do we do something about them, knowing they contains large chunks of repeated snippets?

Contributor

lvwerra commented Oct 5, 2022

How hard would it be to do some analysis of how often this is the case maybe on a subset of data?

ChenghaoMou mentioned this issue

Adding alternative minhash script #11

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment