Duometer allows to efficiently identify near-duplicate pairs of documents in large collections of texts. It is written in Scala and implements a MinHash algorithm.
For example, to extract text from all files in ~/text-files and identify
those that have similar content, run:
./duometer -i ~/text-files -o text-files.duplicatesFor more information about how to use duometer see this tutorial.
- Efficiently finds pairs of documents that contain similar text.
- Automatically extracts text from files in a huge number of different formats (including HTML, PDF, Microsoft Office document formats and the OpenDocument format)
- Works well with very large collections of documents.
- Makes use of multiple CPU cores.
- The default settings should work well in most cases but you can
customize the duplicate detection for your purposes (
run
./duometer --helpfor a full set of options).
The only prerequisite is a Java runtime.
- Download the current version of the tool here.
- Extract the archive and go to
./bin. - Run
./duometer(on Linux and Mac) orduometer.bat(on Windows).
Download a package and run:
sudo dpkg -i duometer_0.1.3_all.debduometer is now installed and should be available as a shell command.
Duometer uses sbt-native-packager.
You can build the tool from source by running the dist command in sbt
to create a .zip archive that can be run on any machine with Java installed.
Debian binary package can be created by executing debian:packageBin.
For background information about the algoritm see a relevant chapter in Manning and Schütze (2008) or read the original Broder (1997) paper.
Duometer uses Apache Tika to extract text from a huge number of different file types. For the full list of supported formats see here.
- Issue Tracker: https://github.com/pmandera/duometer/issues
- Source Code: https://github.com/pmandera/duometer
The tool was developed at Center for Reading Research, Ghent University by Paweł Mandera.
The project is licensed under the Apache License 2.0.
Broder, A. Z. (1997). On the resemblance and containment of documents. In Compression and Complexity of Sequences 1997. Proceedings (pp. 21–29). IEEE.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge University Press.