Duke is a fast and flexible deduplication engine written in Java
Java HTML
Latest commit dda6390 Jun 26, 2016 Lars Marius Garshol Disallow more than one ID property in same config
Permalink
Failed to load latest commit information.
doc Made a start on integration tests. Feb 16, 2014
duke-core Disallow more than one ID property in same config Jun 26, 2016
duke-dist dist: include elasticsearch module Mar 19, 2015
duke-es gitignore cleanup Mar 19, 2015
duke-json
duke-lucene
duke-mapdb gitignore cleanup Mar 19, 2015
duke-mongodb minor: moved mongo README in module folder Apr 20, 2015
duke-server gitignore cleanup Mar 19, 2015
.gitignore gitignore cleanup Mar 19, 2015
.travis.yml Added unit test, made Travis build with JDK 1.8, too. May 28, 2015
LICENSE.md Added a license file, as suggested by Kendall Clark. Feb 15, 2014
README.md Added a note on how to build the sources Aug 26, 2015
changes.txt Style fixes plus disable intermittently failing test Oct 7, 2014
pom.xml Now builds correctly on both JDK 1.7 and 1.8 Aug 29, 2015

README.md

Duke

Duke is a fast and flexible deduplication (or entity resolution, or record linkage) engine written in Java on top of Lucene. The latest version is 1.2 (see ReleaseNotes).

Duke can find duplicate customer records, or other kinds of records in your database. Or you can use it to connect records in one data set with other records representing the same thing in another data set. Duke has sophisticated comparators that can handle spelling differences, numbers, geopositions, and more. Using a probabilistic model Duke can handle noisy data with good accuracy.

Features

  • High performance.
  • Highly configurable.
  • Support for CSV, JDBC, SPARQL, NTriples, and JSON.
  • Many built-in comparators.
  • Plug in your own data sources, comparators, and cleaners.
  • Genetic algorithm for automatically tuning configurations.
  • Command-line client for getting started.
  • API for embedding into any kind of application.
  • Support for batch processing and continuous processing.
  • Can maintain database of links found via JNDI/JDBC.
  • Can run in multiple threads.

The GettingStarted page explains how to get started and has links to further documentation. The examples of use page lists real examples of using Duke, complete with data and configurations. This presentation has more of the big picture and background.

Contributions, whether issue reports or patches, are very much welcome. Please fork the repository and make pull requests.

Supports Java 1.7 and 1.8.

Build status

If you have questions or problems, please register an issue in the issue tracker, or post to the the mailing list. If you don't want to join the list you can always write to me at larsga [a] garshol.priv.no, too.

Using Duke with Maven

Duke is hosted in Maven Central, so if you want to use Duke it's as easy as including the following in your pom file:

<dependency>
  <groupId>no.priv.garshol.duke</groupId>
  <artifactId>duke</artifactId>
  <version>1.2</version>
</dependency>

Building the source

If you have Maven installed, this is as easy as giving the command mvn package in the root directory. This will produce a .jar file in the target/ subdirectory of each module.

Older documentation

This blog post describes the basic approach taken to match records. It does not deal with the Lucene-based lookup, but describes an early, slow O(n^2) prototype. This early presentation describes the ideas behind the engine and the intended architecture