Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Experimental Go version of superfastmatch
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
api
client
document
fixtures
posting
queue
registry
scripts
sparsetable
testutils
.gitignore
Makefile
Readme.md
feeds.json
superfastmatch.go

Readme.md

Introduction

Superfastmatch is a tool for bulk comparison of text. It enables you to discover the longest common substrings between one document and millions of others.

Requirements

One of:

  • 64 bit Linux
  • 64 bit Windows
  • 64 bit OS-X

and a working install of Mongo DB 2.2 or higher. Details here

Installation from source

Go 1.1 is required to build (installation instructions here). Also required are git, bzr and mercurial to enable the dependencies to be checked out.

mkdir gopath
cd gopath
export GOPATH=`pwd`
go get github.com/donovanhide/superfastmatch
./bin/superfastmatch -logtostderr

Example usage

Compare the works of Dickens for Biblical influence

Start superfastmatch in one terminal:

$ ./superfastmatch -logtostderr
2013/01/27 22:58:56 Started in standalone mode with Hash Width: 24 and Window Size: 30
2013/01/27 22:58:56 Starting Posting Server on: 127.0.0.1:8090
2013/01/27 22:58:56 Starting Queue Processor
2013/01/27 22:58:56 Starting Posting Server on: 127.0.0.1:8091
2013/01/27 22:58:56 Starting API server on: 127.0.0.1:8080
127.0.0.1:8090:2013/01/27 22:58:56 Initialising Posting Server with Window Size: 30 Hash Width: 24 Size: 8388608 Offset: 0
127.0.0.1:8090:2013/01/27 22:58:56 Posting Server Initialised with 0 documents in 0.01 secs Average: 0.00 secs/doc
127.0.0.1:8091:2013/01/27 22:58:56 Initialising Posting Server with Window Size: 30 Hash Width: 24 Size: 8388608 Offset: 8388608
127.0.0.1:8091:2013/01/27 22:58:56 Posting Server Initialised with 0 documents in 0.00 secs Average: 0.00 secs/doc

In another load some books by Charles Dickens:

$ ./superfastmatch client add fixtures/gutenberg/Dickens.tar.gz 
Entering:   Dickens.tar.gz
Command: Add Document Target: (1,1)
Added:  Doctype:   1    Docid:   1  Title: bleak_house.txt  QueueId: 5105b40e7b1d26385300000d
Command: Add Document Target: (1,2)
Added:  Doctype:   1    Docid:   2  Title: christmas_carol.txt  QueueId: 5105b40e7b1d26385300000e
Command: Add Document Target: (1,3)
Added:  Doctype:   1    Docid:   3  Title: david_copperfield.txt    QueueId: 5105b40e7b1d26385300000f
Command: Add Document Target: (1,4)
Added:  Doctype:   1    Docid:   4  Title: great_expectations.txt   QueueId: 5105b40e7b1d263853000010
Command: Add Document Target: (1,5)
Added:  Doctype:   1    Docid:   5  Title: hard_times.txt   QueueId: 5105b40e7b1d263853000011
Command: Add Document Target: (1,6)
Added:  Doctype:   1    Docid:   6  Title: little_dorrit.txt    QueueId: 5105b40f7b1d263853000012
Command: Add Document Target: (1,7)
Added:  Doctype:   1    Docid:   7  Title: old_curiosity_shop.txt   QueueId: 5105b40f7b1d263853000013
Command: Add Document Target: (1,8)
Added:  Doctype:   1    Docid:   8  Title: oliver_twist.txt QueueId: 5105b40f7b1d263853000014
Command: Add Document Target: (1,9)
Added:  Doctype:   1    Docid:   9  Title: pickwick_papers.txt  QueueId: 5105b40f7b1d263853000015
Command: Add Document Target: (1,10)
Added:  Doctype:   1    Docid:  10  Title: tale_of_two_cities.txt   QueueId: 5105b40f7b1d263853000016
Successes: 10    Failures:0

And then search for the bible in this collection of documents:

$ ./superfastmatch client search fixtures/gutenberg/bible.txt.gz 
Doc:bleak_house.txt (1,1)   Left:  3728816  Right:    48538 Length:       67    Text: He that is without sin among you, let him first cast a st...
Doc:bleak_house.txt (1,1)   Left:  3728739  Right:    48461 Length:       55    Text: So when they continued asking him, he lifted up himself
Doc:bleak_house.txt (1,1)   Left:  3532111  Right:    48773 Length:       41    Text: Lest coming suddenly he find you sleeping
Doc:bleak_house.txt (1,1)   Left:  2299962  Right:   561545 Length:       40    Text: enter not into judgment with thy servant
Doc:bleak_house.txt (1,1)   Left:  3608991  Right:  1528396 Length:       34    Text: the labourer is worthy of his hire
Doc:bleak_house.txt (1,1)   Left:    68502  Right:   978540 Length:       33    Text: e up early in the morning, and to
Doc:bleak_house.txt (1,1)   Left:   105106  Right:   978540 Length:       33    Text: e up early in the morning, and to
Doc:bleak_house.txt (1,1)   Left:  1460766  Right:   978540 Length:       32    Text: e up early in the morning, and t
Doc:bleak_house.txt (1,1)   Left:    12745  Right:     7100 Length:       31    Text: from the face of the earth; and
Doc:bleak_house.txt (1,1)   Left:    31014  Right:   360067 Length:       31    Text: a mighty hunter before the LORD
Doc:bleak_house.txt (1,1)   Left:  1199228  Right:   978539 Length:       31    Text: be up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:  3439534  Right:  1249000 Length:       31    Text: marrying and giving in marriage
Doc:bleak_house.txt (1,1)   Left:    71574  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   235452  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   239383  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   302791  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   350278  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   593938  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   876859  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:   880393  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:  1141838  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:bleak_house.txt (1,1)   Left:  1968978  Right:   978540 Length:       30    Text: e up early in the morning, and
Doc:old_curiosity_s (1,7)   Left:  2378359  Right:  1082552 Length:       32    Text: heap coals of fire upon his head
Doc:old_curiosity_s (1,7)   Left:  1738254  Right:   462753 Length:       31    Text: and married fourteen wives, and
Doc:old_curiosity_s (1,7)   Left:  3684636  Right:   489980 Length:       30    Text: very early in the morning, the
Doc:great_expectati (1,4)   Left:  3651100  Right:   944296 Length:       31    Text: went up into the temple to pray
Doc:pickwick_papers (1,9)   Left:  1942291  Right:  1730970 Length:       37    Text: he had neither father nor mother, and
Doc:pickwick_papers (1,9)   Left:  1139565  Right:   401542 Length:       34    Text: on the other side: and there was a
Doc:pickwick_papers (1,9)   Left:  1838405  Right:    19021 Length:       30    Text: thousand eight hundred and twe
Doc:tale_of_two_cit (1,10)  Left:  3746360  Right:   628647 Length:       63    Text: he that believeth in me, though he were dead, yet shall h...
Doc:tale_of_two_cit (1,10)  Left:  3746360  Right:   630608 Length:       63    Text: he that believeth in me, though he were dead, yet shall h...
Doc:tale_of_two_cit (1,10)  Left:  3746360  Right:   754350 Length:       63    Text: he that believeth in me, though he were dead, yet shall h...
Doc:tale_of_two_cit (1,10)  Left:  3746431  Right:   754415 Length:       56    Text: And whosoever liveth and believeth in me shall never die
Doc:tale_of_two_cit (1,10)  Left:  3746431  Right:   628712 Length:       40    Text: And whosoever liveth and believeth in me
Doc:tale_of_two_cit (1,10)  Left:  3746431  Right:   630673 Length:       40    Text: And whosoever liveth and believeth in me
Doc:tale_of_two_cit (1,10)  Left:  3063584  Right:   527721 Length:       33    Text: the pavement by the side of the g
Doc:tale_of_two_cit (1,10)  Left:    12745  Right:   475309 Length:       31    Text: from the face of the earth; and
Doc:tale_of_two_cit (1,10)  Left:   342333  Right:   475304 Length:       31    Text: them from the face of the earth
Doc:tale_of_two_cit (1,10)  Left:  3528147  Right:   667929 Length:       30    Text: when all these things shall be
Doc:little_dorrit.t (1,6)   Left:  3608991  Right:  1414177 Length:       34    Text: the labourer is worthy of his hire
Doc:little_dorrit.t (1,6)   Left:  3549670  Right:  1693887 Length:       32    Text: According to the custom of the p
Doc:little_dorrit.t (1,6)   Left:  2479377  Right:   149352 Length:       31    Text: n from generation to generation
Doc:little_dorrit.t (1,6)   Left:  2597837  Right:   149352 Length:       31    Text: n from generation to generation
Doc:little_dorrit.t (1,6)   Left:  2856706  Right:   149352 Length:       31    Text: n from generation to generation
Doc:little_dorrit.t (1,6)   Left:  1838405  Right:   501606 Length:       30    Text: thousand eight hundred and twe
Doc:little_dorrit.t (1,6)   Left:  3140105  Right:   937056 Length:       30    Text: very early in the morning, and
Doc:little_dorrit.t (1,6)   Left:  3716920  Right:   467952 Length:       30    Text: stood on the other side of the
Doc:christmas_carol (1,2)   Left:  3508953  Right:   137229 Length:       53    Text: And he took a child, and set him in the midst of them
Doc:christmas_carol (1,2)   Left:  1533926  Right:    99706 Length:       32    Text: and their children's children: a
Doc:christmas_carol (1,2)   Left:  3406634  Right:   137250 Length:       32    Text: and set him in the midst of them
Doc:oliver_twist.tx (1,8)   Left:  3878715  Right:   796294 Length:       32    Text: Your blood be upon your own head
Doc:oliver_twist.tx (1,8)   Left:  1506502  Right:   376123 Length:       31    Text: that was brought into the house
Doc:oliver_twist.tx (1,8)   Left:  1507334  Right:   376123 Length:       31    Text: that was brought into the house
Doc:oliver_twist.tx (1,8)   Left:  1821376  Right:   376123 Length:       31    Text: that was brought into the house
Doc:oliver_twist.tx (1,8)   Left:  1822484  Right:   376123 Length:       31    Text: that was brought into the house
Doc:david_copperfie (1,3)   Left:  3445580  Right:  1002902 Length:       43    Text: have done it unto one of the least of these
Doc:david_copperfie (1,3)   Left:  3353872  Right:  1606967 Length:       38    Text: thieves do not break through nor steal
Doc:david_copperfie (1,3)   Left:    68504  Right:  1057125 Length:       33    Text: up early in the morning, and took
Doc:david_copperfie (1,3)   Left:   105108  Right:  1057125 Length:       33    Text: up early in the morning, and took
Doc:david_copperfie (1,3)   Left:  1698448  Right:   889032 Length:       31    Text: t was in the front of the house
Doc:david_copperfie (1,3)   Left:  1975302  Right:  1630106 Length:       31    Text: the wicked cease from troubling
Doc:david_copperfie (1,3)   Left:  3446288  Right:  1002917 Length:       31    Text: to one of the least of these, y
Doc:david_copperfie (1,3)   Left:  1460768  Right:  1057125 Length:       30    Text: up early in the morning, and t
Doc:david_copperfie (1,3)   Left:  2598653  Right:   879087 Length:       30    Text: s, and laid the foundations of
Doc:david_copperfie (1,3)   Left:  3208034  Right:   673473 Length:       30    Text: ers into the hand of the child
Doc:david_copperfie (1,3)   Left:  3354972  Right:   653547 Length:       30    Text: toil not, neither do they spin

Successes: 0     Failures:0
Something went wrong with that request. Please try again.