A Ruby library for calculating the similarity between pieces of text using a Term Frequency-Inverse Document Frequency method.
A bag of words model is used. Terms in the source documents are downcased and punctuation is removed, but stemming is not currently implemented.
This library was written to facilitate the creation of diagrams talked
about by Jonathan Stray in his
visualization of the Iraq War Logs post. An example of how to
generate a Gephi compatible file including labelling of nodes with key
words is included in the
The library depends on the GNU Scientific Library, and the gsl ruby gem but does not use sparse matrix representations to speed up the calculations, since there is no support for them in the GSL. I am currently looking into fixing this, and would appreciate any help!
brew install gsl
gsl gem should then install normally. For other platforms,
please add the information to the wiki and I’ll add them to this
corpus = Corpus.new
corpus << Document.new(:content => "Broad powers for hacking inquiry") corpus << Document.new(:content => "UK unemployment level falls again") corpus << Document.new(:content => "NI riots leads to 26 arrests")
similarity_matrix = corpus.similarity_matrix
For more examples, see the
- Performance improvements
- Switch to storing document vector spaces in sparse form, using linalg or csparse?
- (Optional) stemming of source terms
- Fork the project
- Send a pull request
- Don’t touch the .gemspec, I’ll do that when I release a new version
Chris Lowis - BBC R&D