Query-Document Relevance
Python C++ Makefile
Switch branches/tags
Nothing to show
Clone or download
Matthew Peters
Matthew Peters Update README
Latest commit 8d0ba61 Feb 5, 2015



Build Status

Query-Document Relevance ranking functions

This repository implements a few query-document similarity functions, commonly used in information retrieval applications. It supports:

  • TF-IDF
  • Okapi BM25
  • Language Model

This implementation includes pure Python code for iteratively training models from a large corpus, and a C++ implementation of the scoring functions with Cython wrappers for fast evaluation.

Each of these ranking functions has a few "magic" constants. Currently these are hard coded to values recommend in the literature, but if the need arises can be configurable. Relevant references:


All tokenization and word normalization is handled client side, and all methods that accept queries or documents assume they are lists of byte strings, not unicode.

There are two separate steps to using the ranking functions: training and scoring.


The Trainer class supports incremental training from a large corpus, combining separately trained models for map-reduce type data flows, pruning of infrequent tokens from large models and serialization. Typical usage:

from qdr import Trainer

# load corpus -- it's an iterable of documents, each document is a
# list of byte strings
model = Trainer()

# the train method adds documents incrementally so it can be updated with
# additional documents by calling train again

# write to a file

For map-reduce type work, the method update_counts_from_trained will merge the contents of two Trainer instances:

# map step -- typically this is parallelized
for k, corpus in enumerate(corpus_chunks):
    model = Trainer()
    model.serialize_to_file("file%s.gz" % k)

# reduce step
model = Trainer.load_from_file("file0.gz")
for k in xrange(1, len(corpus_chunks)):
    model2 = Trainer.load_from_file("file%s.gz" % k)

# prune the final model if needed
model.prune(min_word_count, min_doc_count)


Typical usage:

from qdr import QueryDocumentRelevance

scorer = QueryDocumentRelevance.load_from_file('trained_model.gz')
# document, query are lists of byte strings
relevance_scores = scorer.score(document, query)

For scoring batches of queries against a single document, the score_batch method is more efficient then calling score repeatedly:

# queries is a list of queries, each query is a list of tokens:
relevance_scores = scorer.score(document, queries)


sudo pip install -r requirements.txt
sudo make install


Contributions welcome! Fork, commit, then open a pull request.