SDM model question

[AviSimkin]

Hi there,

I'm working on the implementation of some old retrieval algorithm and I'm using Pyserini, to try to recover baselines I need to use the SDM model and I have 2 questions regarding this:

1. Scaling - I see that Pyserini's SDM scores aren't coming out as classic log-sums but rather have a completely different scale, I wanted to ask what they are
2. P(ordered bigram|Corpus), P(unordered bigram|Corpus) approximation - Can you please provide some information on how these are approximated for the SDM model (as they are needed with dirichlet smoothing which is supported by the toolkit)

Thanks,
Avi

[octocat20]

Anserini's SDM is a Lucene approximation of Metzler & Croft, not the Indri-style generative LM.

SdmQueryGenerator builds a Boolean query with three boosted parts (defaults 0.85 / 0.1 / 0.05):

• unigram TermQuerys
• ordered SpanNear with slop 1
• unordered SpanNear with slop 8

The returned score is a weighted sum of Lucene similarity scores (usually BM25 if that is your searcher similarity). That is why the scale does not match classic log-sum SDM scores.

On the collection probabilities: Anserini does not estimate P(ordered|corpus) or P(unordered|corpus) with Dirichlet smoothing. The "approximation" is the span window match itself, then Lucene scores those clauses. If you need the original Indri LM formulation for baseline recovery, you will need Indri (or another LM SDM implementation) rather than comparing absolute Anserini scores directly.