Skip to content

castorini/anserini

master
Switch branches/tags
Code

Latest commit

Additional changes:
+ Tweaks to experiments-msmarco-passage.md and experiments-msmarco-doc.md
+ Fixed (some) incorrect dates on when tuning was performed for MS MARCO v1/v2 doc/passage (and d2q-T5)
+ Added missing tuned2 conditions to dl19-doc
+ Added missing ax/bm25prf conditions to dl20-doc and msmarco-doc
+ Fixed bug in neg Rocchio condition on passage d2q (-rerankCutoff 1000)
8010d5c

Git stats

Files

Permalink
Failed to load latest commit information.

Anserini

build codecov Generic badge Maven Central LICENSE doi

Anserini is a toolkit for reproducible information retrieval research. By building on Lucene, we aim to bridge the gap between academic information retrieval research and the practice of building real-world search applications. Among other goals, our effort aims to be the opposite of this.* Anserini grew out of a reproducibility study of various open-source retrieval engines in 2016 (Lin et al., ECIR 2016). See Yang et al. (SIGIR 2017) and Yang et al. (JDIQ 2018) for overviews.

Getting Started

Many Anserini features are exposed in the Pyserini Python interface. If you're looking for basic indexing and search capabilities, you might want to start there. A low-effort way to try out Anserini is to look at our online notebooks, which will allow you to get started with just a few clicks. For convenience, we've pre-built a few common indexes, available to download here.

You'll need Java 11 and Maven 3.3+ to build Anserini. Clone our repo with the --recurse-submodules option to make sure the eval/ submodule also gets cloned (alternatively, use git submodule update --init). Then, build using using Maven:

mvn clean package appassembler:assemble

Note that on Windows, tests may fail due to encoding issues, see #1466. A simple workaround is to skip tests by adding -Dmaven.test.skip=true to the above mvn command. See #1121 for additional discussions on debugging Windows build errors.

The tools/ directory, which contains evaluation tools and other scripts, is actually this repo, integrated as a Git submodule (so that it can be shared across related projects). Build as follows (you might get warnings, but okay to ignore):

cd tools/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make && cd ../../..
cd tools/eval/ndeval && make && cd ../../..

With that, you should be ready to go!

Regression Experiments (+ Reproduction Guides)

Anserini is designed to support experiments on various standard IR test collections out of the box. The following experiments are backed by rigorous end-to-end regression tests with run_regression.py and the Anserini reproducibility promise. For the most part, these runs are based on default parameter settings.

These pages can also serve as guides to reproduce our results. See individual pages for details!

MS MARCO V1 Passage Corpus

dev DL19 DL20
Unsupervised Lexical
BoW baselines + + +
Quantized BM25
WP baselines + + +
doc2query +
doc2query-T5 + + +
Learned sparse lexical (uniCOIL family)
uniCOIL noexp
uniCOIL with doc2query-T5
uniCOIL with TILDE
Learned sparse lexical (other)
DeepImpact
SPLADEv2
SPLADE-distill CoCodenser-medium

MS MARCO V1 Document Corpus

dev DL19 DL20
Unsupervised lexical, complete doc*
BoW baselines + + +
WP baselines + + +
doc2query-T5 + + +
Unsupervised lexical, segmented doc*
BoW baselines + + +
WP baselines + + +
doc2query-T5 + + +
Learned sparse lexical
uniCOIL noexp
uniCOIL with doc2query-T5

MS MARCO V2 Passage Corpus

dev DL21
Unsupervised lexical, original corpus
baselines + +
doc2query-T5 + +
Unsupervised lexical, augmented corpus
baselines + +
doc2query-T5 + +
Learned sparse lexical
uniCOIL noexp zero-shot
uniCOIL with doc2query-T5 zero-shot

MS MARCO V2 Document Corpus

dev DL21
Unsupervised lexical, complete doc
baselines + +
doc2query-T5 + +
Unsupervised lexical, segmented doc
baselines + +
doc2query-T5 + +
Learned sparse lexical
uniCOIL noexp zero-shot
uniCOIL with doc2query-T5 zero-shot

Regressions for BEIR (v1.0.0)

  • F = "flat" baseline
  • MF = "multifield" baseline
  • UCx = uniCOIL (noexp)
  • SPLADE = SPLADE-distill CoCodenser-medium
Corpus flat flat-wp multifield UCx SPLADE
TREC-COVID + + + + +
BioASQ + + + + +
NFCorpus + + + + +
NQ + + + + +
HotpotQA + + + + +
FiQA-2018 + + + + +
Signal-1M(RT) + + + + +
TREC-NEWS + + + + +
Robust04 + + + + +
ArguAna + + + + +
Touche2020 + + + + +
CQADupStack-Android + + + + +
CQADupStack-English + + + + +
CQADupStack-Gaming + + + + +
CQADupStack-Gis + + + + +
CQADupStack-Mathematica + + + + +
CQADupStack-Physics + + + + +
CQADupStack-Programmers + + + + +
CQADupStack-Stats + + + + +
CQADupStack-Tex + + + + +
CQADupStack-Unix + + + + +
CQADupStack-Webmasters + + + + +
CQADupStack-Wordpress + + + + +
Quora + + + + +
DBPedia + + + + +
SCIDOCS + + + + +
FEVER + + + + +
Climate-FEVER + + + + +
SciFact + + + + +

Other Regressions

Available Corpora

Corpora Size Checksum
MS MARCO V1 passage: Quantized BM25 1.2 GB 0a623e2c97ac6b7e814bf1323a97b435
MS MARCO V1 passage: uniCOIL (noexp) 2.7 GB f17ddd8c7c00ff121c3c3b147d2e17d8
MS MARCO V1 passage: uniCOIL (d2q-T5) 3.4 GB 78eef752c78c8691f7d61600ceed306f
MS MARCO V1 passage: uniCOIL (TILDE) 3.9 GB 12a9c289d94e32fd63a7d39c9677d75c
MS MARCO V1 passage: DeepImpact 3.6 GB 73843885b503af3c8b3ee62e5f5a9900
MS MARCO V1 passage: SPLADEv2 9.9 GB b5d126f5d9a8e1b3ef3f5cb0ba651725
MS MARCO V1 passage: SPLADE CoCodenser 4.9 GB f77239a26d08856e6491a34062893b0c
MS MARCO V1 doc: uniCOIL (noexp) 11 GB 11b226e1cacd9c8ae0a660fd14cdd710
MS MARCO V1 doc: uniCOIL (d2q-T5) 19 GB 6a00e2c0c375cb1e52c83ae5ac377ebb
MS MARCO V2 passage: uniCOIL (noexp) 24 GB d9cc1ed3049746e68a2c91bf90e5212d
MS MARCO V2 passage: uniCOIL (d2q-T5) 41 GB 1949a00bfd5e1f1a230a04bbc1f01539
MS MARCO V2 doc: uniCOIL (noexp) 55 GB 97ba262c497164de1054f357caea0c63
MS MARCO V2 doc: uniCOIL (d2q-T5) 72 GB c5639748c2cbad0152e10b0ebde3b804

Additional Documentation

The experiments described below are not associated with rigorous end-to-end regression testing and thus provide a lower standard of reproducibility. For the most part, manual copying and pasting of commands into a shell is required to reproduce our results.

MS MARCO (V1)

MS MARCO (V2)

TREC-COVID and CORD-19

Other Experiments

Other Features

How Can I Contribute?

If you've found Anserini to be helpful, we have a simple request for you to contribute back. In the course of reproducing baseline results on standard test collections, please let us know if you're successful by sending us a pull request with a simple note, like what appears at the bottom of the page for Disks 4 & 5. Reproducibility is important to us, and we'd like to know about successes as well as failures. Since the regression documentation is auto-generated, pull requests should be sent against the raw templates. Then the regression documentation can be generated using the bin/build.sh script. In turn, you'll be recognized as a contributor.

Beyond that, there are always open issues we would appreciate help on!

Release History

Historical Notes

  • Anserini was upgraded to Java 11 at commit 17b702d (7/11/2019) from Java 8. Maven 3.3+ is also required.
  • Anserini was upgraded to Lucene 8.0 as of commit 75e36f9 (6/12/2019); prior to that, the toolkit uses Lucene 7.6. Based on preliminary experiments, query evaluation latency has been much improved in Lucene 8. As a result of this upgrade, results of all regressions have changed slightly. To reproducible old results from Lucene 7.6, use v0.5.1.

References

Acknowledgments

This research is supported in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada. Previous support came from the U.S. National Science Foundation under IIS-1423002 and CNS-1405688. Any opinions, findings, and conclusions or recommendations expressed do not necessarily reflect the views of the sponsors.