Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



28 Commits

Repository files navigation


This allows the information retrieval platform to ingest files written in the Common Index File Format - see [1]. In doing so, a new Terrier index is written.


The Terier-CIFF package provides a tool for ingesting files written in the Common Index File Format. It can be used directly from the Terrier commandline (version 5.3 minimum).

    cd /path/to/terrier
    bin/terrier  -P com.github.terrierteam:terrier-ciff:0.2 ciff-ingest /path/to/ciff.gz

This creates a new index at the default location. The standard -I option can be used to change the location of the generated index.

Additional Weighting Models

We provide variants of BM25 that are aligned with those in the Anserini platform. These are as follows:

  • BM25_log10 - This follows Terrier's standard implementation, but uses log base 10 instead of log base 2.
  • BM25_log10_nonum - This removes the unnecessary (k+1) from the BM25 numerator. It alsos uses log base 10.

For more discussion, see [1] and [2].

Implementation Notes

  • collection statistics (number of tokens/average document length) are calculated using the document lengths file; we do not consider the collection statistics in the header.
  • we use a shaded version of protobuf 3, as Terrier depends on Hadoop, which includes protobuf 2.5


We have a notebook demonstrating using terrier-ciff for ingesting a CIFF index into Terrier. It also creates a CIFF file from a Terrier index. This is a [PyTerrier] notebook. It can be run directly using Colab:


Craig Macdonald, University of Glasgow


[1] Jimmy Lin, Joel Mackenzie, Chris Kamphuis, Craig Macdonald, Antonio Mallia, Michał Siedlaczek, Andrew Trotman, Arjen de Vries. Supporting Interoperability Between Open-Source Search Engines with the Common Index File Format. In Proceedings of SIGIR 2020. arXiv:2003.08276.

[2] Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants. Chris Kamphuis, Arjen de Vries, Leonid Boytsov and Jimmy Lin. In Proceedings of ECIR 2020.


No description, website, or topics provided.






No releases published