Intuitive axiomatic retrieval experimentation.
ir_axioms
is a Python framework for experimenting with axioms in information retrieval in a declarative way.
It includes reference implementations of many commonly used retrieval axioms and is well integrated with the PyTerrier and Pyserini frameworks.
Re-rank your search results today with ir_axioms
and understand your retrieval systems better by analyzing
axiomatic preferences!
We include several example notebooks to demonstrate re-ranking and preference evaluation in PyTerrier using ir_axioms
.
You can find all examples in the examples/
directory.
- Re-ranking top-20 results using KwikSort
- Re-ranking top-20 results using KwikSort learned from ORACLE preferences
- Re-ranking top-20 results using LambdaMART with axiomatic preference features
- Post-Hoc Analysis of Rankings and Relevance Judgments
- Axiomatic prefernces for TREC Deep Learning 2019 runs (passages)
- Axiomatic prefernces for TREC Deep Learning 2019 runs (documents)
- Axiomatic prefernces for TREC Deep Learning 2020 runs (passages)
- Axiomatic prefernces for TREC Deep Learning 2020 runs (documents)
TODO
If you want to play around with ir_axioms
in Jupyter Lab, you can use this command to provision a server via Slurm:
scripts/slurm-start-jupyter-lab.sh
If you use this package or its components in your research, please cite the following paper describing the ir_axioms
framework and its use-cases:
Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen. Axiomatic Retrieval Experimentation with
ir_axioms
. In 45th International ACM Conference on Research and Development in Information Retrieval (SIGIR 2022), July 2022. ACM.
You can use the following BibTeX entry for citation:
@InProceedings{bondarenko:2022d,
author = {Alexander Bondarenko and Maik Fr{\"o}be and {Jan Heinrich} Reimer and Benno Stein and Michael V{\"o}lske and Matthias Hagen},
booktitle = {45th International ACM Conference on Research and Development in Information Retrieval (SIGIR 2022)},
month = jul,
publisher = {ACM},
site = {Madrid, Spain},
title = {{Axiomatic Retrieval Experimentation with ir_axioms}},
year = 2022
}
To build ir_axioms
and contribute to its development you need to install the build
, and setuptools
and wheel
packages:
pip install build setuptools wheel
(On most systems, these packages are already pre-installed.)
Install dependencies for developing the ir_axioms
package:
pip install -e .
If you want to develop the Pyserini backend, install dependencies like this:
pip install -e .[pyserini]
If you want to develop the PyTerrier backend, install dependencies like this:
pip install -e .[pyterrier]
Install test dependencies:
pip install -e .[test]
Verify your changes against our test suite to verify.
flake8 ir_axioms tests
pylint -E ir_axioms tests.unit --ignore-paths=^ir_axioms.backend
pytest ir_axioms/ tests/unit/ --ignore=ir_axioms/backend/
Please also add tests for the axioms or integrations you've added.
Install test dependencies (replace <BACKEND>
with either pyserini
or pyterrier
):
pip install -e .[<BACKEND>]
Verify your changes against our test suite to verify.
pylint -E ir_axioms.backend.<BACKEND> tests.integration.<BACKEND>
pytest tests/integration/<BACKEND>/
A wheel for this package can be built by running:
python -m build
This repository is released under the MIT license. If you use ir_axioms
in your research, we'd be glad if
you'd cite us.