This repository contains the code used for the experiments in "Predictive Uncertainty-based Bias Mitigation in Ranking", which will be published at CIKM 2023.
If you use this code to produce results for your scientific publication, or if you share a copy or fork, please refer to our CIKM 2023 paper:
@inproceedings{heuss-2023-predictive,
author = {Heuss, Maria and Cohen, Daniel and Mansoury, Masoud and de Rijke, Maarten and Eickhoff, Carsten},
booktitle = {CIKM 2023: 32nd ACM International Conference on Information and Knowledge Management},
publisher = {ACM},
title = {Predictive Uncertainty-based Bias Mitigation in Ranking},
year = {2023}}
This repository is published under the terms of the GNU General Public License version 3. For more information, see the file LICENSE.
Predictive Uncertainty-based Bias Mitigation in Ranking
Copyright (C) 2023 Maria Heuss
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>
Packages used in this repository can be found in requirements.txt
To run the experiments that are recorded in the paper create a folder called 'results',
unzip the data in the 'data'-folder and run 'run_msmarco.py'. The file 'msmarco_pointwise_uncertainty.csv'
contains pre-calculated uncertainty estimates in form of mean and standard deviation.
An example of how such uncertainty estimates can be generated can be found in 'uncertainty.laplace_approximation.py'.
By choosing the variable 'experiment' in ['table', 'ablation', 'trade-off-curve']
the results of the table, ablation-study resp. trade-off plot can be generated.
This will create new files in the '\results' folder one with the full candidate
DataFrame containing a column with the new ranks for each method and one with the
calculated metrics for each query and each method.
This project uses parts of the code of
- https://github.com/MilkaLichtblau/BA_Laura, and
- https://networkx.org, which is published under the 3-clause BSD licence
for the implementation of the convex optimization baseline.