We employ Longformer, a BERT-like model for long documents, on the MS MARCO document re-ranking dataset. More details about our model and experimental setting can be found in our paper.
Due to the computing limitations, the hyperparameters were not optimised. We default to the following hyperparameters:
--lr=3e-05
--max_seq_len=4096
--num_warmup_steps=2500
For each query, we randomly sample 10 negative documents from the top 100 documents retrieved in the initial retrieval step.
To train the model, first download all of the necessary data, as described in data/README.md. File names should match the filenames in MarcoDataset.py.
You can then train with:
python run_longformer_marco.py
You can check all available hyperparameters with:
python run_longformer_marco.py --help
| Dev | Test | |
|---|---|---|
| MRR@100 | 0.3366 | 0.307 |
The work is done by Ivan Sekulic (Università della Svizzera italiana), Amir Soleimani (University of Amsterdam), Mohammad Aliannejadi (University of Amsterdam), and Fabio Crestani (Università della Svizzera italiana).
Please consider citing our paper if you use our code or models:
@misc{sekuli2020longformer,
title={Longformer for MS MARCO Document Re-ranking Task},
author={Ivan Sekulić and Amir Soleimani and Mohammad Aliannejadi and Fabio Crestani},
year={2020},
eprint={2009.09392},
archivePrefix={arXiv},
primaryClass={cs.IR}
}