The HIPE-2022 Evaluation Toolkit contains the results of the HIPE-2022 shared task on NE processing of historical documents, as well as the necessary material for replicating them, namely:
- the test data;
- the submitted system responses;
- the HIPE-scorer (as a submodule, commit 6605770);
- a Makefile to run the whole evaluation process.
For more information, also have a look at:
- 🏁 HIPE-2022 results in HIPE_2022_evaluation_results.md and on the website.
- 💻 HIPE-2022 website;
- 📂 Official HIPE-2022 data releases;
- 🔅 HIPE-2022 baselines;
- 📚 HIPE evaluation campaigns zenodo community.
To install the module:
git clone --recurse-submodules git@github.com:hipe-eval/HIPE-2022-eval.git
cd HIPE-2022-eval
python3 -mvenv venv
source venv/bin/activate
pip install -r requirements.txt
( cd HIPE-scorer && pip install -r requirements.txt && python setup.py install )
# if submodule HIPE-scorer is updated, the following might be needed
git submodule update
( cd HIPE-scorer && pip install -r requirements.txt && python setup.py install )
To run the evaluation:
make eval-full # Creates all evaluation steps
# in case you want to start from scratch and refresh all derived files
make eval-full-refresh
If you want to build the targets incrementally:
make eval-system-bundles
make ranking-alldatasets-alllanguages
make generate-rankings-summary
make generate-rankings-summary rankings-summary-ToC
Look at the HIPE section of the Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, September 5th to 8th, 2022.
- CEUR HIPE-2022 extended overview paper:
M. Ehrmann, M. Romanello, S. Najem-Meyer, A. Doucet, and S. Clematide (2022). Extended Overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents. In: Faggioli, Gulielmo; Ferro, Nicola; Hanbury, Alan; Potthast, Martin. Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum (CLEF). Aachen: CEUR-WS, 1038-1063.
@incollection{hipe2022_extended_2022,
title = {{Extended Overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents}},
author = {Maud Ehrmann and Matteo Romanello and Sven Najem-Meyer and Antoine Doucet and Simon Clematide},
series = {CEUR Workshop Proceedings},
booktitle = {Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum (CLEF)},
editor = {Gulielmo Faggioli and Nicola Ferro and Alan Hanbury and Martin Potthast},
address = {Aachen},
publisher = {CEUR-WS},
year = {2022},
pages = {1038--1063},
url = {https://doi.org/10.5167/uzh-230656},
issn = {1613-0073}
}
- LNCS HIPE-2022 Condensed Lab Overview Paper:
M. Ehrmann, M. Romanello, S. Najem-Meyer, A. Doucet, and S. Clematide (2022). Overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Thirteenth International Conference of the CLEF Association (CLEF 2022). Lecture Notes in Computer Science. Springer, Cham (link to accepted version).
@inproceedings{hipe2022_condensed_2022,
title = {{Overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents}},
booktitle = {{Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Thirteenth International Conference of the CLEF Association (CLEF 2022)}},
series = {Lecture Notes in Computer Science (LNCS)},
publisher = {Springer},
author = {Ehrmann, Maud and Romanello, Matteo and Najem-Meyer, Sven and Doucet, Antoine and Clematide, Simon},
year = {2022},
editor = {Barrón-Cedeño, Alberto and Da San Martino, Giovanni and Degli Esposti, Mirko and Sebastiani, Fabrizio and Macdonald, Craig and Pasi, Gabriella and Hanbury, Allan and Potthast, Martin and Faggioli, Guglielmo and Ferro, Nicola
}
- ECIR-2022 Introduction Short Paper:
M. Ehrmann, M. Romanello, A. Doucet, and S. Clematide (2022). Introducing the HIPE 2022 Shared Task: Named Entity Recognition and Linking in Multilingual Historical Documents. In: Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13186. Springer, Cham (link to postprint).
@inproceedings{ehrmann_introducing_2022,
title = {{Introducing the HIPE 2022 Shared Task:Named Entity Recognition and Linking in Multilingual Historical Documents}},
booktitle = {Proceedings of the 44\textsuperscript{d} European Conference on {{IR}} Research ({{ECIR}} 2022)},
author = {Ehrmann, Maud and Romanello, Matteo and Clematide, Simon and Doucet, Antoine},
year = {2022},
publisher = {{Lecture Notes in Computer Science, Springer}},
address = {{Stavanger, Norway}},
url = {https://link.springer.com/chapter/10.1007/978-3-030-99739-7_44}
}
- The test data is released under different licenses, check the data repository for more information.
- HIPE-2022 evaluation code is released under [to be completed]. Copyright (c) 2022 HIPE-2022 team.