This is the repository for FERMAT: An Alternative to Accuracy for Numerical Reasoning by Jasivan Alex Sivakumar and Nafise Sadat Moosavi accepted at ACL 2023.
You can find the codes, templates and test sets in their respective folders. Note that the codes folder is still being updated.
- Codes contains the codes for training and evaluating the model.
- Templates contains the templates written by expert maths teachers, from GSM8K and AQUA too.
- Test_sets contains the Original test set (based on [Illinois](https://aclanthology.org/N16-3011.pdf) and [CommonCore](https://aclanthology.org/D15-1202.pdf)) and the 18 variation test sets used in FERMAT.
Install the packages from the requirements.txt file (transformers, torch, datasets, accelerate, SentencePiece, json, pandas, numpy).
If you use this code in your work, please cite the paper:
@InProceedings{sivakumar-moosavi_2023_FERMAT,
author = { Jasivan Alex Sivakumar, Nafise Sadat Moosavi},
title = {FERMAT: An Alternative to Accuracy for Numerical Reasoning},
year = {2023},
booktitle = {Proceedings of the 61st Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers)},
publisher = {Association for Computational Linguistics},
address = {Toronto, Canada},
}
This code was written by @jasivan.
If you have any queries or comments, please email jasivakumar1@sheffield.ac.uk .