SciRepEval: A Multi-Format Benchmark for Scientific Document Representations

This repo contains the code to train, evaluate and reproduce the representation learning models and results on the benchmark introduced in SciRepEval.

Quick Setup

Clone the repo and setup the environment as follows:

git clone git@github.com:allenai/scirepeval.git
cd scirepeval
conda create -n scirepeval python=3.8
conda activate scirepeval
pip install -r requirements.txt

Usage

Please refer to the following for further usage:

Training - Train multi-task/multi-format transformer models or adapter modules

Inference - Using the trained SciRepEval models to generate embeddings.

Evaluation - Evaluate trained models on custom tasks OR customize existing evaluation config for SciRepEval benchmark tasks

Benchmarking - Simply evaluate models(pretrained from HuggingFace/local checkpoints) on SciRepEval and generate a report

Benchmark Details

SciRepEval consists of 24 scientific document tasks to train and evaluate scientific document representation models. The tasks are divided across 4 task formats- classification CLF, regression RGN, proximity (nearest neighbors) retrieval PRX and ad-hoc search SRCH. The table below gives a brief overview of the tasks with their HuggingFace datasets config names, if applicable. The benchmark dataset can be downloaded from AWS S3 or HuggingFace as follows:

AWS S3 via CLI

mkdir scirepeval_data && mkdir scirepeval_data/train && mkdir scirepeval_data/test && cd scirepeval_data
aws s3 --no-sign-request sync s3://ai2-s2-research-public/scirepeval/train train
aws s3 --no-sign-request sync s3://ai2-s2-research-public/scirepeval/test test

The AWS CLI commands can be run with the --dryrun flag to list the files being copied. The entire dataset is ~24 GB in size.

HuggingFace Datasets

The training, validation and raw evaluation data is available at allenai/scirepeval, while the labelled test examples are available at allenai/scirepeval_test.

import datasets
#training/validation/eval metadata
dataset = datasets.load_dataset(allenai/scirepeval, <hf config name>)

#labelled test examples
dataset = datasets.load_dataset(allenai/scirepeval_test, <hf config name>)

Since we want to evaluate document representations, every dataset consists of two parts: test metadata (text for representation generation available under allenai/scirepeval) and labelled examples (available under allenai/scirepeval_test)

Format	Name	Train	Metric	HF Config	HF Test Config
CLF	MeSH Descriptors	Y	F1 Macro	mesh_descriptors	mesh_descriptors
CLF	Fields of study	Y	F1 Macro	fos	fos
CLF	Biomimicry	N	F1 Binary	biomimicry	biomimicry
CLF	DRSM	N	F1 Macro	drsm	drsm
CLF	SciDocs-MAG	N	F1 Macro	scidocs_mag_mesh	scidocs_mag
CLF	SciDocs-Mesh Diseases	N	F1 Macro	scidocs_mag_mesh	scidocs_mesh
RGN	Citation Count	Y	Kendall's Tau	cite_count	cite_count
RGN	Year of Publication	Y	Kendall's Tau	pub_year	pub_year
RGN	Peer Review Score	N	Kendall's Tau	peer_review_score_hIndex	peer_review_score
RGN	Max Author hIndex	N	Kendall's Tau	peer_review_score_hIndex	hIndex
RGN	Tweet Mentions	N	Kendall's Tau	tweet_mentions	tweet_mentions
PRX	Same Author Detection	Y	MAP	same_author	same_author
PRX	Highly Influential Citations	Y	MAP	high_influence_cite	high_influence_cite
PRX	Citation Prediction	Y	-	cite_prediction	-
PRX	S2AND*	N	B^3 F1	-	-
PRX	Paper-Reviewer Matching**	N	Precision@5,10	paper_reviewer_matching	paper_reviewer_matching, reviewers
PRX	RELISH	N	NDCG	relish	relish
PRX	SciDocs-Cite	N	MAP, NDCG	scidocs_view_cite_read	scidocs_cite
PRX	SciDocs-CoCite	N	MAP, NDCG	scidocs_view_cite_read	scidocs_cocite
PRX	SciDocs-CoView	N	MAP, NDCG	scidocs_view_cite_read	scidocs_view
PRX	SciDocs-CoRead	N	MAP, NDCG	scidocs_view_cite_read	scidocs_read
SRCH	Search	Y	NDCG	search	search
SRCH	NFCorpus	N	NDCG	nfcorpus	nfcorpus
SRCH	TREC-CoVID	N	NDCG	trec_covid	trec_covid

*S2AND requires the evaluation dataset in a specific format so to evaluate your model on the task please follow these instructions.

**Combinations of multiple datasets - 1, 2, 3, also dataset of papers authored by potential reviewers is required for evaluation; hence the multiple dataset configs.

License

The aggregate benchmark is released under ODC-BY license. By downloading this data you acknowledge that you have read and agreed to all the terms in this license. For constituent datasets, also go through the individual licensing requirements, as applicable.

Citation

Please cite the SciRepEval work as:

@article{Singh2022SciRepEvalAM,
  title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},
  author={Amanpreet Singh and Mike D'Arcy and Arman Cohan and Doug Downey and Sergey Feldman},
  journal={ArXiv},
  year={2022},
  volume={abs/2211.13308}
}