Abstract: Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain aspects such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspects from review forms and guidelines, yet data-driven methods for aspect identification are underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review.
Contact person: LU Sheng
https://www.ukp.tu-darmstadt.de/
This repository contains the code for reproducing the aspect set construction, aspect prediction, and practical application experiments, along with the aspect taxonomy file and all evaluation scores.
The files in this repository are largely aligned with the sections of the paper, so you can simply locate the corresponding file based on the section of interest. The ipynb notebook files additionally contain detailed explanations of the code for easier reproduction and understanding.
- Implementation: notebook-azure.ipynb
- Pre- and postprocessing: notebook-processs.ipynb
- Raw GPT-4o identification results: annotation - llm.csv
- Coarsed-grained: aspects - coarse.csv
- Fine-grained: aspects - fine.csv
- LLM annotation consistency: notebook-validity_check_&_aspect_analysis.ipynb
- Annotation guidelines: annotation-guidelines.pdf
- Human annotations: human_annotations/
- Raw review data from Openreview: data/ (see example.md for structure)
- Selected papers: selected_papers/
- Preprocessed data (segmented into sentences): preprocessed/
- Paper aspect prediction: notebook-bow-paper_aspect_prediction.ipynb
- Review aspect prediction: notebook-bow-review_aspect_prediction.ipynb
- Paper aspect prediction: bert-paper_aspect_prediction_training.py
- Review aspect prediction: bert-review_aspect_prediction_training.py
- Implementation: notebook-azure.ipynb
- Paper aspect prediction: evaluation_scores-paper_aspect_prediction.txt
- Review aspect prediction: evaluation_scores-review_aspect_prediction.txt
- Implementation: notebook-validity_check_&_aspect_analysis.ipynb
- Implementation: notebook-review_comparison_&_detection.ipynb
Please use the following citation:
@inproceedings{lu-etal-2025-identifying,
title = "Identifying Aspects in Peer Reviews",
author = "Lu, Sheng and
Kuznetsov, Ilia and
Gurevych, Iryna",
editor = "Christodoulopoulos, Christos and
Chakraborty, Tanmoy and
Rose, Carolyn and
Peng, Violet",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-emnlp.326/",
doi = "10.18653/v1/2025.findings-emnlp.326",
pages = "6145--6167",
ISBN = "979-8-89176-335-7",
abstract = "Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain *aspects* such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspects from review forms and guidelines, yet data-driven methods for aspect identification are underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review."
}
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.