We developed three versions of a fictitious paper, each with a significant error intentionally embedded in one of the main claims. These errors were crafted to be fully contained within the paper's main text, relate directly to a result touted as a key contribution, and not be immediately obvious but require a detailed reading of the paper. The papers are available in the folder Experimental_papers. These papers, formatted according to style files of one popular conference, were then passed to LLMs for evaluation. The evaluation outcomes are detailed in folder LLMevaluation.
In what follows, we detail the error introduced in the three variants:
-
Variant with Error in Claim of Estimator's Optimization Property: This variant claims the first estimator presented is a convex optimization program. The paper emphasizes, "Our result that each of our estimators amount to solving convex optimization problems is of vital importance, and form a key contribution of this paper.'' It also justifies the importance of convexity. However, the provided proof and the stated claim are incorrect. Among the three errors, this error was the most apparent, with its claim raising immediate suspicion.
-
Variant with Error in Empirical Evaluation Claim: In this version, the paper discusses the superior performance of the proposed algorithm on crowdsourced-labeling data compared to other algorithms. It asserts that "The findings presented here constitute the most significant results of this paper.'' However, the algorithm as presented chose its hyperparameters using the test data, thereby overestimating the performance of the proposed algorithm, and hence rendering these evaluations invalid. (See, for example, arxiv.org/pdf/2206.07647 for more discussion on hyperparamter choices.) Identification of this error required an examination of the pseudocode of the algorithm provided in the main text of the paper.
-
Variant with Error in Claim of Statistical Identifiability of Model: The paper introduces a parametric model for the problem setting, and in this variant, claims to establish a necessary and sufficient condition for the model's identifiability. After formally introducing the concept of identifiability and the claim, the paper highlights this result as "This result is particularly significant due to the necessity as well as sufficiency of the stated condition.This result thus establishes a definitive criterion for identifiability under our convex combination GLM model, and hence forms a key contribution of this paper.'' However, the proof for the claim of necessity, and the stated claim, are incorrect. Among the three errors, this was the most subtle and required a very careful examination of the associated proof.