Are the evaluation results extracted automatically from research papers using an automated system, or are they submitted and curated by the end-users, such as researchers and developers?

[zhimin-z]

Dear PapersWithCode maintainers @alefnula @lambdaofgod @rstojnic @mkardas,

As a student researcher at Queen's University, I am interested in understanding how evaluation results are obtained and added to the PapersWithCode platform. Are the evaluation results extracted automatically from research papers using an automated system, or are they submitted and curated by the end-users, such as researchers and developers? For example, end-users can add new evaluation results to the HELM dataset:

I believe this information will help me better comprehend the reliability and scope of the evaluation results presented on the platform, as well as the potential opportunities for contributing to the platform as a researcher.

Thank you for your time and assistance in clarifying this matter.

Best regards,

Jimmy

[alefnula]

HI Jimmy,

It's both. The results are extracted from the papers, but they are also curated and checked for potential errors that might have happened during extraction.

Users are also allowed to add results, but all user added results are curated and only verified results are published.

I hope that this helps :)

[amaralibey]

HI Jimmy,

It's both. The results are extracted from the papers, but they are also curated and checked for potential errors that might have happened during extraction.

Users are also allowed to add results, but all user added results are curated and only verified results are published.

I hope that this helps :)

Hi @alefnula

Is there a history of edits? Is there a way to show which user edited which leaderboard?
I am asking because recently I noticed that the leaderboard was being edited during the ICLR review period, potentially to remove the best performing methods so they don't appear to reviewers that consider looking at paperswithcode to check for the state of the art.

[zhimin-z]

HI Jimmy,
It's both. The results are extracted from the papers, but they are also curated and checked for potential errors that might have happened during extraction.
Users are also allowed to add results, but all user added results are curated and only verified results are published.
I hope that this helps :)

Hi @alefnula

Is there a history of edits? Is there a way to show which user edited which leaderboard? I am asking because recently I noticed that the leaderboard was being edited during the ICLR review period, potentially to remove the best performing methods so they don't appear to reviewers that consider looking at paperswithcode to check for the state of the art.

To the best of my knowledge, there isn't a publicly accessible history of edits or a mechanism to track which user edited a specific leaderboard. While there is some curation involved to ensure accuracy using Axcell, it seems that edits can be made without stringent safeguards or transparency measures from the operators.