AnnotateGPT

AnnotateGPT is a Chrome extension designed to facilitate the peer review process for conference and journal submissions. Leveraging the capabilities of Large Language Models (LLMs), this tool enables reviewers to evaluate manuscripts through annotations directly within their browser. By offering a criterion-based approach, AnnotateGPT effectively addresses the challenges posed by the escalating volume of research submissions, alleviating the strain on traditional peer-review systems.

Main features:

• Ease to install. AnnotateGPT is just two-click away. If you have previously installed any browser extension, installing AnnotateGPT is a doddle! Last but not least, AnnotateGPT is being certified by Chrome, before being uploaded to its web store. So, no security leaks.
• Color-coding highlighting. You define a Review Model (e.g., originality, legibility and so on), each model’s attribute is mapped to a colour to be used during highlighting at review time,
• Qualify highlighting. Highlights can be associated with comments, grades (strengths and weaknesses) or references to the literature. Your comments would undertake a sentiment analysis to avoid offensive wordings.
• Canvas view. Have a global picture of the review so far. The canvas is plotted along with the attributes of the review model. Gradations and highlights are shown within each plot.
• Review-draft generation. A first text draft is generated as a review head-start. Comments are placed by the manuscript quotes for authors to easily spot the rationales for the reviewer comments.
• Sharing. Data is stored locally. Yet, it can be exported as a JSON file and emailed to colleagues who can then import it into their AnnotateGPT installations. On loading the manuscript, your colleagues will see the very same view as you.

References

END-USERS MANUAL

• AnnotateGPT is designed to streamline the academic peer-review process by pre-annotating manuscripts using AI-generated highlights.

Installation and Setup

You can follow these steps to get AnnotateGPT up and running

Step 1: Requirements

Ensure your computer and you meet the minimum requirements for running Chatin.

• Google Chrome browser
• LLM API key. Chatin allows you to interact with GPT and Claude, therefore, in order to use them ensure you have an OpenAI or Anthropic API key

Step 2: Install the Chatin Extension

• Open your web browser and navigate to the AnnotateGPT extension page on the [Browser Extension Store/Marketplace].
• Click the "Add to Browser" or "Install" button to begin the installation process.
• Once the installation is complete, you will see a AnnotateGPT icon appear in your browser’s toolbar. This indicates that AnnotateGPT is successfully installed.

Step 3: Setup

Open AnnotateGPT's options panel. You can do it by right click in AnnotateGPT's browser’s toolbar icon.

In this page, you have to complete the following information:

1. Select your Large Language Model and include your API key

After that, open the "details" panel of the web extension. You can find this page opening "Manage extensions" and then you can find a Details button in AnnotateGPT. Once, in the details page, you have to Allow access to URL files.

Finally, open a PDF file from you local system in the web browser a click on the AnnotateGPT icon. The extension must be executed and you should visualize a sidebar at the left of the page.

If you encounter any difficulties reaching this point, please don't hesitate to contact xabier.garmendiad@ehu.eus.

Getting Started

You can create a new review criteria clicking on the arrow shown in the next figure:

Here, AnnotateGPT provides different options to create a new review criteria to carry out the review

• Create review model. This option creates an empty review criteria which you can later complete creating different criteria with their name and criterion.
• Import review model. This option allows to import a previously exported review criteria created in AnnotateGPT
• Standard review model. This option provides a set of review criteria templates derived from the The ACM SIGSOFT Empirical Standards for Software Engineering.

Once you select one of the options, you will have to give a name for you review criteria, for example, the name of the conference/journal the paper belongs.

Then, the new review criteria will be displayed in the sidebar:

Features and Functionality

We distinguish three different components while reviewing with AnnotateGPT: Annotation, Review criteria and the Review.

Annotation management

Annotion can be performed by two different actors: the reviewer and the LLM.

• Annotations by the reviewer. The user can create their own annotations selecting text in the PDF and clicking on the criterion button in the sidebar.

• Annotations by the LLM. The user can ask the LLM to annotate the criterion.

LLMs provides a number of annotations, which can be established in the options page of the extension. If you double click in the created annotations, you can see details of the annotation which contain the sentiment associated to the annotation and a comment field.

Is it also possible to include the sentiment and comments for the annotations created by the reviewer. The comments can be written manually or you can ask the LLM about the annotated content. This operations can be performed by doing right click on the annotation:

These are the different operations that can be performed with the annotations:

• Clarify. Ask an open question to the LLM to clarify some doubts about annotation content.
• Fact checking. Asks the LLM about the veracity of the text in the annotation.
• Social judging. Asks the LLM the rise of social concerns about the annotated text.
• Render. This displays the same window as with doing the double click.
• Delete. The annotation can be removed.

The questions asked to the LLM display the following window with different options:

• Redo. Repeats the question to the LLM.
• Save. The answer is save in the annotation comment.

Review Criteria management

Criterion can be created from the sidebar. The next figure shows which are the button (1) to create a criteria group and (2) to create a new criterion.

When you are creating a new criterion you have to provide a name and a description, which are important for asking later the LLM.

You can modify criterion's name by double clicking in its button. It will show the following window where you can modify the label and the description:

Operations with review criteria

If you click on a button with the right button four options appear:

These are the different operations that can be performed with the annotations:

• Annotate. Ask the LLM to create annotations.
• Compile. Asks the LLM to provide a summary of the criterion, it uses the PDF content and the annotations.
• Viewpoints. Asks the LLM to provide different viewpoints about the current criterion assessment
• Render. This displays the assessments made for the review criteria, that is, the annotations, and the compilation and viewpoints if they were saved when were returned by the LLM.

In the following image you can see an example of using Compile:

• Redo. Repeats the question to the LLM.
• Save answer. The answer is save in the annotation comment.

Save answer can be visualized by the render option and are also included in the review draft.

Review management

AnnotateGPT provide different tools for managing the review. While you are working with AnnotateGPT you can also ask for review drafts or consult the current status through a Canvas view.

Review drafts

There are two types of review drafts: review by criteria and review by sentiment. They are exported in html format and can be visualized in a browser. Here is a review example for a paper associated with AnnotateGPT [Example]

Canvas view

The canvas view allows to track the current status of the review by the annotations and their sentiment.

Delete all annotations

This option deletes all the annotations of the current manuscript.

Settings

Here you can access the options page and the prompt configuration page where you can customize and adapt the prompts.

Support and Resources

• You can ask support in the following mail: xabier.garmendiad@ehu.eus

• Here there is a video demo: https://youtu.be/lGInf5cqZ7Q?si=8Ks48PHibMiFXli9

Available LLMs

• OpenAI. GPT4
• Anthropic. Claude2.0

DEVELOPERS MANUAL

Architecture Overview

AnnotateGPT is a browser extension and it is developed with web development technologies: JavaScript, Node and Gulp. It is developed over the last Manifest version (Manifest version 3)

The web extension follows the standard web browser extension architecture with content script modules to manage the web content and background scripts to manage the local storage and LLM communications.

Development Environment

To develop from the source code, you will need to install node and gulp to build the extension. This are the required versions

• Node v12.22
• Gulp v4

Library Documentation

AnnotateGPT uses Langchain for LLM communications through the LangChain JS library

Code Structure

The artifact code is organized as follows:

These are the main components:

• images. This folder contains the images within the browser extension.
• pages. This folder contains the html files of the extension.
• resources. This folder contains mind map templates and the pdf.js library to process pdfs.
• scripts. This is the main component, it contains the scripts to make the extension work. Based on a web extension architecture this is the main classes:
  • Content Script: contentScript.js, this is the script that is executed when MindMeister is accessed
  • Service worker: background.js, this script initializes all the background scripts to enable the communications
  • Options file: options.js, this is the script executed when the option page is opened
  • The rest of scripts are organized in the following folders. - contentScript folder. This folder contains the scripts that are executed in the PDF view. - contentAnnotator -> TextAnnotator is the script where annotations are manage. - specific -> here you can find the functions associated with the review criteria. - TagManager.js aims to create the review criteria buttons.
    • llm. Here you can find the scripts aimed to establish the communications with Langchain.

Building and Testing

To compile the project, you only need to execute the following lines, but make sure you have the node and gulp versions specified in the Development Enviroment section.

$ npm install
$ gulp default

Usage

Run $ gulp --watch and load the dist-directory into chrome.

Entryfiles (bundles)

There are two kinds of entryfiles that create bundles.

1. All js-files in the root of the ./app/scripts directory
2. All css-,scss- and less-files in the root of the ./app/styles directory

Tasks

Build

$ gulp

Option	Description
--watch	Starts a livereload server and watches all assets. To reload the extension on change include livereload.js in your bundle.
--production	Minifies all assets
--verbose	Log additional data to the console.
--vendor	Compile the extension for different vendors (chrome, firefox, opera, edge) Default: chrome
--sourcemaps	Force the creation of sourcemaps. Default: !production

pack

Zips your dist directory and saves it in the packages directory.

$ gulp pack --vendor=firefox

Version

Increments version number of manifest.json and package.json, commits the change to git and adds a git tag.

$ gulp patch      // => 0.0.X

or

$ gulp feature    // => 0.X.0

or

$ gulp release    // => X.0.0

Globals

The build tool also defines a variable named process.env.NODE_ENV in your scripts. It will be set to development unless you use the --production option.

Testing

Contribution Guidelines

To contribute please contact xabier.garmendiad@ehu.eus.

Roadmap and Future Plans

This project has been developed in a research setting. The aim is to explore the benefits of AI for problem analysis so future plans seek to evaluate the artifact in real settings

Installation

Use node 10.24.1

$ npm install