Large Language Models as Span Annotators

This repository contains data and resources for the paper "Large Language Models as Span Annotators".

Repository Structure

Inputs

The inputs directory contains the source data:

• d2t-eval: Data-to-text inputs organized by domain (football, gsmarena, openweather). The examples are implicitly indexed by their position in the list.

• mt-eval: A sample from the WMT24 dataset (sampled ids provided in utils/wmt_sample.csv).

The inputs for the propaganda task are identical to the outputs (see below).

Outputs Directory

The outputs directory contains the outputs:

• d2t-eval: Outputs generated by LLMs (phi3-5, gpt4o, gemma2, llama3-3).
• mt-eval: Translations of the sampled inputs from the WMT24 dataset.
• propaganda: Texts from the test split from the Propaganda Techniques Corpus.

Each output file is a JSONL file with the following structure:

{
  "dataset": [dataset_id], 
  "split": [split_id],
  "setup_id": [setup_id],
  "example_idx": [example_index], 
  "output": [output_text]
}

Annotations Directory

The annotations directory contains the span annotations:

• model: Collected annotations from LLMs.
• human: Reference annotations from human annotators.

config.yaml

Configuration files contain:

• For human annotations:

  • annotation_span_categories: Categories for span annotation with name and description.
  • annotator_instructions: Instructions for human annotators.
  • annotation_granularity: Level of annotation (words / characters.)
  • annotation_overlap_allowed: Whether spans can overlap
  • campaign_orig_id: Original id of the internal campaign.

• For model annotations:

  • annotation_span_categories: Categories for span annotation with name and description.
  • prompt_template: The template used to prompt the model.
  • model: The model identifier (e.g., deepseek-r1:70b)
  • model_args: Additional model parameters
  • campaign_orig_id: Original id of the internal campaign.

annotations.jsonl

Each line in an annotations.jsonl file contains a JSON object with:

• dataset: The dataset id
• split: The split id
• setup_id: The setup id
• example_idx: The example index
• annotations: Array of span annotations, each with:
  • type: Integer indicating the annotation category (0-indexed)
  • text: The text of the annotated span
  • start: Character position where the span starts
  • id: Unique identifier for the annotation
• annotator_group: Annotator group (a single output can have annotation sets from different groups).
• thinking_trace (only DeepSeek-R1 and Llama 3.3-CoT): Reasoning trace of the model.

Installing factgenie

For visualizing the data and replicating evaluation, factgenie needs to be installed.

Steps

1. Initialize the factgenie submodule:

   git submodule update --init --recursive

2. Install factgenie as an editable package:

   cd factgenie
   pip install -e .

Visualizing the Data

To visualize the data and annotations using factgenie, make sure that factgenie is installed (see Installing factgenie section above).

Steps

1. Convert the data to factgenie format:

   python utils/convert_to_factgenie.py

2. Start the factgenie server:

factgenie run --port=8890 --host=127.0.0.1

Then open your browser and navigate to http://127.0.0.1:8890 to view the data and annotations.

In the web interface, you'll be able to:

• Browse through different annotation campaigns.
• View the span annotations overlaid on the text.
• Compare annotations from different sources (human vs. model).

Evaluation

To replicate our results, you can use the provided evaluation scripts. The main logic is in the script scripts/run_eval.sh. Make sure factgenie is installed (see Installing factgenie section above).

The evaluation script supports multiple commands for different tasks and datasets:

cd scripts
chmod +x run_eval.sh

# Run specific evaluations (examples)
./run_eval.sh d2t_test
./run_eval.sh mt_zeroshot
./run_eval.sh propaganda_zeroshot

# Run all evaluations
./run_eval.sh all

# See all available commands
./run_eval.sh help

Results will be saved in the scripts/evaluation/results/ directory in CSV format.