Persuasion Mode Classification with VLMs

Code for evaluating Vision-Language Models (VLMs) on Logos, Ethos, and Pathos detection — the three modes of Aristotle's persuasion triangle — using the ImageArg dataset. The pipeline runs a VLM over tweet + image pairs, asks it to judge which persuasion mode(s) the image contributes, and scores the predictions against ground truth with precision/recall/F1.

Abstract

Vision Language Models (VLMs) have demonstrated exceptional performance across various tasks. However, they have not yet been thoroughly evaluated on more complex tasks. The Persuasion Model, conceived by Aristotle, resembles a triangle shape, which highlights its inherent challenges related to personal biases. To assess the progress of VLMs on these complex tasks, we use the ImageArg dataset, focusing on the Logos, Ethos, and Pathos detection tasks. Our findings indicate that models from the Qwen family achieve improved F1 scores, with Qwen3 performing exceptionally well on the Logos and Pathos tasks, while Qwen2 exhibits competitive performance on the more complex Ethos detection task. We release the code to foster research in this direction.

What the code does

Given a dataset of tweets (each paired with an image), the script:

1. Loads a Qwen2-VL or Qwen3-VL model.
2. For each sample, builds a chat-style prompt (tweet text + image + the persuasion-mode question definitions) and asks the model to return a JSON object of the form {"logos": "yes"/"no", "pathos": "yes"/"no", "ethos": "yes"/"no"}.
3. Parses the model's JSON output, one file per tweet_id, plus a combined result.json.
4. Compares predictions against the ground-truth persuasion_mode labels already present in the dataset and reports precision/recall/F1 per category (and macro-averaged), printed to the console and saved as metrics.json.

Requirements

• Python 3.10+
• A CUDA GPU is strongly recommended (VLM inference on CPU will be very slow)
• Packages:

pip install torch transformers accelerate qwen-vl-utils pillow

Note on transformers version: Qwen3-VL support was added to transformers relatively recently. If Qwen3VLForConditionalGeneration fails to import, upgrade to the latest transformers release (or install from source per the Qwen3-VL model card).

Dataset format

The script expects a JSON file that is a list of objects. Each object needs at minimum:

Field	Type	Description
tweet_id	str/int	Unique identifier for the sample
final_prompt	str	The fully-formed text prompt sent to the model (tweet text + task instructions + output-format spec)
media_url	str	URL or local path to the image associated with the tweet
persuasion_mode	object	Ground-truth labels: {"logos": "yes"/"no", "pathos": "yes"/"no", "ethos": "yes"/"no", "logos_reason": [...], "pathos_reason": [...], "ethos_reason": [...]}

Any extra fields (tweet_text, tweet_url, tweet_time, stance, content, argumentConfidenceScore, persuasiveness, *_reason, etc.) are fine — the script only reads tweet_id, final_prompt, media_url, and (for evaluation) persuasion_mode.{logos,pathos,ethos}.

Example entry:

{
    "tweet_id": "1372006571165761538",
    "tweet_text": "...",
    "media_url": "https://pbs.twimg.com/media/EwpYtHCXMAc5iHy.jpg",
    "persuasion_mode": {
        "logos": "yes",
        "pathos": "no",
        "ethos": "no",
        "logos_reason": ["..."],
        "pathos_reason": [],
        "ethos_reason": []
    },
    "final_prompt": "Tweet text: ...\n\nWe aim to study the ARGUMENTATIVE ROLES OF IMAGES in tweets. ... Return ONLY a valid JSON object in the following format:\n{\n\"logos\": \"yes\" or \"no\",\n\"pathos\": \"yes\" or \"no\",\n\"ethos\": \"yes\" or \"no\"\n}"
}

final_prompt should already contain the operational definitions of logos/pathos/ethos and the "return only JSON" instruction — the script sends this text as-is, alongside the image, to the model.

Usage

Basic run (model type is auto-detected from the model name — any name containing "qwen2" or "qwen3" works):

python argument_classification.py \
    --model_name Qwen/Qwen3-VL-8B-Instruct \
    --dataset_path dataset/gun_control.json

Run the Qwen2-VL baseline on the same dataset:

python argument_classification.py \
    --model_name Qwen/Qwen2-VL-7B-Instruct \
    --dataset_path dataset/gun_control.json

CLI arguments

Argument	Default	Description
--model_name	(required)	HuggingFace model name or local path, e.g. Qwen/Qwen3-VL-8B-Instruct
--model_type	auto	qwen2, qwen3, or auto (inferred from --model_name). Set explicitly if your model path doesn't contain "qwen2"/"qwen3".
--dataset_path	dataset/gun_control.json	Path to the input dataset (see format above)
--output_file	outputs/<model_name>/result.json	Where predictions are saved
--batch_size	16	Currently unused by the per-sample inference loop (reserved for a future batched implementation)
--ground_truth_path	(same as --dataset_path)	JSON file with ground-truth persuasion_mode labels, keyed by tweet_id. Defaults to the input dataset itself, since it already carries the labels.
--metrics_output_file	<output_file directory>/metrics.json	Where the precision/recall/F1 report is saved

Output

• result.json — a list of {"tweet_id": ..., "persuasion_mode": {"logos": "yes"/"no", "pathos": "yes"/"no", "ethos": "yes"/"no"}} predictions.
• <tweet_id>.json — one file per sample with the raw parsed JSON prediction (written as inference progresses).
• metrics.json — precision, recall, F1, and support for each of logos/pathos/ethos plus a macro average, e.g.:

{
    "logos": {"precision": 0.633, "recall": 0.813, "f1": 0.709, "support": 32, "tp": 26, "fp": 15, "fn": 6, "tn": 20, "num_evaluated": 67},
    "pathos": {"precision": 0.538, "recall": 1.0, "f1": 0.696, "support": 41, "tp": 41, "fp": 35, "fn": 0, "tn": 12, "num_evaluated": 88},
    "ethos": {"precision": 0.236, "recall": 0.875, "f1": 0.365, "support": 8, "tp": 7, "fp": 23, "fn": 1, "tn": 35, "num_evaluated": 66},
    "macro_avg": {"precision": 0.469, "recall": 0.896, "f1": 0.590},
    "num_common_tweet_ids": 100
}

The same summary is also printed to stdout at the end of the run.

Results

Performance of persuasion mode classification across Logos, Pathos, and Ethos (precision, recall, F1). BASE/T-M denote the best benchmark model reported for that task in the original ImageArg paper. Bold marks the best score per metric within each persuasion mode.

Persuasion Mode	Model	Precision ↑	Recall ↑	F1 ↑
Logos	BASE (bm)	0.405	1.000	0.575
	Qwen2	0.596	0.885	0.709
	Qwen3	0.633	0.813	0.709
Pathos	BASE (bm)	0.554	1.000	0.712
	Qwen2	0.568	0.969	0.714
	Qwen3	0.538	1.000	0.696
Ethos	T-M (bm)	0.168	0.817	0.272
	Qwen2	0.195	0.550	0.277
	Qwen3	0.236	0.875	0.365

Qwen models achieved the strongest overall F1 across all three persuasion modes. Qwen3 performs best on Logos and Ethos; Qwen2 and Qwen3 are effectively tied on Pathos, with Qwen2 marginally ahead on F1.

Conclusion

We evaluate the effectiveness of Vision-Language Models in the complex task of detecting persuasion modes. Persuasion modes — Logos, Pathos, and Ethos — overlap within a persuasion triangle, complicating the influence of inherent personal biases. For our evaluation, we selected the ImageArg dataset, where the authors reported moderate annotator agreement due to the challenges associated with the task. To assess model performance, we focused on two generation models from the Qwen family to understand how these models are evolving to tackle this complex challenge. Overall, we found that Qwen models achieved the highest F1 scores when compared to benchmark models. Specifically, Qwen3 demonstrated superior performance in identifying the Logos and Pathos modes, while Qwen2 performed competitively in the more difficult Ethos detection task.

Future research could explore the effectiveness of these VLMs in detecting more nuanced persuasion modes, such as Ad Hominem and Appeal to Authority, as well as their application to related tasks such as Fallacy Detection.

Repository structure

.
├── argument_classification.py   # main pipeline: inference + evaluation
├── dataset/
│   └── gun_control.json         # ImageArg-style dataset (tweet + image + ground-truth labels)
├── outputs/
│   └── <model_name>/
│       ├── result.json          # all predictions
│       ├── metrics.json         # precision/recall/F1 report
│       └── <tweet_id>.json      # per-sample raw prediction
└── cache/                       # HuggingFace model/processor cache (auto-created)