Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification

Implementation accompanying the paper "Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification".

Paper (arXiv): https://arxiv.org/abs/2607.09104

Dataset (Zenodo): https://doi.org/10.5281/zenodo.21367287

---

Overview

This repository implements a multimodal pipeline for semi-supervised image classification that combines visual representations extracted from multiple vision models with semantic similarity scores generated by a Large Language Model (LLM).

The proposed pipeline supports multiple visual encoders and allows experiments to be performed using different visual representations without modifying the downstream processing stages.

Supported Visual Encoders

• ResNet-152
• DINOv2 ViT-G/14
• Vision Transformer (ViT-B/16)

Main Components

• Visual feature extraction
• Image caption generation using BLIP
• Ranking-based organization of visual neighbors
• LLM-based semantic similarity scoring
• Graph construction
• Semi-supervised image classification using Simplified Graph Convolution (SGC)

---

Project Structure

.
├── notebooks/
│   ├── 01_generate_blip_captions.ipynb
│   ├── 02_preprocess_and_consolidate_outputs.ipynb
│   ├── 03_llm_similarity_scoring.ipynb
│   └── 04_graph_based_multimodal_classification.ipynb
│
├── data/
├── results/
├── requirements.txt
└── README.md

---

Requirements

• Python 3.10+
• Git
• 7-Zip (or p7zip)

Ubuntu / Debian

sudo apt update
sudo apt install p7zip-full

macOS

brew install p7zip

Windows

Install 7-Zip from:

https://www.7-zip.org/

---

Installation

Clone the repository:

git clone https://github.com/<username>/llm-gcn-semisupervised-image-classification.git

cd llm-gcn-semisupervised-image-classification

(Optional) Create a virtual environment:

python -m venv .venv

Activate it:

Linux/macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

Install the required packages:

pip install -r requirements.txt

PyTorch Geometric

The graph-based experiments require PyTorch Geometric (PyG).

Please install PyG according to your PyTorch version and CUDA configuration by following the official installation guide:

https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html

---

Dataset

The datasets required to reproduce the experiments are hosted on Zenodo due to GitHub file size limitations.

DOI:

https://doi.org/10.5281/zenodo.21367287

Download all files from inputs.zip and place them in the data/ directory before running the notebooks.

---

Running the Pipeline

Run the notebooks in the following order.

Notebook	Description
01_generate_blip_captions.ipynb	Generates image captions for the Corel5K dataset using BLIP.
02_preprocess_and_consolidate_outputs.ipynb	Creates ranking files and combines image rankings with BLIP captions.
03_llm_similarity_scoring.ipynb	Uses an LLM to estimate semantic similarity scores between image captions.
04_graph_based_multimodal_classification.ipynb	Performs graph construction, SGC training, evaluation and plot generation.

---

Notebook Inputs and Outputs

01 — Generate BLIP Captions

Input

• corel5k_images.7z

Output

• captions.csv

---

02 — Preprocess and Consolidate Outputs

Input

• corel5k_lists.txt

One ranking file:

• CNN-ResNet.7z
• corel5k_dinov2_vitg14.txt
• rks_VIT-B16_original_corel5k.txt

Output

• rank_<model>.csv
• captions_rank_<model>.csv

---

03 — LLM Similarity Scoring

Input

• captions_rank_<model>.csv

Output

• scores_<llm>_<model>_k<k>.csv

---

04 — Graph-Based Multimodal Classification

Input

• corel5k_lists.txt
• labels_corel5k.py

One feature file:

• cnn-last_linear-resnet152.npz
• features_corel5k_dinov2_vitg14.npy
• features_VIT-B16_corel5k.npy

One LLM score file:

• scores_<llm>_<model>_k<k>.csv

Output

• accuracy_vs_threshold_knn_<llm>_<model>_k<k>.pdf
• accuracy_vs_threshold_reciprocal_knn_<llm>_<model>_k<k>.pdf
• plots.zip

---

Reproducibility

To reproduce the experiments:

1. Clone this repository.
2. Install the required dependencies.
3. Download the datasets from Zenodo.
4. Place all files downloaded from inputs.zip into the data/ directory.
5. Run the notebooks sequentially.

---

Citation

If you use this repository, please cite:

@article{Piscioneri2026,
  title={Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification},
  author={Camila Piscioneri Magalhães and Lucas Pascotti Valem},
  journal={arXiv preprint arXiv:2607.09104},
  year={2026}
}