HTBM

This is the official repository of
Hybrid Transformer with Bridge Layers and Mixed Feature Fusion Network for Multimodal Sentiment Detection
(Pattern Recognition Letters, 2026)

Overview

HTBM is a multimodal sentiment detection framework designed for image-text pairs. The model employs a hybrid transformer with bridge layers to progressively integrate unimodal and cross-modal representations, and a Mixed Feature Fusion Network (MFFN) to adaptively weight unimodal and cross-modal features.

The framework is evaluated on three public datasets:

• MVSA-Single
• MVSA-Multiple
• HFM

---

Requirements

We list the main environments used in our experiments.
Please refer to requirements.txt for detailed dependencies.

python >= 3.8
pytorch >= 1.10
transformers >= 4.x

GPU training is recommended.

---

Data

Datasets

We use the following publicly available datasets:

MVSA-Single / MVSA-Multiple
http://mcrlab.net/research/mvsa-sentiment-analysis-on-multi-view-social-data/

HFM
https://github.com/headacheboy/data-of-multimodal-sarcasm-detection

Data Organization

Please organize the datasets as follows:

dataset/
└── data/
├── MVSA-single/
│   ├── 10-flod-1/
│   │   ├── train.json
│   │   ├── dev.json
│   │   └── test.json
│   └── dataset_image/
├── MVSA-multiple/
│   └── ...
└── HFM/
├── train.json
├── valid.json
├── test.json
└── dataset_image/

Dataset paths are automatically resolved in main.py according to the selected dataset.

---

Run Code

Dataset Selection

The dataset is selected via the -data_type argument:

Dataset	data_type
MVSA-Single	MVSA-single
MVSA-Multiple	MVSA-multiple
HFM	HFM

---

Training Scripts

We provide training scripts for different datasets.

MVSA-Single

sh train-single.sh 0

MVSA-Multiple

sh train-mul.sh 0

HFM

sh train-hfm.sh 0

All scripts call main.py internally and handle model configuration, dataset loading, training, validation, and testing.

---

Manual Training (Optional)

You may also run training manually:

python main.py \
  -cuda \
  -data_type MVSA-single \
  -batch_size 16 \
  -epoch 50

---

Evaluation

During training, the model is evaluated on the validation and test sets automatically.

• MVSA-Single / MVSA-Multiple: Accuracy and weighted-F1
• HFM: Accuracy and macro-F1

The best model is saved based on evaluation performance.