Multimodal Sentiment Analysis

This repository records the proposed Multimodal Sentiment Analysis on Pytorch Framework.

“TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis”, paper has been accepted by Knowledge-Based System.

Trackers

Here is a list of the improved trackers.

MISA: Misa: Modality-invariant and-specific representations for multimodal sentiment analysis, ACMM, 2020. [Paper]
MMIM: Progressive image deraining networks: A better and simpler baseline, arXiv, 2021. [Paper]
HyCon: Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis, IEEE Transactions on Affective Computing, 2022. [Paper]
TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis, Pattern Recognition, 2023. [Paper]
AOBERT: All-modalities-in One BERT for multimodal sentiment analysis, 2023. [Paper]

In this study, we develop a transformer model equipped with a modality-bound learning mechanism to extract unified affective information from multiple modalities. Our proposed structure and method hold potential for broader applications in other fields related to multimodal interaction and fusion, such as multimodal disease diagnosis.
If you have any question, please set the issue and PR, we will process it as soon as possible~

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
CMU-MultimodalSDK-main		CMU-MultimodalSDK-main
CMU_MultimodalSDK		CMU_MultimodalSDK
__pycache__		__pycache__
utils		utils
Jack_model.py		Jack_model.py
Jack_model_AOTrans.py		Jack_model_AOTrans.py
Jack_model_AOTrans_CLIP.py		Jack_model_AOTrans_CLIP.py
Jack_model_AOTrans_CLIP_NOCom.py		Jack_model_AOTrans_CLIP_NOCom.py
Jack_model_AOTrans_CLIP_NOCom_0531.py		Jack_model_AOTrans_CLIP_NOCom_0531.py
Jack_model_SwimT.py		Jack_model_SwimT.py
MultimodalDiary.md		MultimodalDiary.md
README.md		README.md
config.py		config.py
create_dataset.py		create_dataset.py
data_loader.py		data_loader.py
jack_transformer.py		jack_transformer.py
jack_transformer_share.py		jack_transformer_share.py
jack_transformer_share_0530.py		jack_transformer_share_0530.py
jack_transformer_share_0531.py		jack_transformer_share_0531.py
network.py		network.py
new_models.py		new_models.py
new_models_original.py		new_models_original.py
requirements.txt		requirements.txt
solver.py		solver.py
solver0531.py		solver0531.py
solver_AOTrans_CLIP.py		solver_AOTrans_CLIP.py
solver_original.py		solver_original.py
swin_transformer.py		swin_transformer.py
test.py		test.py
train.py		train.py