Skip to content

A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis

License

Notifications You must be signed in to change notification settings

jbdel/MOSEI_UMONS

Repository files navigation

  
    ACL2020

Pytorch implementation of the paper "A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis"
Challenge-HML Best Paper Award

@inproceedings{delbrouck-etal-2020-transformer,
    title = "A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis",
    author = "Delbrouck, Jean-Benoit  and
      Tits, No{\'e}  and
      Brousmiche, Mathilde  and
      Dupont, St{\'e}phane",
    booktitle = "Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)",
    month = jul,
    year = "2020",
    address = "Seattle, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.challengehml-1.1",
    doi = "10.18653/v1/2020.challengehml-1.1",
    pages = "1--7"
}

Model

The model Model_LA is the module used for the UMONS solution to the MOSEI dataset using only linguistic and acoustic inputs.
Results can be replicated at the following Google Colab sheet: Open In Colab

Environement

Create a 3.6 python environement with:

torch              1.2.0    
torchvision        0.4.0   
numpy              1.18.1
scikit-learn       0.24.2

We use GloVe vectors from space. This can be installed to your environement using the following commands :

wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz

Data

Download data from here.
Unzip the files into the 'data' folder
More informations about the data can be found in the 'data' folder

Training

To train a Model_LA model on the emotion labels, use the following command :

python main.py --model Model_LA --name mymodel --task emotion --multi_head 4 --ff_size 1024 --hidden_size  512 --layer 4 --batch_size 32 --lr_base 0.0001 --dropout_r 0.1

Checkpoints are created in folder ckpt/mymodel

Argument task can be set to emotion or sentiment. To make a binarized sentiment training (positive or negative), use --task_binary True

Evaluation

You can evaluate a model by typing :

python ensembling.py --name mymodel

The task settings are defined in the checkpoint state dict, so the evaluation will be carried on the dataset you trained your model on.

By default, the script globs all the training checkpoints inside the folder and ensembling will be performed.

Results:

Results are run on a single GeForce GTX 1080 Ti.
Training performances:

Modality Memory Usage GPU Usage sec / epoch Parameters Checkpoint size
Linguistic + acoustic 320 Mb 2400 MiB 103 ~ 33 M 397 Mb
Linguistic + acoustic + vision

You should approximate the following results :

Task Accuracy val test test ensemble epochs
Sentiment-7 43.61 43.90 45.36 6
Sentiment-2 82.30 81.53 82.26 8
Emotion-6 81.21 81.29 81.48 3

Ensemble results are of max 5 single models
7-class and 2-class sentiment and emotion models have been train according to the instructions here.

Pre-trained checkpoints:

Result Sentiment-7 ensemble is obtained from these checkpoints : Download Link
Result Sentiment-2 ensemble is obtained from these checkpoints : Download Link
Result Emotion ensemble is obtained from these checkpoints : Download Link

License

The source code for the site is licensed under the MIT license, which you can find in the LICENSE file.

About

A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages