A Multitask Framework for Sentiment, Emotion and Sarcasm aware Cyberbullying Detection in Multi-modal Code-Mixed Memes (SIGIR 2022)

This is the official repository accompanying the SIGIR 2022 full paper A Multitask Framework for Sentiment, Emotion and Sarcasm aware Cyberbullying Detection in Multi-modal Code-Mixed Memes . This repository contains codebase, dataset and the annotation guidelines.

Authors

Krishanu Maity*, Prince Jha*, Sriparna Saha, Pushpak Bhattacharyya

Note : * denotes equal contribution

Dataset

1. Text Annotated with Bully, Sentiment, Emotion, Sarcasm, Target, and Harfulness Score
2. Meme Image

Model Training

1. Train the model using auxiliary tasks (Emotion, Sentiment, Sarcasm) and main task together with centralnet and simultaneosuly optimize loss function for all tasks
2. Although by training model on all tasks simultaneously makes the model overfit on training data because of complexity.
3. To overcome that we first train the model individually on all tasks using basic architecture given : (i) Bully (ii) Sentiment (iii) Emotion (iv) Sarcasm
4. Then you can load the pretrained weight from bully, sentiment, emotion and sarcasm and freeze the network parameters for these layers. and only learn the weights for Central Network to predict the output.
5. here you can decide which model should be used for training
6. Here(Emotion,Sarcasm,Sentiment,Bully) you can freeze the network parameters or you can comment if you want to finetune it again for

Annotation Guidelines

To be done

References

I would encourage to read following papers within the given sequence before moving to the code part

1. CLIP
2. The Verbal and Non Verbal Signals of Depression--Combining Acoustics, Text and Visuals for Estimating Depression Level
3. CentralNet: a Multilayer Approach for Multimodal Fusion
4. MOMENTA

Code

We have borrowed our code from following sources, you can also go visit these

1. CLIP Feature Extractor
2. centralnet code
3. MOMENTA

Citation

If you find this repository to be helpful please cite us

@article{maity2022multitask,
  title={A Multitask Framework for Sentiment, Emotion and Sarcasm aware Cyberbullying Detection from Multi-modal Code-Mixed Memes},
  author={Maity, Krishanu and Jha, Prince and Saha, Sriparna and Bhattacharyya, Pushpak},
  year={2022}
}