M2-Mixer

An official implementation for paper: "M2-Mixer: A Multimodal Mixer with Multi-head Loss for Classification from Multimodal Data"

Table Of Contents

• About the Project
• Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Contributing
• Authors
• Acknowledgements

About The Project

In this paper, we propose M2-Mixer, an MLP9 Mixer based architecture with multi-head loss for multimodal classification. It achieves better performances than other baseline models with the main advantage of conceptual and computational simplicity as it only uses simple operations such as matrix multiplication, data layout change, and scalar non-linearity

Built With

This code and experiments were possible due to these awesome open-source projects

• Python
• PyTorch
• PyTorch-Lightning
• wandb

Getting Started

This repository will make it possible for other researchers to reproduce the results.

Prerequisites

Python 3.10.6 is used for the experiments.

Datasets

AV-MNIST

The dataset can be downloaded from here. Don't forget to change the path in the configuration file.

MIMIC-III

The dataset is not public, hence you need to apply for access here. After getting access, you can download the dataset from here, and use the instructions of this repository to get the data in the format we used, or email the author of the repository to get the pre-processed file.

Installation

Clone the repository and change the active directory with

git clone https://github.com/bezirganyan/m2-mixer.git
cd m2-mixer

We recommend to create a new virtual environment with

python3 -m venv venv
source venv/bin/activate

Install requirements with

pip install -r requirements.txt

Usage

There are 2 scripts that can be used for training and testing: run.py and run_for_significance.py. The first one runs the code once, the second one runs several ones (default=10)) without resetting the random seed, to get the error bars. We will show the usage examples on run.py, as the interface is mostly the same for the run_for_significance.py.

To run you need to have the datasets, and the configuration files. You can get the ready configuration files from cfg/ directory. Please refer to the Datasets section for downloading the datasets.

To run run.py for training:

python run.py -c [CONFIG PATH] -n "Experiment Name"

To run for test:

python run.py -c [CONFIG PATH] -n "Experiment Name" -m test -p [CHECKPOINT PATH]

We use wandb for logging the results, hence you need a wandb installed and configured. If you don't have that you can use the --disable-wandb flag to run the code.

Checkpoints

You can download the checkpoints from the following links:

mimic_m2-mixer_H.ckpt
mimic_m2-mixer_LC.ckpt
avmnist_m2-mixer_S.ckpt
avmnist_m2-mixer_M.ckpt
avmnist_m2-mixer_B.ckpt

Citation

If you use this work in your paper please cite using:

@INPROCEEDINGS{10386252,
  author={Bezirganyan, Grigor and Sellami, Sana and Berti-ÉQuille, Laure and Fournier, Sébastien},
  booktitle={2023 IEEE International Conference on Big Data (BigData)}, 
  title={M2-Mixer: A Multimodal Mixer with Multi-head Loss for Classification from Multimodal Data}, 
  year={2023},
  volume={},
  number={},
  pages={1052-1058},
  keywords={Training;Costs;Computational modeling;MIMICs;Computer architecture;Multitasking;Data models;Multimodal Deep Learning;Multimodal Data Fusion;Multimodal Classification;MLP;MLP-mixer},
  doi={10.1109/BigData59044.2023.10386252}}