Multiple Languages and Modalities (MLM) is a dataset consisting of text in three languages (EN, FR, DE), images, location data, and triple classes. The resource is designed to evaluate the strengths of multitask learning systems in generalising on diverse data. The paper defines a benchmark evaluation consisting of the following tasks:
- Cross-modal retrieval
- Location estimation
Additional details on the resource and benchmark evaluation are available at the MLM website: http://cleopatra.ijs.si/goal-mlm/ IR+LE is an architecture for a multitask learning system designed as a baseline for the above benchmark. The pipeline for cross-modal retrieval extends an approach proposed by Marin et al: http://im2recipe.csail.mit.edu/im2recipe-journal.pdf.
Multitask IR+LE Framework
Python version >= 3.7
PyTorch version >= 1.4.0
# clone the repository
git clone https://github.com/GOALCLEOPATRA/MLM.git
cd MLM
pip install -r requirements.txtDownload the dataset hdf5 files from here and place them under the data folder.
Multitask Learning (IR + LE)
python train.py --task mtlCross-modal retrieval task
python train.py --task irLocation estimation task
python train.py --task leFor setting other arguments (e.g. epochs, batch size, dropout), please check args.py.
Multi-task Learning (IR + LE)
python test.py --task mtlCross-modal retrieval task
python test.py --task irLocation estimation task
python test.py --task leAll logs and checkpoints will be saved under the experiments folder.
The repository is under MIT License.
@inproceedings{armitage2020mlm,
title={Mlm: a benchmark dataset for multitask learning with multiple languages and modalities},
author={Armitage, Jason and Kacupaj, Endri and Tahmasebzadeh, Golsa and Maleshkova, Maria and Ewerth, Ralph and Lehmann, Jens},
booktitle={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management},
pages={2967--2974},
year={2020}
}