Abstract: An essential step in FrameNet Semantic Role Labeling is the Frame Identification (FrameId) task, which aims at disambiguating a situation around a predicate. Whilst current FrameId methods rely on textual representations only, we hypothesize that FrameId can profit from a richer understanding of the situational context. Such contextual information can be obtained from common sense knowledge, which is more present in images than in text. In this paper, we extend a state-of-the-art FrameId system in order to effectively leverage multimodal representations. We conduct a comprehensive evaluation on the English FrameNet and its German counterpart SALSA. Our analysis shows that for the German data, textual representations are still competitive with multimodal ones. However on the English data, our multimodal FrameId approach outperforms its unimodal counterpart, setting a new state of the art. Its benefits are particularly apparent in dealing with ambiguous and rare instances, the main source of errors of current systems. For research purposes, we release (a) the implementation of our system, (b) our evaluation splits for SALSA 2.0, and (c) the embeddings for synsets and I MAGINED words.
Please use the following citation:
@InProceedings{botschen2018multimodal,
title = {{Multimodal Frame Identification with Multilingual Evaluation}},
author = {Botschen, Teresa and Gurevych, Iryna and Klie, Jan-Christoph and Mousselly Sergieh, Hatem and Roth, Stefan},
publisher = {Association for Computational Linguistics},
booktitle = {Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018)},
pages = {to appear},
month = jun,
year = {2018},
location = {New Orleans, USA},
}
Contact person: Teresa Botschen, botschen@aiphes.tu-darmstadt.de ; Jan-Christoph Klie, klie@ukp.informatik.tu-darmstadt.de
https://www.ukp.tu-darmstadt.de/
Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.
sudo apt install libarchive-dev
Use a virtual environment with Python 3.
virtualenv --python=/usr/bin/python3 --no-site-packages venv
source venv/bin/activate
pip install -r requirements.txt
The following describes the different resources used. For converting FrameNet, follow the README in framenet-hermann-corenlp. The other data itself can automagically be downloaded with
python download_data.py
As it takes a while to load the embeddings, datasets are pregenerated. In order to do that, run
python preprocess_data.py
after downloading. Comment out the parts that you do not need.
Downloading SALSA requires a password which has to be added in download_data.py. This can be obtained from the SALSA project website at http://www.coli.uni-saarland.de/projects/salsa/corpus/
The frame file for SALSA has two issues which require fixing: it contains an unescaped ampersand (Laut 1&1 -> Laut 1&1) and an unclosed tag (</frames> has to be added to the last line). This has been taken care of in the downloading script. In case this does not work, these changes have to be performed by hand.
Our SALSA splits can be found under https://fileserver.ukp.informatik.tu-darmstadt.de/naacl18-multimodal-frame-identification/salsa_splits.txt .
Each corpus consists of two files, one for the sentences and one for the annotations. The documentation can be found in the formats.py file which can read and load them for you.
| Name | Description | How to get |
|---|---|---|
| CORPUS_HERMANN | FrameNet 1.5 with Das split and Hermann validation | Use the code from framenet-hermann-corenlp (and read the respective documentation) to get these. |
| CORPUS_SALSA2_DE | SALSA 2.0 | Go to http://www.coli.uni-saarland.de/projects/salsa/corpus/request/salsa-corpus-request.cgi and request access. Open download_data.py, change the username and password in download_salsa_file to the credentials you got from the SALSA website. |
This project uses different pretrained word embeddings. To use these, put them into the data/raw/embeddings folder (or adjust the paths in the script) and run preprocess_data.py . Adding new embeddings can be done by defining them in globals.py. Make sure to convert all embeddings to word2vec format (first line contains number of vectors and vector size). If they can be easily downloaded from the Internet, it is recommended to write a download function and put it into download_data.py.
.w2vt : word2vec file text format. Has header with number of rows and embedding dimensions, then embeddings as text one per row
.w2vb : word2vec file binary format. Has header with number of rows and embedding dimensions, then follow the embeddings
.glove : GLOVE file format. Has no header, needs to be converted to word2vec format
| Name | URL | Description |
|---|---|---|
| EMBEDDINGS_GLOVE_EN_6B_300d | https://nlp.stanford.edu/projects/glove/ | Glove trained on Wikipedia 2014 + Gigaword 5 |
| EMBEDDINGS_IMAGINED_VANILIA_GLOVE_IKRL | https://fileserver.ukp.informatik.tu-darmstadt.de/naacl18-multimodal-frame-identification/ | Imagined method (Collell et al., 2017) applied to glove embeddings and our visual embeddings for synsets |
| Name | URL | Description |
|---|---|---|
| EMBEDDINGS_GERM_EVAL_DEPS_MIN_5 | https://www.ukp.tu-darmstadt.de/research/ukp-in-challenges/germeval-2014/ | Pre-trained word vectors for German, trained on Wikipedia and newswire |
| EMBEDDINGS_IMAGINED_VANILIA_GERMAN_5MIN_IKRL | https://fileserver.ukp.informatik.tu-darmstadt.de/naacl18-multimodal-frame-identification/ | Imagined method (Collell et al., 2017) applied to german embeddings and our visual embeddings for synsets |
| Name | URL | Description |
|---|---|---|
| EMBEDDINGS_IKRL_NORM | https://fileserver.ukp.informatik.tu-darmstadt.de/naacl18-multimodal-frame-identification/ | Applied VGG-m-128 (Chatfield et al., 2014) to images of the WN9-IMG dataset (Xie et al., 2017) |
Best is to look at resources.py which defines all the paths to each folder. It creates the empty directories on the first call.
.corpora : Corpora folder
.datasets : Datasets folder
.embeddings : Embeddings folder
.lexicon : Lexicon folder
.raw : Raw files that need processing are stored here
Experiments are defined in experiments.py. They can be specified in a simple manner, just follow the examples there. Models can be saved and loaded. See experiments.py for saving and evaluate_model.py for loading.
Two classifiers are currently supported, the multilayer perceptron and the recurrent neural network. These can be found in mlp_classifier.py and rnn_classifier.py respectively.
Features are defined as Python enums. If adding a new feature, define a new constant and handle it in the loading and model building functions for the respective classifier.
Datasets are pre-generated and loaded for experiments. Their definition can be found in dataset.py. When adding new fields, make sure to add them to the get_state and set_state so that they are properly handled while pickling.