Relational Word Embeddings

Repository containing data (pre-trained embeddings) and code of the paper Relational Word Embeddings (ACL 2019). With the code of this repository you can learn your own relational word embeddings from a text corpus.

Pre-trained embeddings

We release the 300-dimensional embeddings trained on the English Wikipedia used in our experiments:

• FastText word embeddings [~300MB].
• Relative-init relation embeddings (symmetrical): [~6.0GB]
• Output RWE relational word embeddings (as in the reference paper): [~300MB]
• Output RWE relational word embeddings (with default parameters using the code below): [~300MB]

Note 1: All vocabulary words are lowercased.

Note 2: If you want to convert the txt files to bin, you can use convertvec.

Note 3: Underscore "_" is used to separate tokens in a multiword expression (e.g. united_states) in the corpus. Double underscore ("__") is used to separate words within the word pair (e.g. paris__france) in the relation embedding files.

Code

This repository contains the code to learn unsupervised relation word embeddings in a given text corpus.

Requirements:

• Python (version 3.6.7 tested)
• NumPy (version 1.16.4 tested)
• PyTorch (version 1.1.0 tested)

Quick start: Get your own relational word embeddings

python train_RWE.py -word_embeddings INPUT_WORD_EMBEDDINGS -rel_embeddings INPUT_RELATION_EMBEDDINGS -output OUTPUT_RWE_EMBEDDINGS

The code takes as input standard word embeddings (FastText with standard hyperparameters was used in the reference paper) and relation embeddings (i.e. embeddings for pairs of words), both in standard space-sparated txt formats (see pre-trained embeddings for exact format). As input relation embeddings we used the Relative package, mainly due to its efficiency compared to other similar methods, but any relation embeddings can be used as input. You can use the pre-trained relation embeddings included in this package or, alternatively, to learn your own Relative relation embeddings you can simply run the following command (more information in the original Relative repository):

python relative_init.py -corpus INPUT_CORPUS -embeddings INPUT_WORD_EMBEDDINGS -output OUTPUT_RELATIVE_EMBEDDINGS -symmetry true

where INPUT_CORPUS can be any tokenized corpus (English Wikipedia in our experiments).

Example usage:

A short example on how to use the RWE code:

python train_RWE.py -word_embeddings fasttext_wikipedia_en_300d.txt -rel_embeddings relative-init_symm_wiki_en_300d.txt -output rwe_embeddings.txt

Parameters

A number of optional hyperparameters can be specified in the code. Below you can find these parameters and their default values:

-hidden: Size of the hidden layer. Default: 0 (=twice the dimension of the input word embeddings)

-dropout: Dropout rate. Default: 0.5

-epochs: Number of epochs. Default: 5

-interval: Size of intervals during training. Default: 100

-batchsize: Batch size. Default: 10

-devsize: Size of development data (proportion with respect to the full training set, from 0 to 1). Default: 0.015

-lr: Learning rate for training. Default: 0.01

Example:

For example, if you would like more epochs (e.g. 10) and a higher learning rate (e.g. 0.1), you can type the following:

python train_RWE.py -word_embeddings fasttext_wiki_300d.txt -rel_embeddings relative-init_symm_wiki_en_300d.txt -output rwe_embeddings.txt -epochs 10 -lr 0.1

Note: This code has been tested on GPU for a higher speed, but could be run on CPU as well.

Reference paper

If you use any of these resources, please cite the following paper:

@inproceedings{camacho-collados-etal-2019-relational,
    title = "Relational Word Embeddings",
    author = "Camacho-Collados, Jose  and
      Espinosa Anke, Luis  and
      Schockaert, Steven",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P19-1318",
    pages = "3286--3296"
}

If you use FastText or Relative, please also cite their corresponding paper/s as well.

License

Code and data in this repository are released open-source.

Copyright (C) 2019, Jose Camacho Collados.