Code for Mimicking Word Embeddings using Subword RNNs (EMNLP 2017)
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.


Code for Mimicking Word Embeddings using Subword RNNs (EMNLP 2017) and subsequent experiments.

I'm adding details to this documentation as I go. When I'm through, this comment will be gone.


Given a word embedding dictionary (with vectors from, e.g. FastText or Polyglot or GloVe), Mimick trains a character-level neural net that learns to approximate the embeddings. It can then be applied to infer embeddings in the same space for words that were not available in the original set (i.e. OOVs - Out Of Vocabulary).


Please cite our paper if you use this code.

  title={Mimicking Word Embeddings using Subword RNNs},
  author={Pinter, Yuval and Guthrie, Robert and Eisenstein, Jacob},
  booktitle={Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},


The main dependency for this project is DyNet. Get it here.

  • As of November 22, 2017, the code complies with Dynet 2.0. You may access the 1.0 version code via the commit log.

Create Mimick models

The mimick directory contains scripts relevant to the Mimick model: dataset creation, model creation, intrinsic analysis (see readme within). The models directory within contains models trained for all 23 languages mentioned in the paper. If you're using the pre-trained models, you don't need anything else from the mimick directory in order to run the tagging model. If you train new models, please add them here via pull request!

  • December 12, 2017 note, deprecating the one below: pre-trained model are now in DyNet 2.0 format (and employ early-stopping). The 1.0-compatible models are still available in a subdirectory.
  • November 22, 2017 note: the pre-trained models were saved in DyNet 1.0 format. It is still possible to load them using the old_load() function in mimick/, but it could be somewhat of a pain. Another option is to use DyNet 1.0 and an old version of this codebase.

CNN Version (November 2017)

As of the November 22 PR, there is a CNN version of Mimick available for training. It is currently a single-layer convolutional net (conv -> ReLU -> max-k-pool -> fully-connected -> tanh -> fully-connected) that performs the same function as the LSTM version. What is it good for? Wait for the paper 😎

Tag parts-of-speech and morphosyntactic attributes using trained models

The root directory of this repository contains the code required to perform extrinsic analysis on Universal Dependencies data. Vocabulary files are supplied in the vocabs directory.

The entry point is, which can use tagging datasets created using the script. Note that accepts pre-trained Word Embedding models via text files with no header. For Mimick models, this exact format is output into the path in mimick/ script's --output argument. For Word2Vec, FastText, or Polyglot models, one can create such a file using the scripts/ script that accepts a model (.pkl or .bin) and the desired output vocabulary (.txt).