Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

GroC — a Pytorch implementation of GroC, a grounded compositional output model for adaptive language modeling presented at EMNLP 2020 [1]. The model has a fully compositional output embedding layer that is optionally further grounded in information from a structured lexicon (WordNet), namely semantically related words and free-text definitions. It can be applied to both conventional language modeling as well as challenging cross-domain settings with an open vocabulary.


    title = "Grounded Compositional Outputs for Adaptive Language Modeling",
    author = "Pappas, Nikolaos and Mulcaire, Phoebe and Smith, Noah A.",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    publisher = "Association for Computational Linguistics",
    url = "",
    pages = "1252--1267",
    year = "2020"


Note that this repository is based on the codebase of awd-lstm, for a general purpose language model consider using that library directly [2]. Before running the code make sure you have installed the conda environment first:

conda env create -f environment.yml


To obtain the datasets for conventional language modeling please follow the instructions from [2], e.g. the datasets can be easily obtained by running this script.


echo "- Downloading Penn Treebank (PTB)"
wget --quiet --continue
tar -xzf simple-examples.tgz``
mkdir -p penn
cd penn
mv ../simple-examples/data/ptb.train.txt train.txt
mv ../simple-examples/data/ptb.test.txt test.txt
mv ../simple-examples/data/ptb.valid.txt valid.txt
cd ..
rm -rf simple-examples/


Below you can find a few example training commands for training the baseline language model or GroC with different options.

Baseline (tied)

python -W ignore --data penn --dropouti 0.4 --dropouth 0.25 --seed 28 --batch_size 20 --epoch 1000\
 --save tied --cuda --cuda_device 0

GroC (char)

python -W ignore --data penn --dropouti 0.4 --dropouth 0.25 --seed 28 --batch_size 20 --epoch 1000\
--save groc_char --char_emb --char_update_ratio 0.3 --cuda --cuda_device 0

GroC (char, rel, def)

python -W ignore --data penn --dropouti 0.4 --dropouth 0.25 --seed 28 --batch_size 20 \
--epoch 1000 --save groc_full --char_emb --rel_emb --def_emb --char_update_ratio 0.3 --cuda --cuda_device 0

GroC for adaptation (char, rel, def, deep residual net, bias estimator)

python -W ignore --data penn --dropouti 0.4 --dropouth 0.25 --seed 28 --batch_size 20 \
--epoch 1000 --save groc_full --char_emb --predict_bias --joint_emb 400 --joint_emb_depth 4 --joint_dropout 0.6\ 
--joint_locked_dropout --joint_emb_activation Sigmoid --char_update_ratio 0.3 --cuda --cuda_device 0

For those who are interested, we also make our pretrained models and configurations publicly available for our experiments on conventional language modeling and cross-domain language modeling: Pretrained GroC models (Google Drive).


Download the News Crawl data here (, under “Monolingual language model training data”). Place the downloaded files in a directory called “raw” and run scripts/ to recreate our splits.

python --test_data data/news2007_train.news2008_test/ --save saved-models/[our model] --cuda  
--cuda_device 0 --seed 1234 --adapt_method [change_vocab|interpolate_neural|interpolate_unigram]



For questions and requests please email:,


Pytorch implementation of models described in "Grounded compositional outputs for adaptive language modeling", EMNLP 2020.




No releases published


No packages published