October 8: most of the code is here and documented. All parts are validated.
This repository contains code for Max-Margin Markov Graph Models (M3GMs) as described in the paper:
Yuval Pinter and Jacob Eisenstein. Predicting Semantic Relations using Global Graph Properties. EMNLP 2018.
(Stay tuned for official conference PDF and bibtex, around October 30.)
The project was written and tested in Python 3.6. Some packages needed to run it include:
nltk- with the
Write me, or open an issue, if you find more blocking dependencies!
The eventual goal of training an M3GM model and replicating the results from the paper runs through a number of intermediate steps. Here is the hopefully full linearized flowchart, with some detailed descriptions in following sections:
- Create a pickled WordNet prediction dataset in sparse matrix format, using
create_wn18_data.py. To use our exact dataset, obtain the distibution of WN18RR here and point the script at the text version.
- Obtain synset embeddings. These can be AutoExtend-based ones, which map directly to synsets, or any downloadable word embeddings which can then be averaged across synset lexemes, such as those from FastText.
- If your embeddings are word-level, synsetify them using
embed_from_words.py. Run it without parameters to see usage.
- Train an association model (for baseline results or for training an M3GM on top) using
pretrain_assoc.py. Demo command (for the result from the paper) given below.
- Train an M3GM using
predict_wn18.py. Demo command (for results from the paper) given below.
- If so inclined, tune the
optimize_alpha_per_relation.py. You will need to do some math later to translate into results comparable to those in the paper.
Disclaimer: some of the code here takes a while to run. Any suggestions for improving any of the calculations, or for getting things to run on GPUs for that matter, will be most appreciated.
This script trains a local association model using one of several models (see paper for details): Bilinear, TransE, Diag-R1 ("diagonal + rank-1 matrix"), DistMult.
Be sure to keep record of the embedding dimension used (no need to provide the dimension as an argument if initializing from an a pre-trained file) and of the association algorithm (
--assoc-mode), as these will be necessary for downstream M3GM training.
One parameter you may want to add depending on your target setup is
--rule-override, which trains modules for all relations, including the four symmetric ones (in WordNet).
It would also evaluate on trained modules in symmetric relations, rather than with a (high-accuracy) rule-based system.
The default behavior, without this parameter, is training said modules once every five epochs, as it helps with synset embeddings tuning.
--early-stopping method used is: for each dev epoch, if its MRR score is lower than both of the last two epochs, halt and return the best model so far.
- the auto-generated log file (avoid using
--no-log) will output many, many scores and their components for every single instance encountered.
--model-outis readable both by this code for test mode, and by downstream M3GM trainer (
python pretrain_assoc.py --input data/wn18rr.pkl --embeddings data/ft-embs-all-lower.vec --model-out models/pret_transE --nll --assoc-mode transE --neg-samp 10 --early-stopping --eval-dev
Max-Margin Markov Graph Models
The most powerful use case for M3GM is when we've trained a good association model, and augment it with weights for combinatorial graph features by way of M3GM training.
It is best if the association weights, as well as the word embeddings, are frozen from this point on, using the
--no-assoc-bp parameter. If we believe some of them to be bad, they can later be weighted down using the
optimize_alpha_per_relation.py post-processor, which computes a best-performing association component weight for each relation.
--model-only-init is a related parameter, which ensures that the M3GM component is trained over the data (makes more sense when considering that there's also an
--ergm-model input parameter which can be used for picking up training from a saved point).
A prerequesite for this code to run in the common mode is that both
assoc-mode are set to the same values that the association model was trained with.
- the auto-generated log file (avoid using
--no-log) will output ERGM scores for all instances and negative samples in training phase, and all cases of re-ranking in the development data traversals.
--model-outwill save the model in a four-file format that can later be read by both this script and the test-mode code (TODO).
--rerank-outprovides an input file for
optimize_alpha_per_relation.py. It includes all to-be-reranked lists from the dev set and scores from both association and graph components, as well as flags for the true instances.
python predict_wn18.py --input data/wn18rr.pkl --emb-size 300 --model models/pret_transE-ep-14 --model-only-init --assoc-mode transE --eval-dev --no-assoc-bp --epochs 3 --neg-samp 10 --regularize 0.01 --rand-all --skip-symmetrics --model-out models/from_pret_trE-3eps --rerank-out from_pret_trE-3eps.txt
A good entry point to try and play with the ERGM features underlying M3GM would be
ergm_feats.py. Be sure to enter them into the cache and feature set in
model.py so they can have weights trained for them.
Running the dataset creation code with the
--no-symmetrics flag would result in a dataset we called WN18RSR when working on this research. It contains only the seven asymmetric, nonreciprocal relations. All model results on it are abysmal, but you're welcome to try :)
- Add exploration Notebook for WordNet (WN) structure
- Add mapping of Synset codes from WN 1.6 all the way to 3.0.
- Move non-script code into
dy.parameter()calls (deprecated in
- Turn any remaining TODOs from here into repo issues