Skip to content


Repository files navigation



Install prerequistes (preferably inside a virtual python environment):

pip install -r requirements.txt

Get data



There is a rudimentary command-line interface. You can run the training code by executing the function script, and optionally passing in a configuration file.

python --config_file hparams_base.yaml

Configuration files for the ablation experiments in the paper are in the repository, e.g.: hparams_base.yaml.


The file conditions.yaml specifies the run IDs for each ablation configuration.

The corresponding trained models can be found at


Create evaluation data files for trained model checkpoints as specified in the --versions argument:

python --versions 335

After generating evaluation data for all the necessary runs, you can:

python -c 'import pig.plotting as m; m.plots()'
python -c 'import pig.evaluation as m; m.test_run(); m.test_table()'
python -c 'import pig.plotting as m; m.recall_at_1_to_n_plot()
python -c 'import pig.evaluation as e; import pig.plotting as p; e.duration_effect(); p.duration_effect_plot()'

Run the minimal pairs evaluation:

  • First, run the evaluation for specified versions:
python --run --versions 335
  • Then, generate the results table and plot per-word results using a conditions file (see example conditions.yaml):
python --plot --conditions conditions.yaml