Skip to content

gchrupala/peppa

Repository files navigation

Peppa

Installation

Install prerequistes (preferably inside a virtual python environment):

pip install -r requirements.txt

Get data

Run

Train

There is a rudimentary command-line interface. You can run the training code by executing the function script run.py, and optionally passing in a configuration file.

python run.py --config_file hparams_base.yaml

Configuration files for the ablation experiments in the paper are in the repository, e.g.: hparams_base.yaml.

Use

The file conditions.yaml specifies the run IDs for each ablation configuration.

The corresponding trained models can be found at https://surfdrive.surf.nl/files/index.php/s/gNnZ4iSoKBDsOGK.

Evaluate

Create evaluation data files for trained model checkpoints as specified in the --versions argument:

python evaluate.py --versions 335

After generating evaluation data for all the necessary runs, you can:

python -c 'import pig.plotting as m; m.plots()'
python -c 'import pig.evaluation as m; m.test_run(); m.test_table()'
python -c 'import pig.plotting as m; m.recall_at_1_to_n_plot()
python -c 'import pig.evaluation as e; import pig.plotting as p; e.duration_effect(); p.duration_effect_plot()'

Run the minimal pairs evaluation:

  • First, run the evaluation for specified versions:
python evaluation_targeted_triplets.py --run --versions 335
  • Then, generate the results table and plot per-word results using a conditions file (see example conditions.yaml):
python evaluation_targeted_triplets.py --plot --conditions conditions.yaml